A method for cold and hot separation of Kafka data storage and a cluster server system

Abstract

The present invention provides a method for cold and hot separation of Kafka data storage and a cluster server system. The method for cold and hot separation of Kafka data storage includes: writing data on the Kafka cluster server; according to the cold and hot data segmentation duration set in the Kafka cluster server, when the data writing exceeds the set segmentation duration, the data is converted from hot data to cold data; and automatically migrating the cold data to a Ceph remote disk mounted on the Kafka cluster server. Combining the data storage characteristics of Kafka, the present invention separates historical data into cold and hot data. The hot data is still stored on the local disk, and the cold data is stored in the Ceph manner. Kafka automatically and continuously realizes the ability to transfer cold data to Ceph storage, which not only saves storage costs but also hardly increases the manual maintenance cost.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a method for cold and hot separation of Kafka data storage and a cluster server system. Background Art

[0002] Kafka is an open-source message queue software used for data broadcasting and peak shaving for users. Upstream users can write data into Kafka, and multiple downstream users can read the written data at their respective independent paces. As a message queue service, Kafka has good data throughput capacity. By decoupling the production and consumption processes asynchronously, it plays a role in peak shaving and valley filling for business traffic. For example, during peak business traffic periods, when the business cannot process such a large amount of data in real time, the traffic can be first written into Kafka, and then the business can consume the data asynchronously slowly, converting the large traffic written in a short time into small traffic processed over a long time.

[0003] To achieve this decoupling ability, Kafka is designed to save the written data and delete it after a preset expiration time. Since the consumption behavior is asynchronous, the data will wait for a period of time after being written before the business consumes it. It is also possible that the business will consume data from historical data at an earlier time. Therefore, the data written into Kafka generally needs to be saved for several days or even weeks before being deleted, which will occupy a large amount of disk resources to store these historical data. At the same time, the high-availability design and reliable writing requirements of Kafka determine that business data will be stored at least in three copies, one master and two slaves, or even more, magnifying the demand for disk resources by 3 times or more.

[0004] In view of the above characteristics of Kafka, several solutions have been proposed currently. One solution is from the hardware dimension, by increasing the number of disks of a single server and optimizing the hardware architecture to reduce the price per unit of storage; another solution is from the business dimension, by optimizing the processing span and speed of historical data by the business, reducing the data storage duration, and thus reducing the storage usage.

[0005] Although the above methods can be used to reduce the storage cost of Kafka, in practical applications, they have different degrees of defects: (1) Increasing the number of disks on a single server and using the optimization of the hardware architecture to reduce the price per unit of storage. Its disadvantages are obvious. Firstly, the number of disks that can be stacked on a single server is limited, and the room for reducing the price per unit of storage is not much. Secondly, the larger the amount of data stored on a single machine, the higher the cost and slower the speed of recovery after a failure. Finally, the optimization cycle of hardware is often long, and there are many limitations in actual use, which is not conducive to large-scale promotion. (2) The processing span of historical data is long and the speed is slow, reducing the data storage duration. This method requires optimization on the user side. Since users rely heavily on the subscription characteristics of Kafka, in most cases, a piece of data will be used by multiple users at the same time. The cost of optimization from the user level is high, and the optimization effect is uncontrollable, making it difficult to implement in actual use.

[0006] In the prior art, for Kafka storage, in addition to increasing the number of disks on the server, a method of separating hot and cold data has also been proposed. For example, some data processing systems apply a hot data server cluster and a cold data server cluster. When there is cold data in the data stored in the hot data server cluster, the cold data is migrated to the target cold data server cluster, that is, a distributed storage system is adopted, and the data to be processed is divided into hot and cold data and stored on different servers respectively. However, this technical solution only solves the problem that the data storage capacity and data operation capacity provided by a single server cluster cannot meet the growing data processing requirements in the project, and does not further process the hot and cold data. At the same time, in the prior art, a solution applying Ceph technology has been proposed for storing data. As a unified, distributed file system designed for excellent performance, reliability, and scalability, Ceph can store hot data in the cache pool of fast hard disk devices and cold data in the data pool of slow hard disk devices. However, this prior art does not propose to adopt different technical solutions to process hot and cold data, so as to improve the overall utilization rate of data. Summary of the Invention

[0007] To solve the problems of the prior art, an embodiment of the present invention provides a method and a cluster server system for separating hot and cold data in Kafka data storage. The specific technical solution of the present invention is as follows:

[0008] According to one aspect of the present invention, a method for separating hot and cold data in Kafka data storage is provided, and the method includes:

[0009] Write data on the Kafka cluster server;

[0010] According to the cold and hot data segmentation duration set in the Kafka cluster server, when the data writing exceeds the set segmentation duration, the data is converted from hot data to cold data;

[0011] Automatically migrate the cold data to the Ceph remote disk mounted on the Kafka cluster server.

[0012] Optionally, the method further includes: the hot data is stored using the native multi-replica mechanism of Kafka, and the cold data is stored using the EC encoding method provided by Ceph itself. Among them, EC encoding divides the written data into N pieces of original data, and calculates M pieces of parity data through these N pieces of original data. Store the N+M pieces of data in different devices or nodes respectively, and restore all data blocks through any N pieces of data blocks among the N+M pieces. EC can tolerate the failure of M pieces of data, and the failure of any data less than or equal to M pieces can restore the original data through the remaining data. Ceph supports specifying different EC encoding methods in the form of plugins.

[0013] Optionally, the method further includes:

[0014] After writing data on the Kafka server, perform file segmentation on the written data according to a fixed size. The segmented file is only read as a historical file and cannot be modified anymore.

[0015] Optionally, the method further includes:

[0016] When the user's read request arrives, first search for files on the local disk of the Kafka server. If the corresponding data is not found on the local disk, then search on the Ceph remote disk, and return the read data to the user.

[0017] Optionally, the method further includes:

[0018] Traverse the files segmented on the local disk of the Kafka server, determine whether the file has expired, and if it has expired, delete it directly; if the file has not expired, then determine whether the file has exceeded the duration of hot data. If it has exceeded, move the corresponding file to the directory of the Ceph remote disk; traverse the files on the Ceph remote disk, and only need to determine whether the file has expired, and if it has expired, delete it.

[0019] According to another aspect of the present invention, there is provided a cluster server system for cold and hot separation of Kafka data storage, the system includes:

[0020] A Kafka cluster server for storing data written by users;

[0021] The Ceph cluster server mounts the storage space of the Ceph cluster server to each server of the Kafka cluster in the way of Ceph remote disks;

[0022] Wherein, a cold and hot data segmentation duration is set in the Kafka cluster server. When the data writing exceeds the set segmentation duration, the data is changed from hot data to cold data;

[0023] The cold data is automatically migrated to the Ceph remote disk mounted on the Kafka cluster server.

[0024] Optionally, in the system, the Kafka cluster server includes at least 3 servers, and the Ceph cluster server includes at least 13 servers, wherein 9 of the 13 Ceph cluster servers are original data storage servers and 4 are redundant data storage servers.

[0025] Optionally, in the system, each server in the Kafka cluster server mounts only one Ceph remote disk, and the naming and path of the remote disks between different servers are the same.

[0026] Optionally, in the system, when initializing the configuration interface of the Kafka cluster server, a cold and hot separation switch is set, and the cold and hot separation function is enabled when the switch is turned on.

[0027] Optionally, in the system, a hot data duration in the dimension of Topic is set in the Kafka cluster server. When the cold and hot separation function is enabled, the hot data duration of the topic is modified through a preset interface.

[0028] The method and cluster server system for cold and hot separation of Kafka data storage provided by the present invention utilize the cold and hot characteristics of the data itself and the advantage of fewer replicas in EC coding. By customizing and transforming the Kafka kernel code, configurable cold and hot data separation capabilities are provided. The hot data still uses the multi-replica mechanism of Kafka natively to ensure the efficiency and high availability of data access. The cold data is changed to use the EC coding method provided by Ceph itself to compress the number of replicas of at least 3 times to less than 1.5. Since the cold data is accessed much less frequently than the hot data and the cold data is read-only and will not be modified again, it is equivalent to sacrificing the performance of a small number of read scenarios to reduce the storage usage.

[0029] Compared with a Kafka cluster, in the case of 3 replicas, if the cluster is to remain operational, at most 2 machines are allowed to fail. In the case of Ceph approaching 1.5 replicas, based on the erasure coding feature, the data is split into 9 original data blocks + 4 redundant data blocks, allowing at most 4 machines to fail without affecting normal use, with better fault tolerance performance than Kafka. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In the following description of exemplary embodiments with reference to the accompanying drawings, more details, features, and advantages of the present invention are disclosed. In the drawings:

[0031] Figure 1 A block diagram of a cluster server system for cold and hot separation of Kafka data storage according to an exemplary embodiment of the present invention is shown;

[0032] Figure 2 A flowchart of a method for cold and hot separation of Kafka data storage according to an exemplary embodiment of the present invention is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Instead, these embodiments are provided to more thoroughly and completely understand the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0034] It should be understood that the steps recited in the method embodiments of the present invention can be executed in a different order and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this regard.

[0035] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependent relationships.

[0036] It should be noted that the modifiers "a" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".

[0037] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0038] Figure 1 The cluster server system 100 for cold and hot separation of Kafka data storage in an exemplary embodiment is shown. The cluster server system 100 includes a Kafka cluster server 101 for storing data written by users. The Kafka cluster server 101 is composed of multiple servers, preferably 3 servers. Each server has a local disk and a remote disk. The cluster server system 100 also includes a Ceph cluster server 102, which mounts the storage space of the Ceph cluster server 102 to each server of the Kafka cluster server 101 in the form of a Ceph remote disk. A cold and hot data segmentation duration is set in the Kafka cluster server 101. When the data writing is completed and exceeds the set segmentation duration, the data is converted from hot data to cold data, and the cluster will automatically migrate the cold data from the directory of the local disk to the directory of the remote disk, so as to automatically migrate the cold data to the Ceph cluster server 102 corresponding to the remote disk mounted on the Kafka cluster server 101. The cluster server system 100 may have multiple Kafka cluster servers 101, and the multiple cluster servers 101 may correspond to one Ceph cluster server 102. Since Ceph itself is approximately infinitely scalable in storage space and uses EC coding to store data, the replication factor can be controlled within 1.5.

[0039] Compared with the Kafka cluster, in the case of 3 replicas, that is, using 3 Kafka servers, if the cluster is to be kept working properly, at most 2 servers are allowed to fail. In the case of Ceph with a replication factor close to 1.5, based on the EC coding feature, the data is segmented into 9 original data blocks + 4 redundant data blocks, and at most 4 machines are allowed to fail without affecting normal use, which has better fault tolerance performance than Kafka. Therefore, in the cluster server system 100, the Kafka cluster server 100 includes at least 3 servers, and the Ceph cluster server 102 includes at least 13 servers, among which 9 are original data storage servers and 4 are redundant data storage servers among the 13 Ceph cluster servers.

[0040] The deployment of the cluster server system 100 involves two types of cluster servers, Ceph and Kafka. The specific deployment steps are as follows: (1) Before deploying the Kafka cluster server, it is necessary to first deploy the Ceph cluster server. Due to the principle of EC coding, if an encoding scheme of 9 original data blocks + 4 redundant data blocks is adopted, at least 9 + 4 = 13 servers are required to deploy a Ceph cluster and provide the total storage resource sum of the disks of 9 servers. (2) Mount the remote disk of Ceph on the servers to be used for deploying Kafka. For the convenience of subsequent management, each server only mounts one Ceph remote disk, and the naming and path of the remote disks between different servers are kept the same. (3) According to the data volume expected by the user, evaluate the number of machines required for hot data based on a segmentation duration of 1 day for hot data and more than 1 day for cold data, and deploy the Kafka cluster on the corresponding machines, configuring the local disk as the hot data directory and the remote disk as the cold data directory.

[0041] Cold and hot separation configuration needs to be carried out on the cluster server system 100. By adding a configuration interface to the Kafka cluster server 101, flexible cold and hot separation configuration capabilities are provided. First, initialize the configuration interface for the Kafka cluster server 101. In the configuration file of Kafka, add global cold and hot separation switches and cold and hot data segmentation duration configurations. The cold and hot separation switch and the effective duration of hot data can be set. The cold and hot separation function will only be enabled when the switch is turned on, and data exceeding the hot data duration will be considered cold data. After the cluster starts, by reading the configuration file, this configuration takes effect by default for the entire cluster dimension. In general settings, the hot data is 1 day, that is, the cold and hot data segmentation duration is usually set to 1 day. The user sets the data within 1 day from the moment it is written into the Kafka cluster server 101 as hot data, and the data exceeding 1 day written into the Kafka cluster server 101 by the user is set as cold data, and this cold and hot data segmentation duration can be freely set.

[0042] In the cluster server system 100, the duration of hot data in the Topic dimension can be configured. A Topic is a message queue that users can use in Kafka. In a Kafka cluster, there are multiple topics. For different users, different topics are used for data reading and writing, and the requirements for the retention duration of data in the topics are different. A single hot and cold data rule does not meet the needs of complex user scenarios. For example, a user needs to retain historical data for a long time, but except for the data on the first day, other data can be stored as cold data; another user only needs to retain historical data for two or three days, but all the data is hot data and cannot be converted to cold data storage. Facing the above different user requirements, the configuration of the duration of hot data in the Topic dimension can better take into account the characteristics of users' data usage. Since the hot and cold situations of different topics are different, the cluster server system 100 provides an online configuration function for the duration of hot data in the Topic. When the cluster is started and the hot and cold separation function is enabled, the duration of hot data in the Topic can be modified through a preset interface. When the duration of hot data in the Topic dimension is set, the configuration of the duration of hot data in the cluster dimension will no longer take effect for this Topic.

[0043] In the cluster server system 100, it is necessary to manage the cold data storage resources. Since all cold data is stored in the corresponding Ceph cluster server 102, the resource capacity of the Ceph cluster determines the cold data storage capacity of the Kafka cluster, and corresponding management capabilities are required. Specifically:

[0044] Since the minimum scale of the Ceph cluster server 102 is 13, and the minimum scale of the Kafka cluster server 101 is 3, for Kafka clusters with a small scale, there are cases where multiple Kafka cluster servers 101 share the same Ceph cluster server 102. Therefore, when mounting remote disks, it is necessary to clearly distinguish the storage spaces of different clusters on the Ceph cluster server 102 and manage the usage of cold data storage by each cluster.

[0045] When a Ceph cluster server 102 with the minimum scale has been filled with cold data of the corresponding Kafka cluster, it is not allowed to create a new Kafka cluster to continue mounting the remote disk of this Ceph cluster to avoid the scale of the Ceph cluster becoming too large uncontrollably.

[0046] When the usage of the storage space of the Ceph cluster reaches more than 80%, in order to ensure that there is enough redundant space available, it is necessary to add a new cluster in time for cluster expansion.

[0047] Figure 2A method for cold and hot separation of Kafka data storage in an exemplary embodiment is shown. The method includes step 201: A user writes data on a Kafka cluster server and processes the written data. Specifically, the logic of adding a disk write path is added to ensure that the data can only be written to the local disk, that is, the data is saved to the local disk and the files are split by a fixed size. Among them, the split files can be used as historical files and can only be read and cannot be modified anymore. Step 202: According to the cold and hot data split duration set in the Kafka cluster server, when the data write exceeds the set split duration, the data changes from hot data to cold data. Step 203: Automatically migrate the cold data to the Ceph remote disk mounted on the Kafka cluster server. Among them, the hot data is always stored on the Kafka cluster server and stored using the native multi-replica mechanism of Kafka. For example, 3 servers are used for mutual backup storage. The cold data is migrated to the remote Ceph cluster server for storage and stored using the EC encoding method provided by Ceph itself. Among them, EC encoding is to divide the written data into N pieces of original data, and calculate M pieces of check data through these N pieces of original data. The N + M pieces of data are stored in different devices or nodes respectively, and all data blocks can be restored through any N pieces of data blocks among the N + M pieces. EC can tolerate the failure of M pieces of data, and the original data can be restored from the remaining data when any data failure less than or equal to M pieces occurs. In a preferred embodiment of the present invention, 9 (9 pieces of original data) + 4 (4 pieces of check data) = 13 servers are used to deploy a Ceph cluster.

[0048] The method 200 for cold and hot separation of Kafka data storage further includes the step of the user reading a data request in the cluster server system 100. The specific steps are to first search for files on the local disk of the Kafka server 101. If the corresponding data is not found on the local disk, then search on the Ceph remote disk, that is, search for files on the Ceph cluster server 102, and return the read data to the user.

[0049] The method 200 for separating hot and cold data storage of Kafka further includes a step of processing expired file deletion, which needs to consider the differentiation function of local disks and remote disks, as well as the separation function of hot and cold data. The specific step of processing expired file deletion is as follows: after writing data on the Kafka server, the written data is segmented into files according to a fixed size, and then all the segmented files are continuously traversed. When traversing the files segmented on the local disk of the Kafka server, first judge whether the file has expired. If it has expired, it is directly deleted; if the file has not expired, then judge whether the file has exceeded the duration of hot data. If it has exceeded, the corresponding file is moved to the directory of the Ceph remote disk; when traversing the files on the Ceph remote disk, only judge whether the file has expired, and if it has expired, it is normally deleted.

[0050] In the present invention, in combination with the data storage characteristics of Kafka, historical data is separated into hot and cold data. The hot data is still stored on the local disk, using an architecture with at least 3 replicas; the cold data is stored in the Ceph manner, and the replicas can be compressed to less than 1.5. In most user scenarios, the proportion of cold data is much larger than that of hot data, so the storage cost can be greatly saved. By integrating this hot and cold data separation function into the Kafka itself, the ability of Kafka to automatically and continuously transfer cold data to Ceph storage can be realized without manual intervention, which not only saves storage costs but also hardly increases the manual maintenance cost. In the present invention, Kafka selects the default data retention period of 7 days, and sets that the data will become cold data 1 day after being written, that is, the hot and cold data ratio of Kafka is 1:6. Using this solution, about 44.5% of storage resources can be saved.

Claims

1. A method for cold and hot separation of Kafka data storage, characterized in that, The method includes: Mount the storage space of the Ceph cluster server to each server of the Kafka cluster server in the way of Ceph remote disk. Each server has a local disk and the Ceph remote disk. Write data on the local disk of the Kafka cluster server. According to the cold and hot data segmentation duration set in the Kafka cluster server, when the data writing exceeds the set segmentation duration, the data changes from hot data to cold data. Automatically migrate the cold data to the Ceph cluster server corresponding to the Ceph remote disk mounted on the Kafka cluster server.

2. The method for cold and hot separation of Kafka data storage according to claim 1, the method further includes: The hot data is stored using the multi-replica mechanism native to Kafka, and the cold data is stored using the EC coding method provided by Ceph itself.

3. The method for cold and hot separation of Kafka data storage according to claim 1, the method further includes: After writing data on the Kafka cluster server, segment the written data by a fixed size. The segmented files are only read as historical files and cannot be modified anymore.

4. The method for cold and hot separation of Kafka data storage according to claim 1, the method further includes: When the user makes a read request, first search for files on the local disk of the Kafka cluster server. If the corresponding data is not found on the local disk, then search on the Ceph remote disk and return the read data to the user.

5. The method for cold and hot separation of Kafka data storage according to claim 1, the method further includes: Traverse the segmented files on the local disk of the Kafka cluster server, and judge whether the files have expired. If they have expired, delete them directly. If the files have not expired, then judge whether the files have exceeded the duration of hot data. If they have exceeded, move the corresponding files to the directory of the Ceph remote disk. Traverse the files on the Ceph remote disk, and judge whether the files have expired. If they have expired, delete them.

6. A cluster server system for cold and hot separation of Kafka data storage, characterized in that, The system includes: A Kafka cluster server for storing the data written by the user on the local disk. A Ceph cluster server mounts the storage space of the Ceph cluster server to each server of the Kafka cluster server in the way of Ceph remote disk. Each server has the local disk and the Ceph remote disk. Wherein, a cold and hot data segmentation duration is set in the Kafka cluster server. When the data writing exceeds the set segmentation duration, the data changes from hot data to cold data. Automatically migrate the cold data to the Ceph cluster server corresponding to the Ceph remote disk mounted on the Kafka cluster server.

7. The cluster server system for cold and hot separation of Kafka data storage according to claim 6, characterized in that The Kafka cluster server includes at least 3 servers, and the Ceph cluster server includes at least 13 servers, among which 9 of the 13 servers are original data storage servers and 4 are redundant data storage servers.

8. The cluster server system for hot and cold separation of Kafka data storage according to claim 6, wherein Each server in the Kafka cluster server mounts only one Ceph remote disk, and the naming and path of the remote disks among different servers are the same.

9. The cluster server system for hot and cold separation of Kafka data storage according to claim 6, wherein When initializing the configuration interface of the Kafka cluster server, a hot and cold separation switch is set, and the hot and cold separation function is enabled when the switch is turned on.

10. The cluster server system for hot and cold separation of Kafka data storage according to claim 9, wherein The duration of hot data in the Topic dimension is set in the Kafka cluster server, and when the hot and cold separation function is enabled, the duration of hot data of the topic is modified through a preset interface.

Images