Kafka partition copy migration method and device, computer equipment and readable storage medium

Abstract

The invention relates to a kafka partition copy migration method and device, computer equipment and a readable storage medium. The method is applied to a first node in a node cluster, and comprises the following steps: when it is determined that a second node in the node cluster fails, determining a target cloud storage device mounted on the second node, and determining a node for re-mounting the target cloud storage device; under the condition that the determined node is the first node, sending a device mounting request for the target cloud storage device to the cloud server, so that the cloud server mounts the target cloud storage device to the first node after unloading the target cloud storage device from the second node; and under the condition of successful mounting, determining a copy in the target cloud storage device as a target copy, determining a target partition to which the target copy belongs, and updating metadata of the target partition so as to migrate the target copy of the target partition from the second node to the first node. By adopting the method, the migration efficiency of the copies of the partitions can be improved.

Description

Technical Field

[0001] The present application relates to the field of Internet and cloud technology, and in particular to a Kafka partition replica migration method, device, computer equipment and readable storage medium. Background Art

[0002] With the development of distributed and cloud technologies, message middleware has emerged. Message middleware is a subsystem in a distributed system that focuses on sending and receiving data and integrates other subsystems in the distributed system using an efficient and reliable asynchronous information transmission mechanism. It is often used in scenarios such as asynchronous processing, application decoupling, traffic peak shaving, and log processing. An example of message middleware is Kafka.

[0003] In traditional technologies, data migration is achieved by replicating data in Kafka distributed clusters, and the migration efficiency is low. Summary of the invention

[0004] Based on this, it is necessary to provide a Kafka partition replica migration method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the migration efficiency of partition replicas in response to the above technical problems.

[0005] On the one hand, the present application provides a Kafka partition copy migration method, which is applied to a first node in a node cluster, each node in the node cluster is deployed with a Kafka service and mounted with a cloud storage device, and the cloud storage device stores a copy of the partition, including: when it is determined that a second node in the node cluster fails, determining a target cloud storage device mounted on the second node, and determining a node for remounting the target cloud storage device; when the determined node is the first node, sending a device mount request for the target cloud storage device to a cloud server, so that the cloud server mounts the target cloud storage device to the first node after uninstalling the target cloud storage device from the second node; when the mounting is successful, determining the copy in the target cloud storage device as the target copy, determining the target partition to which the target copy belongs, and updating the metadata of the target partition to migrate the target copy of the target partition from the second node to the first node.

[0006] On the other hand, the present application also provides a Kafka partition copy migration device, wherein each node in the node cluster includes the device, each node in the node cluster is deployed with a Kafka service and mounted with a cloud storage device, and a copy of the partition is stored in the cloud storage device. The device in the first node in the node cluster includes: a device determination module, which is used to determine the target cloud storage device mounted on the second node when it is determined that a fault occurs in the second node in the node cluster, and determine the node for remounting the target cloud storage device; a mount request sending module, which is used to send a device mount request for the target cloud storage device to the cloud server when the determined node is the first node, so that the cloud server mounts the target cloud storage device to the first node after uninstalling the target cloud storage device from the second node; a metadata update module, which is used to determine the copy in the target cloud storage device as the target copy when the mounting is successful, determine the target partition to which the target copy belongs, and update the metadata of the target partition to migrate the target copy of the target partition from the second node to the first node.

[0007] On the other hand, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned Kafka partition replica migration method when executing the computer program.

[0008] On the other hand, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned Kafka partition replica migration method are implemented.

[0009] On the other hand, the present application also provides a computer program product, including a computer program, which implements the steps in the above-mentioned Kafka partition replica migration method when executed by a processor.

[0010] The above-mentioned Kafka partition replica migration method, device, computer equipment, computer-readable storage medium and computer program product, when determining that the second node in the node cluster fails, determine the target cloud storage device mounted on the second node, and determine the node for remounting the target cloud storage device, and when the determined node is the first node, send a device mount request for the target cloud storage device to the cloud server, so that the cloud server mounts the target cloud storage device to the first node after uninstalling the target cloud storage device from the second node, and when the mount is successful, determine the replica in the target cloud storage device as the target replica, determine the target partition to which the target replica belongs, and update the metadata of the target partition to migrate the target replica of the target partition from the second node to the first node. Thus, the migration of the replica in the partition is achieved by unloading and mounting, which improves the migration efficiency compared to the migration by copying. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1A It is an application environment diagram of the Kafka partition replica migration method in some embodiments;

[0013] Figure 1B A schematic diagram of a Kafka partition replica migration method in some embodiments;

[0014] Figure 2 A schematic diagram of a process of migrating a Kafka partition replica in some embodiments;

[0015] Figure 3 A schematic diagram of a Kafka partition replica migration method in some other embodiments;

[0016] Figure 4 A schematic diagram of a Kafka partition replica migration method in some further embodiments;

[0017] Figure 5 It is a structural block diagram of a Kafka partition replica migration device in some embodiments;

[0018] Figure 6 is an internal structure diagram of a computer device in some embodiments;

[0019] Figure 7 1 is a diagram of the internal structure of a computer device in some other embodiments. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0021] The Kafka partition replica migration method provided in the embodiment of the present application can be applied to Figure 1A In the application environment shown, the application scenario includes a management node 102, a node cluster 104, and a cloud server 106. The node cluster 104 includes multiple nodes. In the figure, three nodes are taken as an example. These three nodes are node 1 to node 3, but the present application does not limit the number of nodes contained in the node cluster 104. A kafka service is deployed on each node, and a Broker in Kafka exists on each node. The Broker in Kafka refers to an instance of the Kafka service. Each Broker has a unique identifier (i.e., broker.id) and can communicate with other Brokers in the Kafka cluster. The Broker is responsible for processing messages from producers and consumers, and managing stored message data. Each node is mounted with one or more cloud storage devices, and multiple refers to at least two, and the cloud storage device belongs to the cloud server 106. Among them, the management node 102, the node cluster 104, and the cloud server 106 can communicate through the network, and each node in the node cluster 104 can communicate with each other through the network. A kafka service is deployed on each node in the node cluster 104, so the node cluster can also be called a kafka service. The cloud storage device may be any disk or block storage device (referred to as block device). The node cluster may be a distributed Kafka cluster.

[0022] Each node in the node cluster 104 can communicate with the management node 104 respectively. For example, each node can send a heartbeat packet to the management node 104 respectively. The management node 104 can update the heartbeat time of the node according to the heartbeat packet sent by the node. The management node 104 can record and update the heartbeat time of each node and the status of each node. If the heartbeat time of the node is not updated within the preset time, it is determined that the node has a fault, such as downtime, so that the status of the node is changed from online to offline. The preset time can be set and modified according to the actual situation. Each node can monitor the status of each node recorded by the management node 104. For example, node 1 can monitor the status of node 2 and node 3, and determine whether the node has a fault according to the status of the node.

[0023] A topic in Kafka can be divided into multiple partitions, and a partition can have one replica or multiple replicas. The cloud storage device mounted on each node in the node cluster 104 stores a copy of the Kafka partition. The Kafka partition is divided into single-copy partitions and multi-copy partitions. The Kafka partition is referred to as a partition for short. Each partition has a unique identifier. A single-copy partition is a partition with only one replica (Partition), and a multi-copy partition is a partition with multiple replicas (Replica), and multiple means at least two. For multiple replicas of a multi-copy partition, one of the multiple replicas is the leader replica (LeaderReplica), and the rest are follower replicas (FollowerReplica). The master replica is a replica used to process read and write requests. The message sent by the producer is first written to the master replica, and the follower replica is used to copy, i.e., synchronize messages from the master replica. One or more replicas can be stored in a cloud storage device. Such as Figure 1B As shown, it shows a schematic diagram of the distribution of replicas stored in the cloud storage device mounted on the node. There are 6 partitions in the figure, namely single-copy partition 1, single-copy partition 2, single-copy partition 3, multi-copy partition 4, multi-copy partition 5 and multi-copy partition 6. Multi-copy partition 4 has 2 replicas, namely replica R4-1 and replica R4-2, and replica R4-1 is a slave replica, and replica R4-2 is a master replica. Multi-copy partition 5 has 2 replicas, namely replica R5-1 and replica R5-2, and replica R5-1 is a master replica, and replica R5-2 is a slave replica. Multi-copy partition 6 has 2 replicas, namely replica R6-1 and replica R6-2, and replica R6-1 is a master replica, and replica R6-2 is a slave replica. Node 1 is mounted with cloud storage device 1 and cloud storage device 2. Cloud storage device 1 stores a copy R1 of single-copy partition 1 and a copy R4-1 of multi-copy partition 4. Cloud storage device 2 stores a copy R5-1 of multi-copy partition 5. Node 2 is mounted with cloud storage device 3 and cloud storage device 4. Cloud storage device 3 stores a copy R2 of single-copy partition 2 and a copy R4-2 of multi-copy partition 4. Cloud storage device 4 stores a copy R6-1 of multi-copy partition 6. Node 3 is mounted with cloud storage device 3 and cloud storage device 4. Cloud storage device 3 stores a copy R3 of single-copy partition 3 and a copy R6-2 of multi-copy partition 6. Cloud storage device 4 stores partition R5-2 of multi-copy partition 5.

[0024] The management node 104 may also store metadata for each partition, where the metadata for the partition includes identifiers of each replica of the partition, identifiers of the master replica of the partition, identifiers of the slave replicas of the partition, identifiers of the cloud storage devices to which the replicas of the partition belong, and identifiers of the nodes on which the cloud storage devices to which the replicas of the partition belong are mounted.

[0025] The management node 104 may also store metadata of the cloud storage device, and the metadata of the cloud storage device includes the identifier of the copy stored in the cloud storage device, the identifier of the mounting node of the cloud storage device, the mounting status of the cloud storage device, etc. The mounting status can be divided into mounted, in the mounting process, or failed to mount. When the mounting status in the metadata of the cloud storage device is in the mounting process, the identifier of the mounting node in the metadata of the cloud storage device is the identifier of the node to which the cloud storage device is being attempted to be mounted; when the mounting status in the metadata of the cloud storage device is mounted, the identifier of the mounting node in the metadata of the cloud storage device is the identifier of the node to which the cloud storage device has been mounted; when the mounting status in the metadata of the cloud storage device is failed to mount, the identifier of the mounting node in the metadata of the cloud storage device is the identifier of the node on which the cloud storage device failed to mount. Each node in the node cluster 104 may also monitor the metadata of the cloud storage device and the metadata of the partition, so as to monitor the changes in the metadata in real time.

[0026] Specifically, for any node in the node cluster 104, such as the first node (the first node is any node in the node cluster 104 that has not failed), when it is determined that the second node in the node cluster has failed, the first node determines the target cloud storage device mounted on the second node, and determines the node for remounting the target cloud storage device. When the determined node is the first node, the first node sends a device mount request for the target cloud storage device to the cloud server, so that the cloud server mounts the target cloud storage device to the first node after uninstalling the target cloud storage device from the second node. When the mounting is successful, the first node determines the copy in the target cloud storage device as the target copy, determines the target partition to which the target copy belongs, and updates the metadata of the target partition to migrate the target copy of the target partition from the second node to the first node. For example Figure 1B In the example, when node 1 experiences an abnormal downtime, node 2 is determined as the node for remounting cloud storage device 1, and node 3 is determined as the node for remounting cloud storage device 2. Node 2 sends a device mount request for cloud storage device 1 to the cloud server, and node 3 sends a device mount request for cloud storage device 2 to the cloud server.

[0027] Among them, the management node 102 can be a server or a terminal. The management node 102 can be an independent physical server or a server cluster or distributed system composed of multiple physical servers. Each node in the node cluster 104 can be a terminal or a server. The terminal can be but is not limited to a desktop computer, a laptop computer, a smart phone, and a tablet computer. The cloud server 106 is used to provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The management node 102, the node cluster 104, and the cloud server 106 can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0028] In some embodiments, Figure 2 As shown, a kafka partition replica migration method is provided, and each node in the node cluster 104 can execute the method. The method is applied to the first node in the node cluster as an example for explanation. The first node is any node in the node cluster that has not failed, such as downtime, and includes the following steps 202 to 206. Among them:

[0029] Step 202, when it is determined that a second node in the node cluster fails, determine a target cloud storage device mounted on the second node, and determine a node for remounting the target cloud storage device.

[0030] The management node stores the status of each node in the node cluster. When the node status is offline, it indicates that the node has failed. When the node status is online, it indicates that the node has not failed. The node queries the status of any node from the management node to determine whether the node has failed.

[0031] A cloud storage device is a storage device managed by a cloud server. A storage device can be a block storage device such as a disk. The target cloud storage device refers to the cloud storage device mounted on the second node. Mounting a cloud storage device on a node can be understood as assigning a cloud storage device to a node or binding a cloud storage device to a node. Logically, the cloud storage device mounted on a node is the cloud storage device of the node.

[0032] Specifically, each node in the node cluster that has not failed can determine the node for remounting the target cloud storage device. If the determined node is itself, it sends a device mount request for the target cloud storage device to the cloud storage device. If the determined node is not itself, it does not send a device mount request. Each node that has not failed uses the same strategy to determine the node for remounting the target cloud storage device, so that the nodes determined by each node that has not failed are consistent.

[0033] In some embodiments, the first node can use the number of nodes in the node cluster that have not failed as the first number, and use the number of cloud storage devices in the second node, i.e., the target cloud storage devices, as the second number. When the first number and the second number are consistent, an equal distribution strategy can be used to determine the node to remount the target cloud storage device. The equal distribution strategy is to evenly distribute each target cloud storage device to different nodes. For example, the first node can negotiate with the remaining nodes in the node cluster that have not failed, such as voting, to determine the nodes to which each target cloud storage device needs to be remounted. For example, there are two target cloud storage devices, namely device 1 and device 2, and it is determined that device 1 is remounted to the first node and device 3 is remounted to the third node. The third node is different from the first node and the third node has not failed.

[0034] Step 204, when the determined node is the first node, sends a device mount request for the target cloud storage device to the cloud server, so that the cloud server mounts the target cloud storage device to the first node after uninstalling the target cloud storage device from the second node.

[0035] Specifically, the first node may first send a device uninstallation request to the cloud server, the device uninstallation request carries the identifier of the target cloud storage device and the identifier of the second node, and the device uninstallation request is used to request the cloud server to uninstall the target cloud storage device from the second node. After uninstallation, the first node may send a device mount request to the cloud server again, the device mount request may carry the identifier of the target cloud storage device and the identifier of the first node. The device mount request is used to request to mount the target cloud storage device to the first node.

[0036] In some embodiments, the device mount request is used to request the cloud server to unmount the target cloud storage device from the second node and mount the target cloud storage device to the first node. The device mount request may carry the identifier of the target cloud storage device, the identifier of the first node, and the identifier of the second node. In response to the device mount request, the cloud server unmounts the target cloud storage device from the second node, and after unmounting, mounts the target cloud storage device to the first node. In response to the device mount request, the cloud server executes the action of mounting the target cloud storage device to the first node, and returns the mounting status to the first node, such as mounting in progress or mounting successful, to notify the first node of the progress and status of the mounting.

[0037] Step 206, if the mounting is successful, determine the copy in the target cloud storage device as the target copy, determine the target partition to which the target copy belongs, and update the metadata of the target partition to migrate the target copy of the target partition from the second node to the first node.

[0038] Among them, there may be at least one replica, namely the target replica, in the target cloud storage device. The target partition refers to the partition to which the target replica belongs. The metadata of the target partition includes the storage path of the target replica, and updating the metadata of the target partition may be to update the storage path of the target replica. The storage path of the target replica is used to access the target replica. The storage path includes the identifier of the node on which the cloud storage device to which the target replica belongs is mounted. Updating the metadata of the target partition includes but is not limited to updating the identifier of the node on which the cloud storage device to which the target replica belongs in the metadata of the target partition is mounted from the identifier of the second node to the identifier of the first node. The metadata of the cloud storage device includes the identifier of the partition to which the replica in the cloud storage device belongs. The first node can determine the partition to which each target replica belongs, namely the target partition, based on the metadata of the target cloud storage device.

[0039] Specifically, in the process of mounting the target cloud storage device to the first node, the first node may send a first metadata update request for the target cloud storage device to the management node, the metadata update request being used to request that the identifier of the mounting node in the metadata of the target cloud storage device be set to the identifier of the first node, and the mounting status in the metadata of the target cloud storage device be set to mounting, the mounting node being the node to which the cloud storage device has been mounted or is being mounted. After mounting the target cloud storage device to the first node, the first node may send a second metadata update request for the target cloud storage device to the management node, the second metadata update request being used to request that the mounting status in the metadata of the target cloud storage device be set to mounting successfully. Other nodes may determine that the mounting is successful when they monitor that the mounting status in the metadata of the target cloud storage device is updated to mounting successfully.

[0040] In the above-mentioned Kafka partition replica migration method, when it is determined that the second node in the node cluster fails, the target cloud storage device mounted on the second node is determined, and the node for remounting the target cloud storage device is determined. When the determined node is the first node, a device mount request for the target cloud storage device is sent to the cloud server, so that the cloud server mounts the target cloud storage device to the first node after uninstalling the target cloud storage device from the second node. When the mount is successful, the replica in the target cloud storage device is determined as the target replica, the target partition to which the target replica belongs is determined, and the metadata of the target partition is updated to migrate the target replica of the target partition from the second node to the first node. Thus, the migration of the replica in the partition is realized by the unloading and mounting method, without the need for data replication, which improves the migration efficiency compared to the replication migration.

[0041] In addition, if there is a single-copy partition in the target cloud storage device, the data in the single-copy partition cannot be copied due to the failure of the second node, so the data in the single-copy partition cannot be migrated, making the data in the single-copy partition unavailable. The Kafka partition copy migration method provided by the present application makes each copy in the target cloud storage device accessible by remounting, so there is no problem of unavailability of data in the single-copy partition, which improves the high availability of data.

[0042] In some embodiments, the partition is divided into a single-copy partition and a multi-copy partition, the single-copy partition has one and only one copy, the multi-copy partition has at least two copies, and there is a master copy and at least one slave copy among the at least two copies. The method also includes: in the event of a mounting failure, sending a mounting failure notification for the target cloud storage device to the management node, so that the management node updates the mounting status of the target cloud storage device to mounting failure; when the mounting status of the target cloud storage device is updated to mounting failure, there is a master copy of the first multi-copy partition in the target cloud storage device, and there is a slave copy of the first multi-copy partition in the cloud storage device mounted by the first node, then determining a new master node corresponding to the first multi-copy partition; when the determined new master node is the first node, sending a master-slave change request for the first multi-copy partition to the management node, so that the management node changes the slave copy of the first multi-copy partition in the cloud storage device mounted by the first node to the master copy of the first multi-copy partition.

[0043] The master node corresponding to the multi-copy partition refers to the node where the cloud storage device where the master copy of the multi-copy partition is located is mounted. Figure 3In the example, the master copy of multi-copy partition 4 is copy R4-2, which is in cloud storage device 3, and cloud storage device 3 is mounted on node 2. Therefore, the master node corresponding to multi-copy partition 4 is node 2. The first multi-copy partition can be any multi-copy partition. The master-slave change request is used to request to change the slave copy of the first multi-copy partition in the cloud storage device mounted on the first node to the master copy of the first multi-copy partition. In the case of a mount failure, the copy of the single-copy partition in the target cloud storage device is unavailable, for example Figure 3 The replica R1 is unavailable.

[0044] Specifically, each non-faulty node that has a slave copy of the first multi-copy partition can negotiate or elect a new master node corresponding to the first multi-copy partition, and the new master node meets the condition that there is a slave copy of the first multi-copy partition. Figure 3 The middle node is 3, and the second node is Figure 3 Taking node 1 as an example, since there is a replica R5-1 of the multi-copy partition 5 in cloud storage device 2, and replica R5-1 is the master replica, there is a replica R5-2 of the multi-copy partition 5 in node 3, and replica R5-2 is a slave replica, therefore, in the case that the cloud storage device 2 fails to be remounted (it can be remounted by node 2 or node 3), node 3 determines the new master node corresponding to the multi-copy partition 5. Since there is no slave replica of the multi-copy partition 5 in node 2, node 3 is determined to be the new master node of the multi-copy partition 5. Therefore, node 3 sends a master-slave change request for the multi-copy partition 5 to the management node. In response to the master-slave change request, the management node changes the replica R5-2 of the multi-copy partition 5 in the cloud storage device 4 from a slave replica to a master replica, making replica R5-2 the new master replica.

[0045] In some embodiments, in addition to determining a new master node corresponding to the first multi-replica partition, a new slave node corresponding to the first multi-replica partition is also determined, where the slave node refers to the node on which the cloud storage device to which the slave replica belongs is mounted. After the slave replica of the first multi-replica partition in the cloud storage device mounted on the first node is changed to the master replica of the first multi-replica partition, the new slave node corresponding to the first multi-replica partition adds a slave replica of the first multi-replica partition in the designated cloud storage device mounted on the new slave node, and copies the data in the new master replica of the first multi-replica partition to the newly added slave replica. The newly added slave replica can have the same name as the master replica of the first multi-replica partition in the target cloud storage device. Figure 3 In the figure, the new master node corresponding to the multi-replica partition 5 is node 2, a new replica R5-1 of the multi-replica partition 5 is added to the cloud storage device 4 of node 2, and the data in R5-2 is copied to R5-1, so that the newly added replica R5-1 in the cloud storage device 4 becomes the new slave replica of the multi-replica partition 5.

[0046] In some embodiments, if the master copy of the first multi-copy partition is in other normal nodes, there is no need to determine the new master node corresponding to the first multi-copy partition, nor is there a need to change the master-slave node. It is only necessary to determine the new slave node for the first multi-copy partition, and add a slave copy of the first multi-copy partition in a cloud storage device of the new slave node, and copy the data in the master copy to the newly added slave copy.

[0047] In this embodiment, since the master copy is responsible for the read and write functions, in the event of a mounting failure, the slave copy of the first multi-copy partition in the cloud storage device mounted by the first node is changed to the master copy of the first multi-copy partition, so that the first multi-copy partition can be read and written normally.

[0048] In some embodiments, when the mount status of the target cloud storage device is updated to mount failure, a copy of the second multi-copy partition exists in the target cloud storage device, and a copy of the second multi-copy partition does not exist in the cloud storage device mounted on the first node, a new slave node and a new slave cloud storage device corresponding to the second multi-copy partition are determined, and the new slave cloud storage device is mounted on the new slave node; when the new slave node is the first node, a slave copy of the second multi-copy partition is added to the new slave cloud storage device, and the data in the master copy of the second multi-copy partition is copied to the newly added slave copy, so as to realize the migration of the copy of the second multi-copy partition in the target cloud storage device.

[0049] Among them, the slave node corresponding to the multi-replica partition refers to the node where the cloud storage device to which the slave copy of the multi-replica partition belongs is mounted, and each multi-replica partition can correspond to one or more slave nodes. The new slave node and new slave cloud storage device corresponding to the second multi-replica partition can be configured by the user in the management node, or automatically determined by the management node, or elected by each node in the node cluster that has not failed. The new slave cloud storage device can be any cloud storage device in the new slave node. There is no copy of the second multi-replica partition in each cloud storage device mounted on the new slave node.

[0050] Specifically, the master copy of the second multi-copy partition is the new master copy of the second multi-copy partition or the original master copy of the second multi-copy partition in the non-failed node. After adding a slave copy in the new slave cloud storage device, the first node can update the metadata of the cloud storage device. Taking the second multi-copy partition as multi-copy partition 4 and the first node as node 3 as an example, there is a copy R4-1 of multi-copy partition 4 in cloud storage device 1 of node 1. Since there is no copy of the multi-copy partition in node 3, if the new slave node is node 3 and the new slave cloud storage device is cloud storage device 4, a copy R4-1 of multi-copy partition 4 is added to cloud storage device 4 (which can be understood as creating a file), and the data in the master copy of multi-copy partition 4 is copied to the newly added copy R4-1 (which can be understood as writing data to the file). Since there is a master copy of multi-copy partition 4 in node 2, the data in the master copy of multi-copy partition 4 in the node can be copied. If there is a slave copy of multi-copy partition 4 in node 2, the slave copy of multi-copy partition 4 in node 2 is changed to the master copy and then copied.

[0051] In this embodiment, the migration of the replica is achieved by replication, thereby ensuring the normal use of the replica.

[0052] In some embodiments, the method also includes: when the determined node is a third node, obtaining the mounting status of the target cloud storage device from the management node, and the mounting status is updated by the third node; when the mounting status is mounting failure, there is a master copy of the first multi-copy partition in the target cloud storage device, and there is a slave copy of the first multi-copy partition in the cloud storage device mounted by the first node, then determining a new master node corresponding to the first multi-copy partition; when the determined new master node is the first node, sending a master-slave change request for the first multi-copy partition to the management node, so that the management node changes the slave copy of the first multi-copy partition in the cloud storage device mounted by the first node to the master copy of the first multi-copy partition.

[0053] Specifically, when the node determined to be used to remount the target cloud storage device is the third node, the third node triggers mounting and updates the mounting state. The first node can monitor the change of the mounting state, and when the mounting state is mounting failure, there is a master copy of the first multi-copy partition in the target cloud storage device, and there is a slave copy of the first multi-copy partition in the cloud storage device mounted by the first node, a master-slave change request is sent to the management node to change the master copy.

[0054] In this embodiment, since the third node triggers the update of the mount status of the target cloud storage device, other nodes can promptly know whether the mount is successful, and make corresponding operations such as master-slave changes in a timely manner in case of mount failure to ensure the normal use of the partition.

[0055] In some embodiments, when the second node returns to normal, the second node sends a device recovery request to the cloud server so that the cloud server uninstalls the target cloud storage device from the first node and remounts the target cloud storage device to the second node.

[0056] Specifically, the management node may record the identifiers of each cloud storage device mounted by the second node before the failure (i.e., the identifier of the target cloud storage device). After the second node returns to normal, the second node may query the identifiers of each cloud storage device mounted before the failure from the management node, thereby determining that each cloud storage device mounted before the failure is each target cloud storage device. The second node may send a device recovery request, which may carry the identifier of the target cloud storage device. In response to the device recovery request, the management node remounts the target cloud storage device to the second node, thereby quickly completing the replica migration again.

[0057] In this embodiment, after the target cloud storage device is mounted to the first node, the copy in the target cloud storage device can read and write or synchronize data normally (synchronize data from the copy to the master copy). Therefore, when the second node returns to normal and the target cloud storage device is remounted to the second node, the second node does not need to copy a large amount of data to realize data synchronization in the copy, thereby improving the efficiency of copy migration.

[0058] In some embodiments, determining a node for remounting a target cloud storage device includes: obtaining load information of nodes that have not failed in a node cluster; and determining a node for remounting the target cloud storage device from nodes that have not failed based on the load information.

[0059] The node that has not failed is, for example, a node that is online. The load information of the node includes, but is not limited to, at least one of the number of cloud storage devices mounted on the node or the number of partitions allocated to the node.

[0060] Specifically, the first node may determine the load size according to the load information, and preferentially determine nodes with smaller loads as nodes for remounting the target cloud storage device.

[0061] In some embodiments, when multiple cloud storage devices are mounted on the second node, that is, when there are multiple target cloud storage devices, each node in the node cluster that has not failed can determine the node used to remount each target cloud storage device. The nodes determined for different target cloud storage devices can be the same node or different nodes. For example, there are two target cloud storage devices, namely device 1 and device 2, and it is determined that device 1 is remounted to the first node and device 3 is remounted to the third node, which is different from the first node and the third node is not failed; or, it is determined that device 1 and device 2 are remounted to the third node or the first node.

[0062] In some embodiments, at most one target cloud storage device is mounted on each surviving node.

[0063] In this embodiment, the node used to remount the target cloud storage device is determined according to the load information, so that the load balancing degree of each node in the node cluster can be improved.

[0064] In some embodiments, Figure 4 As shown, a schematic diagram of a Kafka partition replica migration method is shown, including: when a node crash is detected, query the crash node block device (the OpenAPI of the block device can be called to query the crash node block device), calculate the block device allocation plan based on the load of other available nodes in the cluster, uninstall the crash node block device, and mount it to other nodes (the cloud OpenAPI can be called to mount), if successful, rebuild the partition metadata based on the state of the block device mount, otherwise, migrate and restore data by replication. OpenAPI is the abbreviation of Open Application Programming Interface. Block device refers to cloud storage device.

[0065] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0066] Based on the same inventive concept, the embodiment of the present application also provides a Kafka partition replica migration device for implementing the Kafka partition replica migration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more Kafka partition replica migration device embodiments provided below can refer to the limitations of the Kafka partition replica migration method above, and will not be repeated here.

[0067] In some embodiments, Figure 5 As shown, a Kafka partition replica migration device is provided, each node in the node cluster includes a device, each node in the node cluster is deployed with a Kafka service and mounted with a cloud storage device, and a partition replica is stored in the cloud storage device, and the device in the first node in the node cluster includes: a device determination module 502, a mount request sending module 504 and a metadata update module 506, wherein:

[0068] The device determination module 502 is used to determine the target cloud storage device mounted on the second node in the node cluster and determine the node for remounting the target cloud storage device when it is determined that the second node in the node cluster fails.

[0069] The mount request sending module 504 is used to send a device mount request for the target cloud storage device to the cloud server when the determined node is the first node, so that the cloud server mounts the target cloud storage device to the first node after uninstalling the target cloud storage device from the second node.

[0070] The metadata update module 506 is used to determine the copy in the target cloud storage device as the target copy, determine the target partition to which the target copy belongs, and update the metadata of the target partition to migrate the target copy of the target partition from the second node to the first node when the mounting is successful.

[0071] In some embodiments, the partition is divided into a single-copy partition and a multi-copy partition, the single-copy partition has one and only one copy, the multi-copy partition has at least two copies, and there is a master copy and at least one slave copy in the at least two copies. The device method also includes a mount failure processing device, the mount failure processing device is used to send a mount failure notification for the target cloud storage device to the management node in the case of a mount failure, so that the management node updates the mount status of the target cloud storage device to a mount failure; when the mount status of the target cloud storage device is updated to a mount failure, there is a master copy of the first multi-copy partition in the target cloud storage device, and there is a slave copy of the first multi-copy partition in the cloud storage device mounted by the first node, then determine a new master node corresponding to the first multi-copy partition; when the determined new master node is the first node, send a master-slave change request for the first multi-copy partition to the management node, so that the management node changes the slave copy of the first multi-copy partition in the cloud storage device mounted by the first node to the master copy of the first multi-copy partition.

[0072] In some embodiments, the mount failure processing device is also used to determine a new slave node and a new slave cloud storage device corresponding to the second multi-copy partition when the mount status of the target cloud storage device is updated to mount failure, a copy of the second multi-copy partition exists in the target cloud storage device, and the cloud storage device mounted on the first node does not have a copy of the second multi-copy partition, and the new slave cloud storage device is mounted on the new slave node; when the new slave node is the first node, a slave copy of the second multi-copy partition is added to the new slave cloud storage device, and the data in the master copy of the second multi-copy partition is copied to the newly added slave copy, so as to realize the migration of the copy of the second multi-copy partition in the target cloud storage device.

[0073] In some embodiments, the mount failure processing device is also used to obtain the mount status of the target cloud storage device from the management node when the determined node is a third node, and the mount status is updated by the third node; when the mount status is mount failure, there is a master copy of the first multi-copy partition in the target cloud storage device, and there is a slave copy of the first multi-copy partition in the cloud storage device mounted by the first node, then determine the new master node corresponding to the first multi-copy partition; when the determined new master node is the first node, send a master-slave change request for the first multi-copy partition to the management node, so that the management node changes the slave copy of the first multi-copy partition in the cloud storage device mounted by the first node to the master copy of the first multi-copy partition.

[0074] In some embodiments, when the second node returns to normal, the second node sends a device recovery request to the cloud server so that the cloud server uninstalls the target cloud storage device from the first node and remounts the target cloud storage device to the second node.

[0075] In some embodiments, the device determination module 502 is further used to obtain load information of nodes that have not failed in the node cluster; and determine the node for remounting the target cloud storage device from the nodes that have not failed based on the load information.

[0076] Each module in the above-mentioned Kafka partition replica migration device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0077] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the Kafka partition replica migration method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a Kafka partition replica migration method is implemented.

[0078] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a kafka partition replica migration method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0079] Those skilled in the art will understand that Figure 6 and Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0080] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps in the above-mentioned Kafka partition replica migration method are implemented.

[0081] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned Kafka partition replica migration method are implemented.

[0082] In some embodiments, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps in the above-mentioned Kafka partition replica migration method.

[0083] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0084] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0085] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0086] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A Kafka partition replica migration method, characterized in that: Applied to the first node in a node cluster, each node in the node cluster is deployed with a Kafka service and mounted with a cloud storage device, and the cloud storage device stores a copy of the partition, the method includes: In the case of determining that a second node in the node cluster fails, determining a target cloud storage device mounted on the second node, and determining a node for remounting the target cloud storage device; In the case where the determined node is the first node, sending a device mount request for the target cloud storage device to the cloud server, so that the cloud server mounts the target cloud storage device to the first node after uninstalling the target cloud storage device from the second node; If the mounting is successful, the copy in the target cloud storage device is determined as the target copy, the target partition to which the target copy belongs is determined, and the metadata of the target partition is updated to migrate the target copy of the target partition from the second node to the first node.

2. The method according to claim 1, characterized in that The partition is divided into a single-copy partition and a multi-copy partition, the single-copy partition has and only has one copy, the multi-copy partition has at least two copies, and there is a master copy and at least one slave copy in the at least two copies, and the method further includes: In the case of a mount failure, sending a mount failure notification for the target cloud storage device to the management node, so that the management node updates the mount status of the target cloud storage device to mount failure; In the case where the mount status of the target cloud storage device is updated to mount failure, a master copy of the first multi-copy partition exists in the target cloud storage device, and a slave copy of the first multi-copy partition exists in the cloud storage device mounted by the first node, a new master node corresponding to the first multi-copy partition is determined; When the new master node determined to be the first node is sent, a master-slave change request for the first multi-replica partition is sent to the management node, so that the management node changes the slave replica of the first multi-replica partition in the cloud storage device mounted by the first node to the master replica of the first multi-replica partition.

3. The method according to claim 2, characterized in that The method further comprises: When the mount state of the target cloud storage device is updated to mount failure, a copy of the second multi-copy partition exists in the target cloud storage device, and the cloud storage device mounted by the first node does not have a copy of the second multi-copy partition, determining a new slave node and a new slave cloud storage device corresponding to the second multi-copy partition, the new slave cloud storage device being mounted on the new slave node; In the case that the new slave node is the first node, a slave replica of the second multi-replica partition is added in the new slave cloud storage device, and the data in the master replica of the second multi-replica partition is copied to the newly added slave replica to achieve migration of the replica of the second multi-replica partition in the target cloud storage device.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: In the case where the determined node is a third node, obtaining a mount status of the target cloud storage device from a management node, wherein the mount status is updated by triggering the third node; In the case where the mount status is mount failure, a primary copy of the first multi-copy partition exists in the target cloud storage device, and a secondary copy of the first multi-copy partition exists in the cloud storage device mounted by the first node, determining a new primary node corresponding to the first multi-copy partition; When the new master node determined to be the first node is sent, a master-slave change request for the first multi-replica partition is sent to the management node, so that the management node changes the slave replica of the first multi-replica partition in the cloud storage device mounted by the first node to the master replica of the first multi-replica partition.

5. The method according to any one of claims 1 to 3, characterized in that: When the second node returns to normal, the second node sends a device recovery request to the cloud server, so that the cloud server uninstalls the target cloud storage device from the first node and remounts the target cloud storage device to the second node.

6. The method according to any one of claims 1 to 3, characterized in that: The determining of a node for remounting the target cloud storage device includes: Obtaining load information of nodes that have not failed in the node cluster; According to the load information, a node for remounting the target cloud storage device is determined from nodes that have not failed.

7. A Kafka partition replica migration device, characterized in that: Each node in the node cluster includes the device, each node in the node cluster is deployed with a kafka service and mounted with a cloud storage device, the cloud storage device stores a copy of the partition, and the device in the first node in the node cluster includes: A device determination module, configured to determine a target cloud storage device mounted on a second node in the node cluster and determine a node for remounting the target cloud storage device when it is determined that the second node in the node cluster fails; a mount request sending module, configured to send a device mount request for the target cloud storage device to the cloud server when the determined node is the first node, so that the cloud server mounts the target cloud storage device to the first node after uninstalling the target cloud storage device from the second node; The metadata update module is used to determine the copy in the target cloud storage device as the target copy when the mounting is successful, determine the target partition to which the target copy belongs, and update the metadata of the target partition to migrate the target copy of the target partition from the second node to the first node.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images