Master-slave switching method, device, storage medium and distributed message queue system

Abstract

This application provides a method, device, storage medium, and distributed message queue system for master / replica switching. In this method, for a target partition under a target topic, when changing its corresponding master replica from a second service node to a first service node, the term of office of the master replica is used to identify the differences between the old and new master replicas. The master replica with the larger term value is the newly determined master replica for the target partition, i.e., the master replica that needs to take over the partition data operation permissions of the target partition. Then, the first service node, as the new master replica, changes the partition data operation permissions from the second service node to the first service node using its corresponding term value. Finally, the first service node changes its own replica identifier to a master replica identifier to indicate to the client that the message read / write service corresponding to the target partition is provided by the first service node, thereby achieving efficient and accurate switching between the old and new master replicas.

Description

Technical Field

[0001] This application relates to the field of message queue technology, and in particular to a master-replica switching method, device, storage medium, and distributed message queue system. Background Technology

[0002] In distributed systems, message queues are an infrastructure used for asynchronous message passing between different services or components; they are often also called message middleware. In message queues such as Kafka, message categories are organized by topics. Each topic can be further divided into one or more partitions, and each partition contains a series of ordered messages. Topics logically categorize messages, while partitions physically store them in different storage spaces. For example, messages 1 and 2, belonging to the same topic x, would be stored in storage space 1 corresponding to partition 1 of topic x, and message 2 in storage space 2 corresponding to partition 2 of topic x, and so on. Specifically, messages are stored in log files within the storage space.

[0003] To achieve high availability and fault tolerance, for a given partition, one service node can be elected from a group of service nodes (Brokers) in the cluster running the message queue as the leader replica for that partition, and several other service nodes can be elected as different follower replicas for that partition. The leader replica handles all message read and write requests for that partition, while the follower replicas become the new leader replicas and handle all message read and write requests for that partition when the leader replica is shut down or fails.

[0004] In practical applications, under a storage-compute separation architecture, message queues store only one copy of the partition data in a distributed file system for each partition. This partition data includes a log file used to store messages. The primary replica handles all message read and write requests for that partition by holding the operation permissions for that partition data and performing read and write operations on the log file. When the primary replica changes, for example, when the primary replica shuts down or crashes, other secondary replicas are elected as the new primary replica. The new primary replica needs to take over the operation permissions for the partition data held by the old primary replica and handle all message read and write requests for that partition. However, existing takeover schemes suffer from problems such as long takeover times and failure to take over. Summary of the Invention

[0005] This application provides a primary replica switching method, device, storage medium, program product, and distributed message queue system to improve the efficiency and success rate of a new primary replica obtaining operation permissions for partition data of a partition, and to ensure efficient and accurate switching between the old and new primary replicas.

[0006] In a first aspect, embodiments of this application provide a primary replica switching method applied to a first service node in a distributed computing cluster. The distributed computing cluster further includes a second service node and a control node. The second service node is the primary replica for the first term corresponding to a target partition under a target topic. The distributed computing cluster is interconnected with a distributed file system storing partition data corresponding to the target partition. The method includes:

[0007] In response to the primary replica change notification of the target partition sent by the control node, it is determined that the first service node is changed from a secondary replica of the target partition to a primary replica of the target partition for the second term, wherein the first term value corresponding to the first term is less than the second term value corresponding to the second term.

[0008] By using the second term value, the operation permission for the partition data corresponding to the target partition in the distributed file system is changed from the second service node to the first service node;

[0009] Change the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier. The primary replica identifier is used to identify to the client that the read and write service for the message corresponding to the target partition is provided by the first service node.

[0010] Secondly, embodiments of this application provide a primary replica switching device applied to a first service node in a distributed computing cluster, wherein the distributed computing cluster further includes a second service node and a control node, the second service node being the primary replica of a target partition under a target topic for a first term, and the device comprising:

[0011] The determination module is configured to, in response to a primary replica change notification for the target partition sent by the control node, determine that the first service node has been changed from a secondary replica of the target partition to a primary replica of the target partition for the second term, wherein the first term value corresponding to the first term is less than the second term value corresponding to the second term.

[0012] The sending module is used to change the operation permission of the partition data corresponding to the target partition in the distributed file system from the second service node to the first service node through the second term value;

[0013] The processing module is used to change the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier. The primary replica identifier is used to indicate to the client that the read and write service for the message corresponding to the target partition is provided by the first service node.

[0014] Thirdly, embodiments of this application provide a distributed message queue system, including: an interconnected distributed file system and a distributed computing cluster;

[0015] The distributed file system is used to store partition data corresponding to the target partition under the target topic;

[0016] The distributed computing cluster includes: a first service node, a second service node, and a control node, wherein the second service node is the primary replica corresponding to the first term of the target partition;

[0017] The control node is used to send a primary replica change notification for the target partition to the first service node;

[0018] The first service node is configured to, based on the primary replica change notification, determine that it is changed from a secondary replica of the target partition to a primary replica of the target partition for the second term, wherein the first term value corresponding to the first term is less than the second term value corresponding to the second term; and, through the second term value, change the operation permissions of the partition data corresponding to the target partition in the distributed file system from the second service node to the first service node; and change the replica identifier corresponding to the first service node from a secondary replica identifier to a primary replica identifier, wherein the primary replica identifier is used to indicate to the client that the read and write services for messages corresponding to the target partition are provided by the first service node.

[0019] Fourthly, embodiments of this application provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores a computer program, and when the computer program is executed by the processor, the processor can at least implement the master-slave switching method as described in the first aspect.

[0020] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor of an electronic device, enables the processor to at least implement the master-replica switching method as described in the first aspect.

[0021] In a sixth aspect, embodiments of this application provide a computer program product, including: a computer program or instructions that, when executed by a processor of an electronic device, enable the processor to at least implement the master-slave switching method as described in the first aspect.

[0022] In the solution provided in this application embodiment, for a target partition under a target topic, in the scenario where its corresponding primary replica is changed from a second service node to a first service node, on the one hand, the difference between the old and new primary replicas is identified by the term of office corresponding to the primary replica. The primary replica with the larger term value is the newly determined primary replica for the target partition, that is, the primary replica that needs to take over the partition data operation permissions of the target partition. On the other hand, based on the difference in the term values ​​corresponding to the primary replicas of the target partition, corresponding operation permissions are set for the partition data of the target partition in the distributed file system storing the partition data of the target partition. This restricts the operation of the partition data of the target partition by the old and new primary replicas, enabling the new primary replica to change the operation permissions of the partition data corresponding to the target partition from the old primary replica to the new primary replica based on its own term value, that is, to acquire the operation permissions of the partition data, thus achieving efficient and accurate conversion of the operation permissions of the partition data of the target partition between the old and new primary replicas.

[0023] Based on the term of the primary replica of the target partition and the configuration of operation permissions for the partition data in the distributed file system, in the scenario where operation permissions for partition data are transferred between the old and new primary replicas: Assume the current primary replica of the target partition is the second service node in the distributed computing cluster, and the term of the second service node is the first term. Based on this assumption, in response to the primary replica change notification for the target partition sent by the control node, the first service node first determines that it has changed from a secondary replica of the target partition to the primary replica of the target partition corresponding to the second term, where the first term value is less than the second term value. Then, using the second term value, it changes the operation permissions for the partition data corresponding to the target partition in the distributed file system from the second service node to the first service node. Finally, it determines that the first service node has the operation permissions to perform file meta-operations on the partition data of the target partition in the distributed file system, and changes the replica identifier of the first service node from a secondary replica identifier to a primary replica identifier. The primary replica identifier is used to identify to clients that the read and write services for messages corresponding to the target partition are provided by the first service node.

[0024] In summary, in this embodiment, for a target partition under a target topic, by determining the term value of the primary replica of the target partition and changing the partition data operation permissions in the distributed file system based on the new primary replica's term value, the new primary replica can obtain operation permissions for the partition data of the target partition, achieving efficient and accurate conversion of the partition data operation permissions between the old and new primary replicas. Furthermore, after obtaining operation permissions for the partition data of the target partition, the new primary replica indicates to the client that the read and write services for the messages corresponding to the target partition are provided by the new primary replica by changing the replica identifier to the primary replica identifier, thereby completing the efficient and accurate switch between the old and new primary replicas. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram of a distributed publish-subscribe message queue integrating in-memory computing provided in an embodiment of this application;

[0027] Figure 2 A schematic diagram of a distributed publish-subscribe message queue with storage-compute separation provided in an embodiment of this application;

[0028] Figure 3 A flowchart illustrating a primary / replica switching method provided in this application embodiment;

[0029] Figure 4 This is a schematic diagram of a distributed message queue system provided in an embodiment of this application;

[0030] Figure 5 A flowchart illustrating a method for changing the replica identifier of a first service node, provided as an embodiment of this application;

[0031] Figure 6 A schematic diagram illustrating the process of changing the replica identifier of a first service node, as provided in an embodiment of this application;

[0032] Figure 7 A flowchart illustrating another method for changing the replica identifier of a first service node, provided in an embodiment of this application;

[0033] Figure 8 A schematic diagram illustrating another process for changing the replica identifier of a first service node, provided in an embodiment of this application;

[0034] Figure 9 A schematic diagram of a master-replica switching device provided in an embodiment of this application;

[0035] Figure 10 To and Figure 9 The illustrated embodiment provides a schematic diagram of the electronic device corresponding to the master / replica switching device. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] It should be noted that, in the cases involving user information in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0038] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0039] To facilitate understanding, before introducing the master-replica switching method provided in the embodiments of this application, we will first briefly explain the message queues of in-memory computing integration and in-memory computing separation.

[0040] Let's take Kafka, a message queue, as an example to illustrate this.

[0041] Kafka, as a high-throughput distributed publish-subscribe message queue, can, on the one hand, receive messages on a specific topic sent by clients acting as message publishers and store the messages in the partitions corresponding to that specific topic; on the other hand, it can read messages on a specific topic from the partitions corresponding to that specific topic based on the subscription information sent by clients acting as message subscribers and then send those messages back to the subscribers.

[0042] In the aforementioned distributed publish-subscribe message queue, topics are used to logically divide messages into different classes, and partitions are used to physically store messages in different storage spaces. For example, for messages 1 and 2 that belong to the same topic x, message 1 is stored in storage space 1 corresponding to partition 1 of topic x, and message 2 is stored in storage space 2 corresponding to partition 2 of topic x, and so on.

[0043] In practical applications, when a message for a topic is stored in the storage space of the corresponding partition, it is usually written to the log file of that partition in an append-only manner. Different partitions correspond to different log files.

[0044] It's worth noting that, to improve the efficiency of log file management and maintenance, the log file for each partition can optionally be divided into multiple log segment files. In other words, the log file for each partition consists of a series of log segment files. Each log segment file typically contains: a log segment data file (.log file), an index file (.index file), and a timestamp index file (.time index file). The log segment data file stores message data for the corresponding topic; specifically, each message is appended sequentially to the end of the log segment data file. The index file and timestamp index file are used to quickly locate and find the position of a message within the log segment data file. In practice, when a log segment file reaches a certain size limit or after a certain period of time, a new log segment file can be created to continue writing new messages. By dividing the log file for a partition into multiple smaller log segment files, it becomes easier to perform deletion or archiving operations on older message data.

[0045] In this embodiment of the application, no distinction is made between log files and log segment files that constitute log files; they are all referred to as log files.

[0046] For ease of description, in this embodiment of the application, the data stored in the corresponding storage space of a certain partition is collectively referred to as partition data, which includes the log file corresponding to the partition.

[0047] To ensure the reliability and fault tolerance of the distributed publish-subscribe message queue, each partition has its own replica set, which typically contains one leader replica and several (e.g., one or more) follower replicas. The leader replica handles all message read and write requests for that partition, while the follower replicas act as the new leader replica when the leader replica is unavailable or malfunctions, handling all message read and write requests for that partition.

[0048] In this context, the primary and secondary replicas of the same partition are different service nodes within a group of brokers running the Kafka cluster. For example, assuming a Kafka cluster has three service nodes: Broker1, Broker2, and Broker3, then for partition 1 corresponding to a given topic, Broker1 can be used as the primary replica of partition 1, and Broker2 and Broker3 can be used as different secondary replicas of partition 1, and so on. In practical applications, a topic can have multiple different partitions, and a single service node can simultaneously serve as either a primary or secondary replica of different partitions. The same partition can also have different service nodes selected as both primary and secondary replicas.

[0049] In this embodiment of the application, the service node may be a physical server containing an independent host, or it may be a virtual server or a cloud server.

[0050] Figure 1 This is a schematic diagram of a distributed publish-subscribe message queue that integrates in-memory computing, as provided in an embodiment of this application. Figure 1 As shown, a Kafka cluster with three service nodes, Broker1, Broker2, and Broker3, is used as an example. Partition 1 has three replicas, with Broker1 serving as the primary replica of partition 1 and Broker2 and Broker3 serving as different secondary replicas of partition 1.

[0051] In a compute-in-store architecture, each service node integrates computation and storage, handling both message reading / writing and message storage. When storing messages, both the primary and secondary replicas of the corresponding partition store the messages for that partition. Figure 1 As shown, Broker1, Broker2, and Broker3 all store partition 1 data on their local disks. In the actual storage process, Broker1 acts as the primary replica, responsible for all message reads and writes for partition 1. Messages corresponding to partition 1 are first stored on Broker1's local disk. Then, Broker2 and Broker3 synchronize the partition 1 data stored on Broker1's local disk to their local disks through communication with Broker1.

[0052] It is easy to understand that, in Figure 1In the illustrated in-memory computing architecture, each replica of partition 1 (including the primary replica and the secondary replica) holds the same partition 1 data. Therefore, when the primary replica of partition 1 changes, for example, when the primary replica of partition 1 changes from Broker1 to Broker2, Broker2 can directly read and write the messages corresponding to partition 1 based on the partition 1 data stored on its own local disk. The operation is simple and efficient.

[0053] However, in practical applications, in-memory computing architectures suffer from problems such as weak ability of a single hard drive to handle hot data, limited storage capacity, binding of storage and computing ratios, low storage resource utilization, and slow scaling efficiency. Based on this, a distributed publish-subscribe message queue with storage and computing separation is proposed. By offloading the storage of partition data (i.e., the messages corresponding to each partition) to a distributed file system (DFS), the actual storage volume of partition data for each partition is optimized from multiple copies to a single copy.

[0054] Figure 2 This is a schematic diagram of a distributed publish-subscribe message queue with storage-compute separation provided in an embodiment of this application. Figure 2 As shown, the system includes a distributed computing cluster and a distributed file system.

[0055] In this distributed computing cluster, service nodes such as Broker1, Broker2, and Broker3 handle computation-related data processing tasks. For example, they might receive a message from a client and store it in the corresponding partition of the distributed file system; or they might receive a message consumption request from a client, retrieve the message from the partition data stored in the distributed file system, and return it to the client. The distributed file system stores the messages corresponding to each partition.

[0056] It is worth noting that, Figure 1 and Figure 2 The diagram only illustrates the storage result of the message corresponding to partition 1—partition 1 data. In actual applications, a topic often corresponds to more than one partition. Figure 1 Local disks of service nodes or Figure 2 The distributed file systems in the system can all store partition data from different partitions.

[0057] exist Figure 2In the illustrated storage-compute separation architecture, each partition stores only one copy of its data in the distributed file system. This means that multiple replicas of each partition (including primary and secondary replicas) share the same stored partition data. To ensure correct reading and writing of messages corresponding to a partition, the primary replica performs read and write operations on partition 1 by holding the necessary permissions for that partition's data, thus handling all message read and write requests for partition 1. When the primary replica changes, for example, when the primary replica of partition 1 is changed from Broker1 to Broker2, Broker2, as the new primary replica, takes over the access permissions for partition 1 held by the old primary replica Broker1, performing read and write operations on partition 1's data, and continuing to handle all message read and write requests for partition 1.

[0058] As described above, in a compute-storage separated architecture, for a target partition under a target topic, the ability of the new primary replica to quickly and accurately take over the operation permissions of the target partition's partition data held by the old primary replica is crucial to ensuring normal message read and write operations. Therefore, this application provides a primary replica switching method to improve the efficiency and success rate of the new primary replica acquiring operation permissions for the target partition's partition data in a compute-storage separated architecture.

[0059] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0060] Figure 3 A flowchart of a primary replica switching method provided in this application embodiment is applied to a first service node in a distributed computing cluster. The distributed computing cluster also includes a second service node and a control node. The second service node is the primary replica for the first term corresponding to the target partition under the target topic. The distributed computing cluster is interconnected with a distributed file system storing partition data corresponding to the target partition. Figure 3 As shown, it may include the following steps:

[0061] 301. In response to the primary replica change notification of the target partition sent by the control node, determine that the first service node is changed from the secondary replica of the target partition to the primary replica of the target partition in the second term, and the first term value corresponding to the first term is less than the second term value corresponding to the second term.

[0062] 302. By using the second term value, change the operation permission of the partition data corresponding to the target partition in the distributed file system from the second service node to the first service node.

[0063] 303. Change the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier. The primary replica identifier is used to identify to the client that the read and write service of the message corresponding to the target partition is provided by the first service node.

[0064] In summary, for a target partition under a target theme, in a scenario where its corresponding primary replica is changed from the second service node to the first service node, to achieve efficient and accurate transfer of partition data operation permissions between the old and new primary replicas, the differences between the old and new primary replicas are identified by the term of office of the primary replicas. Specifically, the primary replica with the larger term value (epoch) is the newly determined primary replica for the target partition, that is, the primary replica that needs to take over the partition data operation permissions for the target partition. In other words, for different primary replicas corresponding to the same partition, the term value of the corresponding term monotonically increases according to the order in which they are determined (or elected).

[0065] For example, suppose a distributed computing cluster contains three service nodes: Broker1, Broker2, and Broker3. At time T0, Broker1 is determined as the primary replica of the target partition, with a term value of epoch = 1. Broker2 and Broker3 are both secondary replicas of the target partition. At time T1, suppose Broker1 can no longer serve as the primary replica of the target partition due to reasons such as normal shutdown or abnormal closure. In this case, a new primary replica of the target partition needs to be determined from the secondary replicas Broker2 and Broker3. Assuming Broker2 is determined as the new primary replica, then the term value of Broker2 is epoch = 2.

[0066] By analogy, the target partition can continuously determine new primary replicas from the service nodes within the distributed computing cluster, and the tenure value corresponding to the new primary replica is continuously incremented. Therefore, based on the tenure value of the primary replica, it is easy and direct to determine which service node in the cluster is the latest primary replica for the target partition, used to provide message read and write services.

[0067] On the other hand, based on the monotonically increasing term value of the primary replica corresponding to the target partition, a corresponding operation permission acquisition mechanism is set up for the target partition's partition data in the distributed file system storing the target partition's partition data. Thus, the primary replica with a larger term value can acquire the operation permissions corresponding to the target partition's partition data through its own term value, effectively changing the operation permissions for the target partition's partition data from the primary replica with the smaller term value to the primary replica with the larger term value.

[0068] As an optional method for determining the operation permissions corresponding to the partition data of the target partition based on the term value, a corresponding operation permission lock can be set for the partition data of the target partition in the distributed file system storing the partition data of the target partition. This operation permission lock matches the term value corresponding to the primary replica of the target partition. Specifically, this operation permission lock has at least the following two characteristics:

[0069] First, regarding the target partition, if the lock value carried in the file meta-operation request sent by the primary replica for the target partition's partition data during a certain term matches the current lock value of that operation permission lock in the distributed file system, then the distributed file system determines that the primary replica during that term has the permission to perform file meta-operations on the target partition's partition data; that is, it determines that the primary replica during that term has the permission to operate on the target partition's partition data. If the lock value carried in the file meta-operation request sent by the primary replica for the target partition's partition data during a certain term does not match the current lock value of that operation permission lock in the distributed file system, then the distributed file system determines that the primary replica during that term does not have the permission to perform file meta-operations on the target partition's partition data.

[0070] The lock value carried in the file meta-operation request is the tenure value of the primary replica that sent the file meta-operation request.

[0071] File metadata operations refer to operations on metadata. In this embodiment, file metadata operations on the partition data of the target partition include, but are not limited to: opening / deleting log files contained in the partition data, creating new log files in the storage space where the partition data is located (i.e., in the storage space corresponding to the target partition), etc. The implementation of message reading and writing of the target partition depends on the opening of the log files in the partition data. For example, messages can only be written to or read from the log files after the log files are opened.

[0072] Second, in a distributed file system, the lock value of the partition data operation permission lock for the target partition is updated based on the term value of the primary replica's term.

[0073] Specifically, for a target partition, the primary replica of that partition, during a given term, can send a lock update request to the distributed file system. This lock update request is used to request the distributed file system to update the current lock value of the partition data operation permission lock corresponding to the target partition to the lock value included in the lock update request. The lock value included in the lock update request is the term value of the primary replica that sent the lock update request.

[0074] When a distributed file system receives a lock update request, if it determines that the lock value included in the lock update request is greater than the current lock value of the partition data operation permission lock corresponding to the target partition, then it updates the lock value of the partition data operation permission lock to the lock value included in the lock update request; if it determines that the lock value included in the lock update request is less than or equal to the current lock value of the partition data operation permission lock corresponding to the target partition, then it keeps the current lock value of the partition data operation permission lock corresponding to the target partition unchanged.

[0075] It's easy to understand that in a scenario where the primary replica of a target partition is changed from an old primary replica to a new primary replica, the new primary replica can send a lock value change request to the distributed file system to change the lock value of the partition data operation permission lock of the target partition in the distributed file system to a larger value (i.e., the tenure value of the new primary replica). This makes the old primary replica no longer have the metadata operation permission of the target partition's partition data, while the new primary replica has the metadata operation permission of the target partition's partition data, thus achieving an efficient and accurate conversion of the partition data operation permission of the target partition between the old and new primary replicas.

[0076] For ease of understanding, combined with Figure 4 The primary / replica switching method provided in the embodiments of this application will be described by way of example, but it is not intended to be used in practice. Figure 4 The situation indicated is limited to the cases described.

[0077] Figure 4 This is a schematic diagram illustrating a scenario of a distributed message queue system provided in an embodiment of this application. Figure 4 As shown, the distributed message queue system is a storage-compute separation architecture, which includes an interconnected distributed computing cluster and a distributed file storage system. The distributed file system is used to store the partition data corresponding to the target partition under the target topic, and the partition data of the target partition is stored in the target storage space in the distributed file system. The distributed computing cluster includes a first service node, a second service node, and a control node.

[0078] The control node can be any service node in the distributed computing cluster. Figure 4 The following example illustrates the process using the control node as the first service node. In a distributed message queue system, the control node is used to determine or elect a new primary replica from one or more primary replicas of the target partition when the primary replica of the target partition needs to be changed. The election strategy can be found in relevant technologies and will not be elaborated in this embodiment.

[0079] In practice, the control node can determine whether to elect a new primary replica for the target partition by obtaining the current status of the primary replica. For example, if the control node determines that the current primary replica of the target partition needs to be taken offline, has crashed abnormally, or has become unresponsive (commonly known as hanging), then it will elect a new primary replica for the target partition. Taking offline refers to the process where a service node stops providing service due to planned maintenance, failure, or other reasons.

[0080] In this embodiment, it is assumed that before the primary replica of the target partition is changed, its primary replica is the second service node, corresponding to the first term, and the specific term value is the first term value m. When it is necessary to change the primary replica of the target partition, the control node elects a new primary replica of the target partition as the first service node, corresponding to the second term, and the specific term value is the second term value m+1. Wherein, m is a positive integer.

[0081] Based on the above introduction to partition data operation permission locks, it is easy to understand that during the period when the second service node acts as the primary replica of the target partition, the second service node has the permission to perform file meta-operations on the partition data of the target partition. At this time, the lock value of the partition data operation permission lock corresponding to the target partition is the first term value m.

[0082] When the primary replica of the target partition is changed to the first service node, the lock value of the partition data operation permission lock corresponding to the target partition needs to be changed from the first term value m to the second term value m+1. This makes the second service node no longer have the permission to perform file meta-operations on the partition data of the target partition, while the first service node has the permission to perform file meta-operations on the partition data of the target partition, so that the reading and writing of messages can be completed through the first service node.

[0083] In the specific implementation process, in response to the target partition primary replica change notification sent by the control node, the first service node first determines that the first service node has changed from a secondary replica of the target partition to a primary replica of the target partition, and the term value corresponding to the primary replica (i.e., the second term value m+1).

[0084] Subsequently, the first service node sends a lock update request containing the second term value m+1 to the distributed file system. The lock update request contains the second term value m+1. This lock update request is used to request the distributed file system to update the current lock value (i.e., the first term value m) of the partition data operation permission lock corresponding to the target partition to the lock value (i.e., the second term value m+1) contained in the lock update request when it determines that the second term value m+1 is greater than the current lock value (i.e., the first term value m) of the partition data operation permission lock corresponding to the target partition.

[0085] After receiving a lock update request, the distributed file system determines that the lock value included in the update request—the second term value m+1—is greater than the current lock value of the partition data operation permission lock corresponding to the target partition—the first term value m. Furthermore, it updates the partition data operation permission lock value from the current first term value m to the second term value m+1.

[0086] Understandably, in a distributed file system, after the lock value of the partition data operation permission lock for the target partition is updated, for the target partition, since the lock value included in the file meta-operation request sent by the first service node to the distributed file system is the second term value m+1, which is equal to the current lock value of the partition data operation permission lock for the target partition, the first service node has the metadata operation permission for the partition data of the target partition and can complete the reading and writing of the messages corresponding to the target partition; while the lock value included in the file meta-operation request sent by the second service node to the distributed file system is the first term value m, which is less than the current lock value of the partition data operation permission lock for the target partition, therefore, the second service node no longer has the metadata operation permission for the partition data of the target partition and can no longer read and write the messages corresponding to the target partition.

[0087] In response to the successful update of the partition data operation permission lock value, the first service node changes its corresponding replica identifier from a secondary replica identifier to a primary replica identifier. The primary replica identifier is used to indicate to clients that the first service node provides the read and write services for messages corresponding to the target partition.

[0088] After the first service node changes its replica identifier to the primary replica identifier, the client sends a message read / write request corresponding to the target partition to the first service node. Upon receiving the message read / write request for the target partition, the first service node sends a file meta-operation request (e.g., opening a log file in the target partition) to the distributed file system. This causes the distributed file system to determine that the lock value carried by the file meta-operation request and the lock value of the partition data operation permission lock are both the second-term value m+1. The distributed file system then executes the file meta-operation corresponding to the file meta-operation request to read and write partition data in the target storage space. For example, it might open a log file in the target partition and append the message contained in the message read / write request to the log file, or read the message requested in the message read / write request from the log file.

[0089] In summary, in this embodiment, for a target partition under a target topic, by determining the term value of the primary replica of the target partition and changing the partition data operation permissions in the distributed file system based on the new primary replica's term value, the old primary replica no longer has operation permissions for the target partition's partition data, while the new primary replica does. This achieves an efficient and accurate conversion of the target partition's partition data operation permissions between the old and new primary replicas. Furthermore, after acquiring the operation permissions for the target partition's partition data, the new primary replica identifies the target partition's message read / write service to the client by changing its replica identifier to the primary replica identifier. Based on the primary replica identifier, the client sends message read / write requests to the new primary replica, which then processes these requests, thus completing the efficient and accurate switchover between the old and new primary replicas.

[0090] As mentioned earlier, in practical applications, scenarios where the primary replica of the target partition changes include, but are not limited to: the current primary replica of the target partition needs to be taken offline, the current primary replica crashes abnormally, or the current primary replica becomes unresponsive (commonly known as hanging). In at least one of these situations, the control node needs to elect a new primary replica for the target partition to replace the current primary replica (i.e., the old primary replica) and provide message read and write services to the client.

[0091] In related technologies, when an older primary replica needs to be taken offline, to ensure the integrity and accuracy of the partition data of the target partition contained in the target storage space, the older primary replica typically first stops providing read and write services for messages of the target partition to clients, then writes the data cached in its own memory to the target storage space corresponding to the target partition, and then completes the offline process, releasing its own operation permissions on the partition data of the target partition. Afterwards, the new primary replica takes over the operation permissions on the partition data of the target partition, providing read and write services for messages of the target partition to clients.

[0092] However, the sequential takeover of partition data operation permissions by the old and new primary replicas may result in low takeover efficiency. For example, if the old primary replica fails to write the data cached in memory to the target storage space for some reason, causing the write process to take too long, the new primary replica will not be able to take over the partition data operation permissions of the target partition in a timely manner, and consequently, it will not be able to provide timely read and write services for the target partition to the client.

[0093] In the event of an abnormal failure (e.g., a power outage on the service node) or loss of responsiveness, the old primary replica may fail to promptly and correctly release access permissions to the target partition's data. This can prevent the new primary replica from taking over access permissions to the target partition's data. For example, when the old primary replica becomes unresponsive, it may mistakenly believe it is still functioning correctly and can provide read / write services to clients on the target partition, thus forcibly seizing access permissions to the target partition's data, preventing the new primary replica from taking over access permissions.

[0094] Through the above Figure 3 The master replica switching method provided in this embodiment utilizes the tenure of the master replica of the target partition and the partition data operation permission lock of the target partition. Without waiting for the old master replica to release the partition data operation permission of the target partition, the new master replica can efficiently and accurately obtain the partition data operation permission of the target partition.

[0095] Understandably, the old primary replica writes cached data from memory to the target storage space, which helps ensure the integrity of the partition data in the target partition of the distributed file system. Therefore, if a timing can be determined that ensures both the data cached in memory is written to the target storage space and the timely provision of message read / write services by the new primary replica, and the replica identifier of the new primary replica is changed to the primary replica identifier at that timing, then the downtime of message read / write services for the target partition during the transition between the old and new replicas can be effectively shortened while ensuring the integrity of the partition data in the target partition.

[0096] The following detailed explanation, with reference to specific embodiments, details the process of determining when to change the replica identifier of a new primary replica to the primary replica identifier.

[0097] The following embodiments are based on Figure 3 In the embodiment shown, the first service node is the new primary replica corresponding to the target partition, and the second service node is the old primary replica corresponding to the target partition.

[0098] In one optional embodiment, in the distributed file system, the partition data corresponding to the target partition includes: a first identifier file generated when the replica identifier of the second service node is changed to the primary replica identifier. The first identifier file is sealed by the second service node after it has completely written the data cached in its own memory to the target storage space; the sealed first identifier file cannot be written to again.

[0099] In this embodiment of the application, taking advantage of the characteristic that the write operation will be blocked after the first identifier file is sealed, the first service node can determine whether the second service node has completed the write operation by judging whether the first identifier file can be written with a preset number of bytes of data during the process of the second service node writing the data cached in its own memory to the target storage space. When the write time consumed by the second service node is long, the first service node can promptly change its own replica identifier to the primary replica identifier to provide the client with read and write services for the target partition message.

[0100] Figure 5 A flowchart illustrating a method for changing the replica identifier of a first service node, provided in an embodiment of this application, is applied to a first service node in a distributed computing cluster, such as... Figure 5 As shown, it may include the following steps:

[0101] 501. In response to the successful update of the lock value of the partition data operation permission lock, a file open request corresponding to the first identifier file is sent to the distributed file system. The file open request is used to open the first identifier file when it is determined that the lock value carried by the file open request and the lock value of the partition data operation permission lock are both the second term value.

[0102] 502. In response to the opening of the first identifier file, a preset number of bytes of data are continuously written to the first identifier file at a preset first time interval.

[0103] 503. If the data of the preset bytes fails to be written due to the sealing of the first identifier file, the replica identifier corresponding to the first service node is changed from the secondary replica identifier to the primary replica identifier; wherein, the first identifier file is sealed by the second service node after it has completely written the data cached in its own memory to the target storage space.

[0104] 504. If the duration for which the preset bytes of data are successfully written to the first identifier file is greater than the preset first time threshold, the replica identifier corresponding to the first service node will be changed from the secondary replica identifier to the primary replica identifier. The primary replica identifier is used to indicate to the client that the read and write service for the message corresponding to the target partition is provided by the first service node.

[0105] For ease of understanding, combined with Figure 6 To explain, Figure 6 This is a schematic diagram illustrating the process of changing the replica identifier of a first service node, as provided in an embodiment of this application.

[0106] like Figure 6As shown, assuming that at time T0, the second service node is determined as the primary replica of the target partition, and the corresponding first term value epoch = m (i.e., the first term value m described above). In response to the replica identifier being changed to the primary replica identifier, the second service node generates a first identifier file in the target storage space of the target partition. The first identifier file, like the log file, belongs to the partition data of the target partition.

[0107] If after a period of time, for example, in Figure 6 At time T1, the primary replica of the target partition needs to be changed from the second service node to the first service node. The term value epoch of the second term corresponding to the first service node is m+1 (i.e., the second term value m+1 described above).

[0108] At time T1, if the second service node is offline normally, after receiving the offline notification, the second service node writes the data cached in its memory to the target storage space, and after the write is complete, it seals the first identifier file.

[0109] Upon successful update of the partition data operation permission lock value, the first service node, while writing the cached data in its own memory to the target storage space, simultaneously sends a file open request corresponding to the first identified file to the distributed file system. The distributed file system, upon confirming that both the lock value carried in the file open request and the lock value of the partition data operation permission lock are the second term value m+1, opens the first identified file. The process of updating the partition data operation permission lock value can be referred to the aforementioned embodiment and will not be repeated here.

[0110] Subsequently, in response to the opening of the first identifier file, the first service node continuously writes a preset number of bytes of data to the first identifier file at a preset first time interval, such as writing 1 byte of data to the first identifier file every 50ms.

[0111] If the write of the preset bytes of data fails due to the first identifier file being archived, it indicates that the second service node has completely written the data cached in its own memory to the target storage space. At this time, the first service node can change its corresponding replica identifier from the secondary replica identifier to the primary replica identifier, thereby indicating to the client that the read and write service for the messages corresponding to the target partition is provided by the first service node.

[0112] It is understandable that the second service node may fail to write the data cached in its own memory to the target storage space for some reason, which may result in the first identification file not being sealed in time.

[0113] Therefore, if the duration for which the preset number of bytes of data is successfully written to the first identifier file exceeds a preset first time threshold (e.g., 5 seconds), the first service node will change its corresponding replica identifier from a secondary replica identifier to a primary replica identifier. This effectively avoids the problem of the first service node being unable to provide timely read and write services for the messages corresponding to the target partition to the client.

[0114] Understandably, in the event of a second service node crashing or becoming unresponsive, the first identifier file will continue to be successfully written with a preset number of bytes of data because the sealing of the first identifier file cannot be triggered. Therefore, continuously writing a preset number of bytes of data to the first identifier file at a preset first time interval, and changing the replica identifier of the first service node from a secondary replica identifier to a primary replica identifier when the duration of successfully writing the preset number of bytes of data to the first identifier file exceeds a preset first time threshold, can also solve the problem that the first service node cannot provide timely read and write services for messages corresponding to the target partition to the client when the second service node crashes or becomes unresponsive.

[0115] It is worth noting that, to ensure the smooth progress of the next round of changes to the primary replica of the target partition, similar to the second service node generating the first identifier file, the first service node, in response to changing its own replica identifier to the primary replica identifier, generates a third identifier file. Subsequently, if the first service node receives a service node offline notification, it writes the data cached in its own memory to the target storage space according to the received notification; and in response, it completely writes the data cached in its own memory to the target storage space, archives the third identifier file, and completes the offline process.

[0116] In another optional embodiment, in the distributed file system, the partition data corresponding to the target partition includes: a second identifier file generated by the second service node after writing the data cached in its own memory to the target storage space.

[0117] In this embodiment, the timing for generating the second identifier file is as follows: when the second service node completely writes the data cached in its own memory to the target storage space, the first service node can determine whether the second service node has completed the write operation by judging whether the second identifier file exists in the partition data of the target partition during the process of the second service node writing the data cached in its own memory to the target storage space. When the write time consumed by the second service node is long, the first service node promptly changes its own replica identifier to the primary replica identifier to provide the client with read and write services for the target partition messages.

[0118] Figure 7A flowchart illustrating another method for changing the replica identifier of a first service node provided in this application embodiment, applied to a first service node in a distributed computing cluster, such as... Figure 7 As shown, it may include the following steps:

[0119] 701. In response to the successful update of the lock value of the partition data operation permission lock, if a second identifier file is detected in the target storage space at a preset second time interval, the replica identifier corresponding to the first service node is changed from the secondary replica identifier to the primary replica identifier. The second identifier file is generated by the second service node after writing the data cached in its own memory to the target storage space.

[0120] 702. If the duration of the second identifier file is not detected to be greater than the preset second time threshold, the replica identifier corresponding to the first service node is changed from the secondary replica identifier to the primary replica identifier. The primary replica identifier is used to indicate to the client that the read and write service of the message corresponding to the target partition is provided by the first service node.

[0121] For ease of understanding, combined with Figure 8 To explain, Figure 8 This is a schematic diagram illustrating another process for changing the replica identifier of a first service node, as provided in an embodiment of this application.

[0122] like Figure 8 As shown, assuming that at time T0, the second service node is determined as the primary replica of the target partition, and the corresponding first term value epoch = m (i.e., the first term value m described above). If after a period of time, for example, at... Figure 8 At time T1, the primary replica of the target partition needs to be changed from the second service node to the first service node. The term value epoch of the second term corresponding to the first service node is m+1 (i.e., the second term value m+1 described above).

[0123] At time T1, if the second service node is offline normally, after receiving the offline notification, the second service node will write the data cached in its own memory to the target storage space, and after the write is complete, it will generate the second identifier file.

[0124] In response to the successful update of the partition data operation permission lock value, the first service node, while writing the data cached in its own memory to the target storage space, checks whether the second identifier file exists in the target storage space at a preset second time interval (e.g., 50ms).

[0125] If a second identifier file exists, it indicates that the second service node has completely written the data cached in its memory to the target storage space. At this time, the first service node can change its corresponding replica identifier from a secondary replica identifier to a primary replica identifier, thereby indicating to the client that the read and write service for the messages corresponding to the target partition is provided by the first service node.

[0126] It is understandable that the second service node may fail to write the data cached in its own memory to the target storage space for some reason, which may result in the second identifier file not being generated in time.

[0127] Therefore, if the duration of the second identifier file is not detected to exceed a preset second time threshold (e.g., 5 seconds), the first service node will change its corresponding replica identifier from a secondary replica identifier to a primary replica identifier. This effectively avoids the problem of the first service node being unable to provide timely read and write services for messages corresponding to the target partition to the client.

[0128] Understandably, if the second service node crashes or becomes unresponsive, the second identifier file cannot be generated, and therefore cannot be successfully detected by the first service node. Thus, by checking the target storage space for the existence of the second identifier file at a preset second time interval, and changing the replica identifier of the first service node from a secondary replica identifier to a primary replica identifier if the duration of the undetected second identifier file exceeds a preset second time threshold, this approach can also resolve the issue of the first service node being unable to provide timely read / write services for messages corresponding to the target partition to the client when the second service node crashes or becomes unresponsive.

[0129] It is worth noting that, in order to ensure the smooth progress of the next round of changes to the primary replica of the target partition, similar to the second service node generating the second identifier file, if the first service node receives a service node offline notification, it will also write the data cached in its own memory to the target storage space according to the received service node offline notification; and in response, it will completely write the data cached in its own memory to the target storage space, generate the fourth identifier file, and complete the offline process.

[0130] In summary, in this embodiment of the application, the first identifier file generated by the second service node when the replica identifier is changed to the primary replica identifier, and / or the second identifier file generated by the second service node after writing the data cached in its own memory to the target storage space, determines the appropriate time for the first service node to provide message read and write services for the target partition to the client. Thus, while ensuring the integrity of the partition data of the target partition, the unavailability of message read and write services for the target partition during the transition between the old and new replicas can be effectively shortened.

[0131] In practical applications, during the process of changing between a new and old primary replica, incomplete message data may occur in the target log file corresponding to the target partition. For example, if the second service node goes offline normally, and the time taken for the second service node to write the data cached in memory to the target log file exceeds a preset time threshold (e.g., the first or second time threshold mentioned above), the first service node may directly change the corresponding replica identifier from a secondary replica identifier to a primary replica identifier, resulting in the data cached in the second service node's memory not being completely written to the target log file. Another example is when the second service node crashes or becomes unresponsive; due to the sudden crash or loss of response, messages sent by the client to the target partition may not be completely written to the target log file.

[0132] Incomplete message data in the target log file can affect the reading of the target log file. For example, when reading message data from the target log file, the reading will stop when the incomplete message data is encountered, which is commonly known as "getting stuck". As a result, it will be impossible to read other message data after the incomplete message data.

[0133] To avoid the adverse effects of incomplete message data on the target log file, in this embodiment of the application, after changing the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier, it further detects whether there is incomplete message data in the target log file used to store messages sent by the client regarding the target topic. Based on the detection results, incomplete message data is identified in the target log file and deleted from it.

[0134] Alternatively, the completeness of message data in the target log file can be determined by the message's structure, format, or specific delimiters, or by the message's size, etc.

[0135] It is understandable that if the first service node determines that the first identifier file has been sealed, or detects the second identifier file, it indicates that the second service node has completely written the data cached in memory to the target log file within the preset time threshold (e.g., the first time threshold or the second time threshold mentioned above). In this case, the message data in the target log file is usually the complete target data. Optionally, the detection of the incomplete message data mentioned above may not be performed.

[0136] In this embodiment of the application, after changing the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier, the correctness of the message data in the target log file can be guaranteed by identifying incomplete message data in the target log file and deleting the incomplete message data from the target log file, thereby improving the service quality of the message read and write service provided by the first service node.

[0137] The above embodiments provide a detailed description of the master-replica switching method provided in this application. This application also provides a distributed message queue system, such as... Figure 4 As shown, this includes: interconnected distributed file systems and distributed computing clusters;

[0138] The distributed file system is used to store partition data corresponding to the target partition under the target topic;

[0139] The distributed computing cluster includes: a first service node, a second service node, and a control node, wherein the second service node is the primary replica corresponding to the first term of the target partition;

[0140] The control node is used to send a primary replica change notification for the target partition to the first service node;

[0141] The first service node is configured to, based on the primary replica change notification, determine that it is changed from a secondary replica of the target partition to a primary replica of the target partition for the second term, wherein the first term value corresponding to the first term is less than the second term value corresponding to the second term; and, through the second term value, change the operation permissions of the partition data corresponding to the target partition in the distributed file system from the second service node to the first service node; and change the replica identifier corresponding to the first service node from a secondary replica identifier to a primary replica identifier, wherein the primary replica identifier is used to indicate to the client that the read and write services for messages corresponding to the target partition are provided by the first service node.

[0142] In an optional embodiment, when the first service node changes the operation permission of the partition data corresponding to the target partition in the distributed file system from the second service node to the first service node using the second term value, it is specifically configured to: send a lock value update request containing the second term value to the distributed file system, wherein the lock value update request is configured to update the lock value of the partition data operation permission lock from the currently corresponding first term value to the second term value when it is determined that the second term value is greater than the current lock value of the partition data operation permission lock corresponding to the target partition; and in response to the successful update of the lock value of the partition data operation permission lock, determine that the operation permission of the partition data corresponding to the target partition has been changed from the second service node to the first service node.

[0143] In an optional embodiment, the partition data of the target partition is stored in the target storage space of the distributed file system. The first service node is further configured to receive a message read / write request corresponding to the target partition sent by the client; and send a file meta-operation request corresponding to the message read / write request to the distributed file system. The file meta-operation request is used to read and write the partition data in the target storage space by executing the file meta-operation corresponding to the file meta-operation request when it is determined that the lock value carried by the file meta-operation request and the lock value of the partition data operation permission lock are both the second term value.

[0144] In an optional embodiment, the partition data of the target partition includes: a first identifier file generated when the replica identifier of the second service node is changed to the primary replica identifier. When the first service node changes the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier, it specifically performs the following: sending a file open request corresponding to the first identifier file to the distributed file system. The file open request is used to open the first identifier file when it is determined that the lock value carried by the file open request and the lock value of the partition data operation permission lock are both the second term value; in response to the opening of the first identifier file, continuously writing a preset number of bytes of data to the first identifier file at a preset first time interval; if the writing of the preset number of bytes of data fails due to the first identifier file being sealed, then changing the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier; wherein, the first identifier file is sealed by the second service node after it has completely written the data cached in its own memory to the target storage space.

[0145] In an optional embodiment, the first service node is further configured to: if the duration for which the preset bytes of data are successfully written to the first identifier file is greater than a preset first time threshold, then change the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier.

[0146] In an optional embodiment, when the first service node changes the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier, it is specifically used to change the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier if a second identifier file is detected in the target storage space at a preset second time interval; wherein, the second identifier file is generated by the second service node after writing the data cached in its own memory to the target storage space.

[0147] In an optional embodiment, the first service node is further configured to: if the duration of the second identifier file is not detected to be greater than a preset second time threshold, change the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier.

[0148] In an optional embodiment, the first service node is further configured to: generate a third identifier file in response to a change in the replica identifier corresponding to the first service node from a secondary replica identifier to a primary replica identifier; write the data cached in the memory of the first service node to the target storage space according to the received service node offline notification; and seal the third identifier file and complete the offline process in response to completely writing the data cached in the memory of the first service node to the target storage space.

[0149] The detailed execution process and technical effects of the aforementioned distributed message queue system are described in the foregoing embodiments and will not be repeated here.

[0150] The following describes in detail one or more embodiments of the master-replica switching apparatus of this application. Those skilled in the art will understand that these apparatuses can all be configured using commercially available hardware components through the steps taught in this solution.

[0151] Figure 9 This is a schematic diagram of a master-replica switching device provided in an embodiment of this application, as shown below. Figure 9 As shown, the device includes: a determining module 11, a sending module 12, and a processing module 13.

[0152] The determination module 11 is used to determine, in response to the primary replica change notification of the target partition sent by the control node, that the first service node has been changed from a secondary replica of the target partition to a primary replica of the target partition for the second term, wherein the first term value corresponding to the first term is less than the second term value corresponding to the second term.

[0153] The sending module 12 is used to change the operation permission of the partition data corresponding to the target partition in the distributed file system from the second service node to the first service node through the second term value.

[0154] Processing module 13 is used to change the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier. The primary replica identifier is used to indicate to the client that the read and write service of the message corresponding to the target partition is provided by the first service node.

[0155] In an optional embodiment, when the sending module 12 changes the operation permission of the partition data corresponding to the target partition in the distributed file system from the second service node to the first service node using the second term value, it is specifically configured to: send a lock value update request containing the second term value to the distributed file system, wherein the lock value update request is used to update the lock value of the partition data operation permission lock from the currently corresponding first term value to the second term value when it is determined that the second term value is greater than the current lock value of the partition data operation permission lock corresponding to the target partition; and in response to the successful update of the lock value of the partition data operation permission lock, determine that the operation permission of the partition data corresponding to the target partition has been changed from the second service node to the first service node.

[0156] In an optional embodiment, the partition data of the target partition is stored in the target storage space in the distributed file system. The processing module 13 is further configured to receive a message read / write request corresponding to the target partition sent by the client; and send a file meta-operation request corresponding to the message read / write request to the distributed file system. The file meta-operation request is used to read and write the partition data in the target storage space by executing the file meta-operation corresponding to the file meta-operation request when it is determined that the lock value carried by the file meta-operation request and the lock value of the partition data operation permission lock are both the second term value.

[0157] In an optional embodiment, the partition data of the target partition includes: a first identifier file generated when the replica identifier of the second service node is changed to the primary replica identifier; when the processing module 13 changes the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier, it is specifically used to: send a file open request corresponding to the first identifier file to the distributed file system; the file open request is used to open the first identifier file when it is determined that the lock value carried by the file open request and the lock value of the partition data operation permission lock are both the second term value; in response to the opening of the first identifier file, continuously write a preset number of bytes of data to the first identifier file at a preset first time interval; if the writing of the preset number of bytes of data fails due to the first identifier file being sealed, then change the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier; wherein, the first identifier file is sealed by the second service node after it has completely written the data cached in its own memory to the target storage space.

[0158] In an optional embodiment, the processing module 13 is further configured to: if the duration for which the preset bytes of data are successfully written to the first identifier file is greater than a preset first time threshold, then change the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier.

[0159] In an optional embodiment, when the processing module 13 changes the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier, it is specifically used to change the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier if a second identifier file is detected in the target storage space at a preset second time interval; wherein, the second identifier file is generated by the second service node after writing the data cached in its own memory to the target storage space.

[0160] In an optional embodiment, the processing module 13 is further configured to: if the duration of the second identifier file is not detected to be greater than a preset second time threshold, change the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier.

[0161] In an optional embodiment, the processing module 13 is further configured to: generate a third identifier file in response to the change of the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier; write the data cached in the memory of the first service node to the target storage space according to the received service node offline notification; and seal the third identifier file and complete the offline process in response to the complete writing of the data cached in the memory of the first service node to the target storage space.

[0162] Figure 9 The device shown can perform the steps described in the foregoing embodiments. For detailed execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0163] In one possible design, the above Figure 9 The structure of the master-replica switching device shown can be implemented as an electronic device, such as... Figure 10 As shown, the electronic device may include: a memory 21, a processor 22, and a communication interface 23. The memory 21 stores a computer program, which, when executed by the processor 22, enables the processor 22 to at least implement the master-slave switching method provided in the foregoing embodiments.

[0164] The aforementioned memory 21 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0165] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium.

[0166] Accordingly, this application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above method embodiments.

[0167] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. This application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] Finally, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0170] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for switching between primary and replica copies, characterized in that, The method involves applying a first service node in a distributed computing cluster, which further includes a second service node and a control node. The second service node is a primary replica of the first term corresponding to the target partition under the target topic. The distributed computing cluster is interconnected with a distributed file system storing partition data corresponding to the target partition. The method includes: In response to the primary replica change notification of the target partition sent by the control node, it is determined that the first service node is changed from a secondary replica of the target partition to a primary replica of the target partition for the second term, wherein the first term value corresponding to the first term is less than the second term value corresponding to the second term. By using the second term value, the operation permission for the partition data corresponding to the target partition in the distributed file system is changed from the second service node to the first service node; Change the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier. The primary replica identifier is used to identify to the client that the read and write service for the message corresponding to the target partition is provided by the first service node.

2. The method according to claim 1, characterized in that, The step of changing the operation permission of the partition data corresponding to the target partition in the distributed file system from the second service node to the first service node through the second term value includes: A lock update request containing the second term value is sent to the distributed file system. The lock update request is used to update the lock value of the partition data operation permission lock from the current first term value to the second term value when it is determined that the second term value is greater than the current lock value of the partition data operation permission lock corresponding to the target partition. In response to the successful update of the lock value of the partition data operation permission lock, it is determined that the operation permission of the partition data corresponding to the target partition has been changed from the second service node to the first service node.

3. The method according to claim 2, characterized in that, The partition data of the target partition is stored in the target storage space of the distributed file system, and the method further includes: Receive message read / write requests corresponding to the target partition sent by the client; The file meta-operation request corresponding to the message read / write request is sent to the distributed file system. The file meta-operation request is used to read and write partition data in the target storage space by executing the file meta-operation corresponding to the file meta-operation request when it is determined that the lock value carried by the file meta-operation request and the lock value of the partition data operation permission lock are both the second term value.

4. The method according to claim 2, characterized in that, The partition data of the target partition includes: a first identifier file generated when the replica identifier of the second service node is changed to the primary replica identifier, wherein changing the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier includes: The file open request corresponding to the first identified file is sent to the distributed file system. The file open request is used to open the first identified file when it is determined that the lock value carried by the file open request and the lock value of the partition data operation permission lock are both the second term value. In response to the opening of the first identifier file, a preset number of bytes of data are continuously written to the first identifier file at a preset first time interval; If the data of the preset bytes fails to be written due to the sealing of the first identifier file, the replica identifier corresponding to the first service node is changed from the secondary replica identifier to the primary replica identifier; wherein, the first identifier file is sealed by the second service node after it has completely written the data cached in its own memory to the target storage space.

5. The method according to claim 4, characterized in that, The method further includes: If the duration for which the preset bytes of data are successfully written to the first identifier file is greater than a preset first time threshold, then the replica identifier corresponding to the first service node is changed from the secondary replica identifier to the primary replica identifier.

6. The method according to claim 1, characterized in that, The step of changing the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier includes: If a second identifier file is detected in the target storage space at a preset second time interval, the replica identifier corresponding to the first service node is changed from the secondary replica identifier to the primary replica identifier; wherein, the second identifier file is generated by the second service node after writing the data cached in its own memory to the target storage space.

7. The method according to claim 6, characterized in that, The method further includes: If the duration of the second identifier file is not detected to be greater than the preset second time threshold, the replica identifier corresponding to the first service node will be changed from the secondary replica identifier to the primary replica identifier.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: In response to the change of the replica identifier corresponding to the first service node from the secondary replica identifier to the primary replica identifier, a third identifier file is generated; Based on the received service node offline notification, the data cached in the memory of the first service node is written to the target storage space; In response, the data cached in the memory of the first service node is completely written to the target storage space, the third identification file is sealed, and the process is completed offline.

9. A distributed message queue system, characterized in that, include: Interconnected distributed file systems and distributed computing clusters; The distributed file system is used to store partition data corresponding to the target partition under the target topic; The distributed computing cluster includes: a first service node, a second service node, and a control node, wherein the second service node is the primary replica of the target partition for the first term; The control node is used to send a primary replica change notification for the target partition to the first service node; The first service node is configured to, based on the primary replica change notification, determine that it is changed from a secondary replica of the target partition to a primary replica of the target partition for the second term, wherein the first term value corresponding to the first term is less than the second term value corresponding to the second term; and, through the second term value, change the operation permissions of the partition data corresponding to the target partition in the distributed file system from the second service node to the first service node; and change the replica identifier corresponding to the first service node from a secondary replica identifier to a primary replica identifier, wherein the primary replica identifier is used to indicate to the client that the read and write services for messages corresponding to the target partition are provided by the first service node.

10. An electronic device, characterized in that, include: The device includes a memory, a processor, and a communication interface; wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the master-replica switching method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor of an electronic device, causes the processor to perform the master-replica switching method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, include: A computer program or instruction that, when executed by a processor of an electronic device, causes the processor to perform the master-replica switching method as described in any one of claims 1 to 8.

Images