A management method and computing device for a distributed messaging system

Abstract

This application provides a management method and computing device for a distributed messaging system. The method includes: determining a first node in the distributed messaging system, the first node storing copies of one or more partitions, the one or more partitions including the first partition, and the first node storing the Nth segment of the first partition, the Nth segment being the last segment of the first partition; and configuring the second node to store the (N+1)th segment of the first partition when it is determined that the first partition will be migrated from the first node to a second node in the distributed messaging system. In this method, the second node does not store any segments of the first partition before and after the (N+1)th segment of the first partition. This method can save bandwidth occupied due to data synchronization in the partitions during partition migration and can avoid the impact of partition migration on the message service provided to users by both nodes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a management method of a distributed message system and a computing device. BACKGROUND

[0002] The distributed message system (DMS) is a message middleware service system based on high-availability distributed cluster technology. The DMS has the characteristics of large scale, high reliability, high concurrent access, scalability and full hosting.

[0003] In the related art, when a message queue in the distributed message system is migrated from a node 1 to a node 2, all messages in the message queue need to be synchronized from the node 1 to the node 2. In this scheme, the amount of data synchronization is large, and the synchronization process occupies the network bandwidth of the node 1 and the node 2, so that the node 1 provides normal message publishing services and message consuming services to the clients of the distributed message system. Moreover, since the data synchronization takes a long time, the node 2 cannot provide the message publishing services and the message consuming services to the clients within a certain time. Therefore, the migration of the message queue between the two nodes in the related art seriously reduces the user experience. SUMMARY

[0004] The present application provides a management method of a distributed message system and a computing device, which can solve the problem that data synchronization occupies the bandwidth of a node when migrating a partition, and affects the node to provide message services to users.

[0005] In a first aspect, the present application provides a management method of a distributed message system. The method comprises: determining a first node in the distributed message system, the first node being used to store one or more partitions; determining a first partition to be migrated in the first node, the one or more partitions comprising the first partition, the first node storing an Nth segment of the first partition, the Nth segment being the last segment of the first partition; in a case where it is determined to migrate the first partition from the first node to a second node in the distributed message system, configuring the second node to store an (N+1)th segment of the first partition; wherein, before and after migrating the first partition, the second node does not store a first segment to an Nth segment of the first partition. Wherein, the partition comprises a message queue, and the messages in the message queue belong to the same topic.

[0006] In the above embodiments, each partition of a topic in the distributed messaging system is divided into multiple segments. When a partition needs to be migrated, only the new node (the second node) is configured to store the next segment of that partition, without synchronizing the segments of that partition stored by the original node (the first node) to the new node. This avoids data synchronization affecting the provision of messaging services to users by both the new and original nodes, thereby improving the user experience. In conjunction with the first aspect, in one possible embodiment, the method further includes: adding metadata of the (N+1)th segment to the first metadata set of the first partition, wherein the first metadata set includes the metadata of the Nth segment, and the metadata of the Nth segment indicates that the first node stores the Nth segment.

[0007] In the above implementation, after configuring the second node to store the N+1th segment of the first partition, the metadata corresponding to the N+1th segment of the first partition is added to the metadata of the copy of the first partition stored by the original node.

[0008] In conjunction with the first aspect, in one possible implementation, when the first node is the master node of the first partition, the method further includes: configuring the second node as the master node of the first partition.

[0009] In the above implementation, when the first node is the master node of the first partition, the master node of the first partition can be set as the second node, thereby realizing the switching of the master node of the partition.

[0010] In conjunction with the first aspect, in one possible implementation, the method further includes: notifying the client of the distributed messaging system that the second node is the master node of the first partition.

[0011] In the above implementation, after setting the master node of the first partition as the second node, the client in the distributed messaging system can be notified that the master node of the first partition is the second node, so that the client can interact with the second node and obtain messages from the first partition.

[0012] In conjunction with the first aspect, in one possible implementation, determining the first node in the distributed messaging system includes: determining the first node based on the remaining storage capacity and / or load access volume of each node in the distributed messaging system.

[0013] In the above implementation, the first node that needs to be migrated from the distributed messaging system can be determined by the remaining storage capacity and / or load access volume of the node.

[0014] In a possible implementation of the first aspect, the determining the first partition to be migrated in the first node comprises: comparing load access amounts of one or more partitions in the first node, and determining the first partition.

[0015] In the above implementation, the first partition to be migrated can be determined from the first node according to the load access amounts of the respective partitions stored in the first node.

[0016] In a possible implementation of the first aspect, before the determining the first node in the distributed message system, the method further comprises: adding the second node in the distributed message system to expand the distributed message system.

[0017] In the above implementation, the method provided in the present application can be executed after the expansion of the distributed message system.

[0018] In a second aspect, the present application further provides a message processing method. The message processing method can be applied to the second node in the first aspect and the optional implementation thereof. In a case where the second node is a master node of the first partition, the method comprises: receiving a message pull request sent by a client; in a case where it is determined according to the message pull request that the client requests to obtain a first message in an n th segment of the first partition, obtaining the first message from a third node storing the n th segment of the first partition, and sending the first message to the client, the n th segment of the first partition comprising any one of a first segment to an N th segment of the first partition.

[0019] In the above implementation, in a case where the first partition stored in the first node is migrated to the second node, the client can obtain the message of the first partition stored in the first node from the second node.

[0020] In a possible implementation of the second aspect, when the n th segment of the first partition is the N th segment of the first partition, the third node comprises the first node.

[0021] In a third aspect, the present application further provides a management apparatus. The management apparatus can comprise a determining module and a migration module.

[0022] The determining module is configured to determine a first node in a distributed message system, and determine a first partition to be migrated in the first node. The first node is configured to store one or more partitions, and the one or more partitions comprise the first partition. The first node stores an N th segment of the first partition, and the N th segment is a last segment of the first partition.

[0023] The migration module is configured to, in a case where it is determined to migrate the first partition from the first node to a second node in the distributed message system, configure the second node to store an (N+1)th segment of the first partition; and the second node does not store a first segment to an Nth segment of the first partition before and after the migration of the first partition.

[0024] In combination with the third aspect, in a possible implementation, the migration module is further configured to: add metadata of the (N+1)th segment in a first metadata set of the first partition, the first metadata set including metadata of the Nth segment, and the metadata of the Nth segment indicating that the first node stores the Nth segment.

[0025] In combination with the third aspect, in a possible implementation, in a case where the first node is a master node of the first partition, the migration module is further configured to: configure the second node to be the master node of the first partition.

[0026] In combination with the third aspect, in a possible implementation, the migration module is further configured to: notify a client of the distributed message system that the second node is the master node of the first partition.

[0027] In combination with the third aspect, in a possible implementation, the determination module is specifically configured to: determine the first node according to residual storage capacities and / or load access amounts of respective nodes in the distributed message system.

[0028] In combination with the third aspect, in a possible implementation, the determination module is specifically configured to: compare load access amounts corresponding to one or more partitions in the first node to determine the first partition.

[0029] In combination with the third aspect, in a possible implementation, before determining the first node in the distributed message system, the apparatus includes an expansion module. The expansion module is configured to add the second node in the distributed message system to expand the distributed message system.

[0030] In the fourth aspect, the present application further provides a message processing apparatus. The message processing method can be applied to a second node in the distributed message system in the first aspect and the optional implementations thereof, and in a case where the second node is a master node of the first partition, the apparatus includes a receiving module and a processing module.

[0031] The receiving module is configured to receive a message pulling request sent by a client.

[0032] In a case where it is determined according to the message pulling request that the client requests to acquire a first message in an nth segment of the first partition, the first message is acquired from a third node storing the nth segment of the first partition, and the first message is sent to the client, the nth segment of the first partition includes any one of a first segment to an Nth segment of the first partition.

[0033] In combination with the fourth aspect, in a possible implementation, when the nth segment of the first partition is the Nth segment of the first partition, the third node includes the first node.

[0034] In the fifth aspect, the present application further provides a computing device. The computing device includes a processor and a memory, and the processor is configured to execute a computer program stored in the memory to implement the method in the first aspect and the optional implementation of the first aspect, or to implement the method in the second aspect and the optional implementation of the second aspect.

[0035] In the sixth aspect, the present application further provides a distributed message system. The distributed message system includes a plurality of computing devices as described in the fifth aspect.

[0036] In the seventh aspect, the present application further provides a computer readable storage medium, and the computer readable storage medium stores instructions, when the instructions are executed on a computer, the computer is caused to execute the management method provided by the first aspect or any one of the possible implementation of the first aspect, or execute the management method provided by the second aspect or any one of the possible implementation of the second aspect.

[0037] In the eighth aspect, the present application further provides a computer program product including instructions, when the instructions are executed on a computer, the computer is caused to execute the management method provided by the first aspect or any one of the possible implementation of the first aspect, or execute the management method provided by the second aspect or any one of the possible implementation of the second aspect.

[0038] Any one of the apparatus, the computing device, the distributed message system, the computer readable storage medium or the computer program product provided above is used to execute the method provided above, and thus the beneficial effects achieved can refer to the beneficial effects of the corresponding scheme in the corresponding method provided above, which will not be described herein again. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 FIG. 1 is a structural schematic diagram of a distributed message system provided by an embodiment of the present application;

[0040] Figure 2 FIG. 2 is a storage location schematic diagram of each segment of a partition in the distributed message system provided by an embodiment of the present application;

[0041] Figure 3 This is a flowchart of a management method for a distributed messaging system provided in an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of partition segmentation before and after partition migration in a distributed messaging system provided in an embodiment of this application;

[0043] Figure 5 This is a schematic diagram illustrating the storage locations of various segments of a partition before and after migration in a distributed messaging system provided in this application embodiment;

[0044] Figure 6 This is a schematic diagram of the metadata structure in a distributed messaging system provided in an embodiment of this application;

[0045] Figure 7 This is a flowchart of a message processing method in a distributed messaging system provided in an embodiment of this application;

[0046] Figure 8 This is a schematic diagram of the structure of a management device in a distributed messaging system provided in an embodiment of this application;

[0047] Figure 9 This is a schematic diagram of the structure of a message processing device in a distributed messaging system provided in an embodiment of this application;

[0048] Figure 10 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0050] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0051] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals.

[0052] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0053] Before introducing the embodiments of this application, a brief introduction to distributed messaging systems will be given below.

[0054] A distributed messaging system is a system that provides messaging services to users. In its implementation, a distributed messaging system may include a cloud platform that provides messaging services. These messaging services include message publishing and message consumption. Message consumption can be implemented through message subscription and / or message retrieval. Message subscription refers to a user requesting to subscribe to a specific type of message from the distributed messaging system; after receiving the user's subscription, the distributed messaging system pushes messages belonging to that type to the user. Message retrieval refers to a user actively retrieving messages belonging to a specific type from the system.

[0055] A distributed messaging system may include multiple message broker nodes and one or more management nodes. Message broker nodes provide messaging services to users. Management nodes configure the message broker nodes in the distributed messaging system, enabling them to provide messaging services to users. Configuring message broker nodes in the distributed messaging system may include adding message broker nodes or deleting message broker nodes that are unable to provide services to users. It should be noted that message broker nodes and management nodes may include applications deployed on computing devices. Message broker nodes and management nodes can be deployed on computing devices using virtualization methods, such as deploying virtual machines or containers on the computing device, and deploying the applications corresponding to the message broker nodes or management nodes within the virtual machines or containers. When the application corresponding to a message broker node is deployed on a computing device, the computing device performs the functions of the message broker node. When the application corresponding to a management node is deployed on a computing device, the computing device performs the functions of the management node.

[0056] A producer is a client that publishes messages to a distributed messaging system. A consumer is a client that consumes messages from a distributed messaging system. Clients can include applications deployed on the user's terminal device.

[0057] A topic refers to the category or type of a message. Distributed messaging systems use topics to define message categories or types for easier message management. Different topics can be predefined in a distributed messaging system, and users can publish or consume messages on topics of interest. In other words, every message in a distributed messaging system has its own topic.

[0058] A partition is a message queue that stores messages for a single topic. In a distributed messaging system, a topic can include multiple partitions, and all messages stored in a partition belong to the same topic. In a distributed messaging system, message broker nodes can use a sequential append-only approach to write user-published messages into the corresponding message queue (i.e., partition) for the appropriate topic. A partition is stored on one message broker node. Multiple partitions can be stored on a single message broker node or on different message broker nodes.

[0059] To ensure the security and reliability of a distributed messaging system, a master-slave mechanism is configured. Specifically, each partition of a topic has one master node (leader) and one or more slave nodes (followers) in the distributed messaging system.

[0060] The master node of a partition provides messaging services to users for that partition. The slave nodes of a partition are used to back up and store messages from the master node. A slave node will only become the master node when the master node is unable to provide services to users, and then it will provide messaging services. The master node can synchronize messages from the partition to the slave nodes according to a pre-configured synchronization mechanism, thus achieving message backup.

[0061] Figure 1 This is a schematic diagram of the structure of a distributed messaging system 100 provided in an embodiment of this application. Figure 1 In the distributed messaging system 100, nodes 110 to 113 may be included. Node 110 is the management node of the distributed messaging system 100, and nodes 111 to 113 are message broker nodes of the distributed messaging system 100. It should be noted that... Figure 1This is merely an example of a distributed messaging system in this application and does not constitute a limitation on the structure of a distributed messaging system. In specific applications, the distributed messaging system 100 may include Kafka, RocketMQ, or RabbitMQ, etc.

[0062] Nodes 110 to 113 can communicate with client 130 to provide messaging services to users. Client 130 is deployed on the user's terminal device. The terminal device may include a laptop, desktop computer, smartphone, tablet, or in-vehicle computer, etc. Nodes 111 to 113 can communicate with each other to back up partition messages or perform other interactions.

[0063] Taking node 111 as an example, node 110 can be either the master node or the slave node of a partition. That is, for a given partition, the node storing that partition is either the master node or the slave node of that partition. Of course, a node can be the master node or the slave node of one or more partitions. When a node is the master or slave node of a partition, it should store a copy of that partition.

[0064] Node 110 can be used to store the metadata of each replica corresponding to a partition of each topic in the distributed messaging system. The metadata of each replica corresponding to a partition indicates the storage location of messages in each replica corresponding to that partition.

[0065] In related technologies, when a partition needs to be migrated from one node to another, all messages for that partition stored on the first node are migrated to the second node. When there are many messages in the partition, the amount of data being migrated is substantial, consuming significant network and storage bandwidth on both the original and new nodes. This impacts the ability of both nodes to provide messaging services to users, potentially disrupting normal business operations.

[0066] Based on this, embodiments of this application provide a management method for a distributed messaging system, which can solve the above-mentioned problems.

[0067] In this embodiment of the application, to solve the above problems, a segmentation mechanism is adopted to manage the various partitions of the topic, that is, each partition is divided into multiple segments. The segments in each replica corresponding to a partition can be stored on different message broker nodes. For example... Figure 2As shown, a topic comprises three partitions, partitions 1 through 3. Taking partition 2 as an example, it can be divided into segments 1 through N for management. The segments of partition 2 can be stored on different nodes. For example, segments 1 and 2 are stored on node 111, and segments 3 through N are stored on node 112. Furthermore, an upper limit can be set for the number of messages in a segment. When the number of messages in a partition's segments equals this upper limit, a new segment can be created for that partition.

[0068] In the management method provided in the embodiments of this application, based on Figure 2 The segmentation mechanism shown creates a new segment for the partition based on the last segment stored on the previous node when migrating a partition to another, and then configures the other node to store this new segment. The other node does not need to store segments prior to the new segment. This way, during partition migration, messages stored on previous nodes do not need to be synchronized, thus avoiding the aforementioned problems caused by message synchronization.

[0069] The following is combined Figure 3 The management method provided in the embodiments of this application will be described in detail.

[0070] Figure 3 This is a flowchart illustrating a management method for a distributed messaging system provided in an embodiment of this application. This method can be applied to... Figure 1 In the distributed messaging system 100 shown, node 110 (the management node) is executed by node 110. For example... Figure 3 As shown, the method may include the following steps S301-S303.

[0071] S301, determine the first node in the distributed messaging system. The first node refers to the node in the distributed messaging system 100 that stores the partition to be migrated.

[0072] In this step, after expanding the distributed messaging system 100, node 110 can find the first node storing the partition to be migrated from among the nodes of the distributed messaging system 100. Node 110 can determine the first node based on the remaining storage capacity and / or load access volume of each node in the distributed messaging system 100.

[0073] For example, when node 113 is added to the distributed messaging system 100, node 110 can compare the remaining storage capacity of node 111 with that of node 112 and select the node with the least remaining storage capacity as the first node.

[0074] For example, if node 113 is added to the distributed messaging system 100, node 110 can compare the load access volume of node 111 with that of node 112 and select the node with the largest load access volume as the first node.

[0075] For example, node 110 can also assign certain weights to the remaining storage capacity and load access volume, and then calculate the first value corresponding to each node based on the weights, and determine the first node based on the first value corresponding to each node.

[0076] Furthermore, in other embodiments, node 110 can periodically analyze the remaining storage capacity and / or load access volume of each node in the system to determine the first node in the system that stores the partition to be migrated, instead of only executing the method in this embodiment after system expansion. This avoids the user experience being affected by nodes in the system having insufficient storage capacity and / or high load access volume.

[0077] S302, determine the first partition to be migrated in the first node. The first node stores the Nth segment of the first partition, and the Nth segment is the last segment of the first partition.

[0078] In this step, node 110 can determine the partition with the highest load access volume among the partitions stored in the first node as the first partition to be migrated. Specifically, when the first node is a slave node of a partition it stores, the load access volume corresponding to that partition is 0. Thus, node 110 can only count the load access volume of the partition where the first node is the master node.

[0079] Taking node 111 as an example, if node 111 stores partitions 1 to 3 (partitions 1 to 3 can belong to the same topic or different topics), and if node 111 is a slave node of partition 1 and a master node of partitions 2 and 3, the load access volume corresponding to partition 1 is 0. The load access volumes corresponding to partitions 2 and 3 are determined based on the specific numbers counted by node 111. For example, when the load access volumes corresponding to partition 2 and partition 3 are 10000 and 5000 respectively, partition 2 has the highest load access volume among the three partitions, therefore partition 2 is the partition to be migrated from node 111.

[0080] Before migrating partition 2, partition 2 consists of N segments. Node 111 stores at least the last segment of partition 2. For example... Figure 4 As shown, partition 2 can include segments 1 through 6. Segment 6 is the last segment of partition 2. Since partition 2 is a message queue, segments 1 through 6 constitute a portion of a message queue. Figure 5As shown, node 111 stores segments 6, 5, and 4 of partition 2. This embodiment does not specifically limit whether node 111 stores segments 3 to 1 of partition 2. Node 111 may or may not store segments 3 to 1 of partition 2.

[0081] S303, if it is determined that the first partition will be migrated from the first node to the second node, configure the second node to store the (N+1)th segment of the first node. Before and after migrating the first partition, the second node does not store the first to Nth segments of the first partition.

[0082] In this step, taking partition 2 of node 111 as an example, if it is determined that partition 2 will be migrated from node 111 to node 113, node 110 can configure node 113 to store the (N+1)th segment of partition 2. After migrating partition 2, the last segment of partition 2 will be the (N+1)th segment.

[0083] Specifically, configuring the N+1th segment of storage partition 2 on node 110 may include creating the N+1th segment for partition 2 and sending a first notification message to node 113 to indicate the N+1th segment of storage partition 2. This first notification message may include the identifier of partition 2, the identifier of the topic to which partition 2 belongs, and the identifier of the N+1th segment of partition 2. Figure 4 As shown, before migrating partition 2, the last segment of partition 2 is segment 6. Node 110 can be configured to store segment 7 of partition 2 on node 113 to migrate partition 2; the last segment of partition 2 is segment 7. Figure 5 As shown, after migrating partition 2, segments 6 to 4 of the storage partition of node 111 and segment 7 of the storage partition of node 2 are migrated.

[0084] exist Figure 3 In the method shown, node 110 can store metadata for each replica of each partition in the distributed messaging system 100. A partition can include multiple replicas, and the metadata of each replica includes the metadata of each segment of that partition. Figure 6 As shown, taking a topic including partitions 1 to 3 as an example, node 110 can be configured according to... Figure 6 The data structure shown stores the metadata for each copy of the three partitions of this topic.

[0085] Partition 2 includes Figure 6Taking replicas 1 and 2 as examples, replica 1 of partition 2 can be stored on node 111, and replica 2 can be stored on node 112. Node 111 is the master node of partition 2, and node 112 is the slave node of partition 2. The metadata of replicas of partition 2 includes the metadata of each segment of partition 2. For example, the metadata of a replica of a partition can be represented as: / brokers / topics / {topic} / partitions / {partition} / replicas / replica identifier|-members / segment metadata. Here, {topic} represents the identifier of the topic corresponding to the partition, {partition} represents the identifier of the partition, the replicas field represents the replica, and the members field represents the members (segments) of the partition. Taking replica 2 of partition 2 as an example, the replica identifier is 2.

[0086] The metadata for a segment can include the segment's identifier, its location within the partition, the identifier of the node storing the segment, and the state of the node storing the segment. If the segment is not the last segment of the partition, its location within the partition can include the offsets of the first and last messages of the segment. If the segment is the last segment of the partition, its location within the partition can include the offset of the first message of the segment and the target identifier. The node's state can include online and offline status.

[0087] by Figure 4 Taking partition 2 as an example, before migrating partition 2, partition 2 includes segments 1 to 6. If the upper limit of the number of messages in each segment is 1000, the metadata of segment 1 can be represented as / 1: {"0, 999": 111, "state": "online"}, and the metadata of segment 6 can be represented as / 6: {"5000, -1": 111, "state": "online"}. Here, in the metadata of segment 1, "0, 999" represents the offset of the first message and the offset of the last message in segment 1, respectively. In the metadata of segment 6, "5000, -1" represents the offset of the first message and the target identifier in segment 6, respectively.

[0088] exist Figure 3In the illustrated method embodiment, after node 110 configures the second node to store the (N+1)th segment of the first partition, if the second node receives a message published by a user belonging to the first partition, it can write the published message to the storage area corresponding to the (N+1)th segment of the first partition and send a metadata update message to node 110 to add the metadata of the (N+1)th segment of the first partition to the first metadata set. The first metadata set of the first partition includes the metadata of each segment of the first partition, and the metadata of the Nth segment in the first metadata set indicates that the first node stores the Nth segment.

[0089] Taking partition 2 as an example, the first metadata dataset refers to the metadata corresponding to replica 1 of partition 2. After receiving the metadata update message, node 110 adds the metadata of the (N+1)th segment to the metadata of replica 1 corresponding to partition 2. After migrating partition 2, partition 2 includes segments 1 to 7. The metadata of segments 1 to 6 can remain unchanged. The metadata of segment 7 can be represented as / 1:{"X, -1":113, "state": "online"}, where X represents the offset of the first message stored in segment 7. The offset of the first message corresponding to segment 7 can be determined based on the offset of the last message corresponding to segment 6.

[0090] exist Figure 3 In the method shown, if the first node is the master node of the first partition, node 110 can also be configured as the master node of the first partition. For example... Figure 5 As shown, if node 111 is the master node of partition 2, node 110 can also configure node 113 as the master node of partition 2. After setting node 113 as the master node of partition 2, node 111 becomes a non-master node of partition 2. In this case, node 111 can act as a slave node of partition 2, or it can no longer be used to store copies of partition 2.

[0091] The above Figure 3 In the illustrated method embodiment, when it is determined that a partition needs to be migrated, only the new node is used as the storage node for the next segment of that partition, without synchronizing the various segments of that partition stored on the original node to the new node. This avoids the problem of node bandwidth being occupied due to data synchronization, saving network resources. On the other hand, it avoids the impact of data synchronization on the original node and the new node's ability to provide messaging services to users, thus improving the user experience.

[0092] based on Figure 3 As shown in the illustrated method embodiments, after migrating a partition, and with the new node acting as the master node for that partition, the new node does not store the segments (historical segments) of that partition stored by the original node. Therefore, the client cannot obtain messages from the historical segments from the new node. To address this issue, this application also provides a message processing method.

[0093] Figure 7 This is a flowchart of a message processing method provided in an embodiment of this application. The method may include the following steps S701-S704. The following uses… Figure 1 The following is an introduction using nodes 111, 113, client 1, and client 2 in the distributed messaging system 100 shown as examples.

[0094] S701, Client 1 sends a message retrieval request to Node 113. This message retrieval request is used to retrieve the first message of the first partition.

[0095] S702, Node 113 receives a message retrieval request and determines that Client 1 needs to retrieve the first message based on the message retrieval request. Then, it sends a message retrieval request to Node 111 to retrieve the first message.

[0096] In this step, node 113 can receive a message retrieval request from client 1. This request may include a topic identifier, a partition identifier, and a message identifier. Node 113 can determine the first message that client 1 needs to retrieve based on the topic identifier, partition identifier, and message identifier in the message retrieval request. The message identifier may include the offset of the first message within the partition corresponding to the topic identifier and partition identifier.

[0097] If it is determined that the first message obtained by client 1 belongs to the nth segment of the first partition (including any segment from the 1st to the Nth segment), a message retrieval request is sent to node 111 to obtain the first message. After the first partition stored by node 111 is migrated to node 113, node 113 begins storing the (N+1)th segment of the first partition, and does not store the 1st to the Nth segments preceding the (N+1)th segment of that partition. If the first message belongs to any segment from the 1st to the Nth segment of the first partition, node 113 can retrieve the first message from node 111.

[0098] Taking the first partition as the aforementioned partition 2 as an example, node 113 stores segment 7 of partition 2, but does not store segments 1 to 6 of partition 2. When the offset S in the message retrieval request is within the offset range corresponding to segment 6 of partition 2, node 113 can send a message retrieval request to node 111 to retrieve the first message belonging to segment 6 from node 111.

[0099] S703, after receiving the message retrieval request, node 111 sends the first message to node 113.

[0100] In this step, the message retrieval request may include a topic identifier, a partition identifier, and a message identifier for the first message. Node 111 can retrieve the first message from its corresponding storage space based on the topic identifier, partition identifier, and message identifier in the message retrieval request, and then send the first message to node 113.

[0101] S704, Node 113 sends this first message to Client 1.

[0102] The above Figure 7 In the method embodiment shown, when node 113 determines that the message the client needs to obtain belongs to the historical segment before the partition migration corresponding to the message, it obtains the message from the node storing the historical segment. This allows the user to obtain the message belonging to the partition stored in the node before the migration from the node after the migration of the partition.

[0103] Furthermore, the new node after partition migration (e.g., node 113) can directly send notification messages to clients to inform them that the messages they retrieved are stored on the original node (e.g., node 111). In a specific implementation, when the distributed messaging system 100 is Kafka, the notification message may include a preferred read replica message. The preferred read replica message may include the node identifier of the original node to instruct the client to retrieve the first message before partition migration from the original node.

[0104] based on Figure 3 The method embodiments shown in this application also provide a management device 800. This management device 800 can be applied to... Figure 1 In node 110 of the distributed messaging system 100 shown, the following is used to implement Figure 3 The steps are shown.

[0105] Figure 8 This is a schematic diagram of the structure of a management device 800 provided in an embodiment of this application. The management device 800 may include a determination module 801 and a migration module 802.

[0106] The determination module 801 is used to determine the first node in the distributed messaging system and the first partition to be migrated within the first node. The first node stores one or more partitions, including the first partition. The first node stores the Nth segment of the first partition, which is the last segment of the first partition.

[0107] The migration module 802 is configured to store the N+1th segment of the first partition when it is determined that the first partition will be migrated from the first node to the second node in the distributed messaging system. Before and after migrating the first partition, the second node does not store the first to Nth segments of the first partition.

[0108] It should be noted that, Figure 8 The management device 800 provided in the illustrated embodiment is executing Figure 3 The management method shown is only an example of the above-described division of functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the management device 800 provided in the above embodiment and... Figure 3 The management method embodiments shown belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0109] based on Figure 7 The method embodiment shown in this application also provides a message processing apparatus 900. This message processing apparatus 900 can be applied to... Figure 1 In node 113 of the distributed messaging system 100 shown, the following is used to implement Figure 7 The method steps S702 and S704 are shown.

[0110] Figure 9 This is a schematic diagram of the structure of a message processing device 900 provided in an embodiment of this application. The message processing device 900 may include a receiving module 901 and a processing module 902.

[0111] The receiving module 901 is used to receive message retrieval requests sent by the client.

[0112] The processing module 902 is used to retrieve the first message from the third node storing the nth segment of the first partition when it is determined from the message retrieval request that the client requests to retrieve the first message in the nth segment of the first partition, and send the first message to the client. The nth segment of the first partition includes any segment from the 1st segment to the Nth segment of the first partition.

[0113] It should be noted that, Figure 9 The message processing apparatus 900 provided in the illustrated embodiment is only described using the above-described division of functional modules as an example when executing the message processing method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the message processing apparatus 900 provided in the above embodiment and... Figure 7The message processing method embodiments shown belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0114] Figure 10 This is a schematic diagram of the hardware structure of a computing device 1000 provided in an embodiment of this application. The computing device 1000 can be a node in the aforementioned distributed messaging system 100. The computing device 1000 may include devices such as servers.

[0115] See Figure 10 The computing device 1000 includes a processor 1001, a memory 1002, a communication interface 1003, and a bus 1004. The processor 1001, memory 1002, and communication interface 1003 are connected to each other via the bus 1004. The processor 1001, memory 1002, and communication interface 1003 may also be connected using other connection methods besides the bus 1004.

[0116] The memory 1002 can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical storage, hard disk, etc.

[0117] The processor 1001 may be a general-purpose processor, which can be a processor that performs specific steps and / or operations by reading and executing contents stored in memory (e.g., memory 1002). For example, the general-purpose processor may be a central processing unit (CPU). The processor 1001 may include at least one circuit to perform... Figure 3 or Figure 7 All or part of the steps of the method provided in the illustrated embodiments.

[0118] The communication interface 1003 includes input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting devices within the computing device 1000, as well as interfaces for interconnecting the computing device 1000 with other devices (such as other computing devices or user equipment). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.

[0119] The bus 1004 can be any type of communication bus used to interconnect the processor 1001, memory 1002 and communication interface 1003, such as a system bus.

[0120] The aforementioned devices can be disposed on separate chips, or at least partially or entirely on the same chip. Whether to dispose of the devices independently on different chips or integrate them on one or more chips often depends on the needs of the product design. This application does not limit the specific implementation of the aforementioned devices.

[0121] Figure 10 The computing device 1000 shown is merely an example. In the implementation process, the computing device 1000 may also include other components, which will not be listed one by one in this article.

[0122] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0123] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. It should be understood that in the embodiments of this application, the order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0124] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

Claims

1. A management method for a distributed messaging system, characterized in that, The method includes: Determine a first node in the distributed messaging system, the first node being used to store one or more partitions; The first partition to be migrated in the first node is determined, and one or more of the partitions include the first partition. The first node stores the Nth segment of the first partition, and the Nth segment is the last segment of the first partition. If it is determined that the first partition will be migrated from the first node to the second node in the distributed messaging system, the second node is configured to store the (N+1)th segment of the first partition; wherein, before and after migrating the first partition, the second node does not store the first to Nth segments of the first partition.

2. The method according to claim 1, characterized in that, The method further includes: Add the metadata of the (N+1)th segment to the first metadata set of the first partition. The first metadata set includes the metadata of the Nth segment, and the metadata of the Nth segment indicates that the first node stores the Nth segment.

3. The method according to claim 1 or 2, characterized in that, When the first node is the master node of the first partition, the method further includes: Configure the second node as the master node of the first partition.

4. The method according to claim 3, characterized in that, The method further includes: The client of the distributed messaging system is notified that the second node is the master node of the first partition.

5. The method according to any one of claims 1-4, characterized in that, Determining the first node in the distributed messaging system includes: The first node is determined based on the remaining storage capacity and / or load access volume of each node in the distributed messaging system.

6. The method according to any one of claims 1-5, characterized in that, Determining the first partition to be migrated in the first node includes: The first partition is determined by comparing the load access volume corresponding to one or more partitions in the first node.

7. The method according to any one of claims 1 to 6, characterized in that, Before determining the first node in the distributed messaging system, the method further includes: The second node is added to the distributed messaging system to expand its capacity.

8. A message processing method, characterized in that, The method, applied to a second node in any one of claims 1 to 7, wherein the second node is the master node of the first partition, comprises: Receive message fetch requests sent by the client; If, based on the message retrieval request, it is determined that the client requests to retrieve the first message in the nth segment of the first partition, the first message is retrieved from the third node storing the nth segment of the first partition, and the first message is sent to the client. The nth segment of the first partition includes any one of the segments from the 1st segment to the Nth segment of the first partition.

9. The method according to claim 8, characterized in that, When the nth segment of the first partition is the Nth segment of the first partition, the third node includes the first node.

10. A computing device, characterized in that, The computing device includes a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the method of any one of claims 1 to 7, or to implement the method of claim 8 or 9.

11. A distributed messaging system, characterized in that, The distributed messaging system includes multiple computing devices as described in claim 10.

12. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 7, or the method of claim 8 or 9.

Images