A message queue rebalancing method, device, equipment and storage medium

Abstract

The present invention discloses a message queue rebalancing method, apparatus, device, and storage medium. The method comprises: obtaining storage space information of each source storage node in a message queue, storing the data in the message queue in segments in the form of data segments in each source storage node; determining the data to be migrated in each source storage node based on the storage space information of the source storage node; concurrently migrating the data to be migrated in each source storage node to a target storage node; and clearing the data to be migrated in each source storage node. The embodiment of the present application can concurrently migrate the segmented stored data to be migrated to the target storage node, thereby improving data migration efficiency; in the case of a lack of expansion resources, rebalancing is performed using storage nodes with less storage pressure, thereby improving the utilization rate of the storage nodes. The present invention can be widely applied in the field of data processing technology.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a message queue rebalancing method, apparatus, device and storage medium. Background Art

[0002] Message queues are widely used in the internet industry for high-concurrency, high-load, and large-scale data scenarios. Distributed message queues, as a common middleware product, offer advantages such as asynchrony, decoupling, and peak load shaving for calls between multiple services, enabling business systems to more confidently address the challenges of big data. Massive amounts of data require message queues to have massive storage capacity. However, when the amount of data accumulated across storage nodes in a message queue is unbalanced, the storage capacity of nodes with large storage volumes will be exhausted more quickly, while the storage capacity of nodes with small storage volumes will not be fully utilized, significantly impacting the overall availability of the message queue.

[0003] Traditional technologies typically add new storage nodes to the message queue to balance nodes with large backlogs. However, because traditional message queues typically use partitioned storage, migrating data to new storage nodes results in low balancing efficiency due to the large data volumes in each partition. Furthermore, different partitions have a master-slave relationship, and migrating the master partition can affect data writes to the message queue. Therefore, migrating data cannot balance the backlog on the storage node where the master partition resides, resulting in limited flexibility. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a message queue rebalancing method, apparatus, device, and storage medium to improve the efficiency and flexibility of the rebalancing process.

[0005] One aspect of the present application provides a message queue rebalancing method, comprising:

[0006] Acquire storage space information of each source storage node in a message queue, wherein data segments in the message queue are stored in each source storage node;

[0007] Determining the data to be migrated in each of the source storage nodes according to the storage space information of the source storage nodes;

[0008] Concurrently migrating the data to be migrated in each of the source storage nodes to a target storage node, wherein the target storage node is a storage node newly added to the message queue or a storage node whose space utilization in the message queue is lower than a storage threshold;

[0009] The data to be migrated in each of the source storage nodes is cleared.

[0010] Another aspect of the present application provides a message queue rebalancing device, comprising:

[0011] An acquisition module, configured to acquire storage space information of each source storage node in a message queue, wherein the data segments in the message queue are stored in each of the source storage nodes;

[0012] a determination module, configured to determine the data to be migrated in each of the source storage nodes according to the storage space information of the source storage nodes;

[0013] A concurrent migration module, configured to concurrently migrate the data to be migrated in each of the source storage nodes to a target storage node, wherein the target storage node is a storage node newly added to the message queue or a storage node whose space utilization in the message queue is lower than a storage threshold;

[0014] The clearing module is used to clear the data to be migrated in each of the source storage nodes.

[0015] Another aspect of the present application provides an electronic device, comprising a processor and a memory;

[0016] The memory stores a program;

[0017] The processor executes the program to implement the method described above.

[0018] Another aspect of the present application provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0019] The embodiment of the present application can obtain the storage space information of each source storage node in the message queue, wherein the data in the message queue is stored in each source storage node in the form of data segments, and then the data to be migrated in each source storage node is determined according to the storage space information of the source storage node, and then the data to be migrated in each source storage node is concurrently migrated to the target storage node, and finally the data to be migrated in each source storage node is cleared. The embodiment of the present application refines the partition storage method in the traditional message queue into a segmented storage method, which can reduce the amount of data in the data migration process and improve the rebalancing efficiency of the message queue; in addition, the data to be migrated in the embodiment of the present application is in the form of segmented storage, so there is no division of master and slave partitions, and the content of any data segment can be migrated and balanced without restriction, which improves flexibility; at the same time, the embodiment of the present application can concurrently migrate the data to be migrated to the target storage node, which improves the data migration efficiency; in addition, the target storage node of the embodiment of the present application is a storage node newly added to the message queue or a storage node whose space utilization in the message queue is lower than the storage threshold, which can make full use of the node with less storage pressure in the message queue for rebalancing, thereby improving the utilization of the storage node. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 A schematic diagram of an implementation environment of an embodiment of the present application;

[0022] Figure 2 A flowchart of a message queue rebalancing method provided in an embodiment of the present application;

[0023] Figure 3 A flowchart for obtaining storage space information of a source storage node provided in an embodiment of the present application;

[0024] Figure 4 A flowchart for determining data to be migrated provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of segmented storage of data in storage nodes provided in an embodiment of the present application;

[0026] Figure 6 A flowchart of concurrently migrating data to be migrated to a target storage node provided in an embodiment of the present application;

[0027] Figure 7A flowchart of concurrently migrating data to be migrated to a target storage node according to parallel threads provided in an embodiment of the present application;

[0028] Figure 8 Schematic diagram of data partition migration and data segment migration provided in an embodiment of the present application;

[0029] Figure 9 A schematic diagram of the composition of the topic provided in the embodiment of this application;

[0030] Figure 10 A schematic diagram of a message queue rebalancing method provided in an embodiment of the present application;

[0031] Figure 11 A flowchart of a message queue rebalancing method based on a Pulsar environment provided in an embodiment of the present application;

[0032] Figure 12 A flowchart of updating the metadata of the data to be migrated in the target storage node through the Zookeeper terminal device cluster provided in an embodiment of the present application;

[0033] Figure 13 A flowchart of clearing data to be migrated from each source storage node provided in an embodiment of the present application;

[0034] Figure 14 A flowchart of clearing data to be migrated from each source storage node based on Zookeeper provided in an embodiment of the present application;

[0035] Figure 15 A flowchart of a single segment migration based on a Pulsar environment provided in an embodiment of the present application;

[0036] Figure 16 A schematic diagram of the structure of a message queue rebalancing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0038] To facilitate understanding of the embodiments of the present application, the following briefly introduces the relevant concepts involved in the present application:

[0039] Kafka: A message middleware developed and open-sourced by LinkedIn. Kafka message middleware mainly consists of producers, brokers, and consumers. Producers publish messages, brokers forward messages from producers to consumers, and consumers receive and process messages. Producers and brokers act as the client and server of messages, respectively. In actual use, multiple Kafka brokers are generally clustered to form Kafka. Kafka provides message persistence capabilities with a time complexity of 0(1). Time complexity 0(1) means that no matter how the scale of input data increases, it will not affect Kafka's processing performance. That is, Kafka's time complexity is independent of the scale of input data, and constant time access performance can be guaranteed even for data above TB level. Kafka supports both offline data processing and real-time data processing.

[0040] Pulsar: Similar to Kafka, Pulsar is a message middleware developed by Yahoo. Each partition of Pulsar uses a segment (data segment) as the unit of physical storage and supports concurrent reading and writing of segments. It has high throughput, low latency, separation of computing and storage, multi-tenancy, and remote replication.

[0041] Zookeeper: It is a distributed coordination service and a middleware that can provide master-slave coordination, server node control, unified configuration management, distributed shared locks, unified name service and other functions. In essence, it manages and reads user-submitted data and provides monitoring services for the data.

[0042] Rebalancing: The rebalancing in the embodiments of the present application can be regarded as a data migration method. During the rebalancing process of the message queue, if the space utilization rate of the current storage node is high, that is, the remaining storage space is small, the data is migrated from the current storage node to another storage node with larger remaining storage space, so that the data in the message queue tends to be evenly distributed among the various storage nodes, thereby achieving the purpose of relieving the pressure on the storage nodes and making full use of the existing storage node space. It should be noted that the uniform distribution here does not mean that the amount of data between the various storage nodes is absolutely the same, but is a conceptual description, referring to the space utilization rate of each storage node tending to be balanced.

[0043] In the related art, during the balancing process of the message queue, new storage nodes are usually added to the message queue to balance the nodes with large backlogs. However, this method of adding new storage nodes has the following disadvantages:

[0044] 1. Data is stored in partitions. When a new storage node is added, the data of a single partition is migrated to the new storage node. Since each partition has a large amount of data and multiple partitions cannot be migrated concurrently, the migration efficiency is low and inflexible.

[0045] 2. Different partitions have a master-slave relationship. If the data in the master partition is migrated, it will affect the writing of new data. Therefore, traditional technologies cannot balance the storage nodes of the master partition.

[0046] 3. When the number of newly added storage nodes is less than the number of files to be written, since the message queue stores data in partitions, each file to be written needs to occupy a storage node. Therefore, when the number of files to be written exceeds the number of newly added storage nodes, the files to be written need to occupy the old storage nodes, and the storage pressure on the old storage nodes cannot be relieved.

[0047] Therefore, an embodiment of the present application provides a message queue rebalancing method that can obtain the storage space information of each source storage node in the message queue, wherein the data in the message queue is stored in each source storage node in the form of data segments, and then the data to be migrated in each source storage node is determined based on the storage space information of the source storage node, and then the data to be migrated in each source storage node is concurrently migrated to the target storage node, and finally the data to be migrated in each source storage node is cleared. The embodiment of the present application can concurrently migrate the segmented stored data to be migrated from the source storage node with a large amount of data accumulation to the target storage node, thereby improving the efficiency of data migration; in the case of a lack of expansion resources, it can make full use of the nodes with less storage pressure in the message queue for rebalancing, thereby improving the utilization rate of the storage nodes in the message sequence.

[0048] refer to Figure 1 , which shows a schematic diagram of an implementation environment of an embodiment of the present application, in which at least one terminal device 101 is included. The terminal device can be any electronic product that can run the message queue in the embodiment of the present application, such as a personal computer (PC), a mobile phone, a smart phone, a personal digital assistant (PDA), a wearable device, a handheld computer (PPC), a tablet computer, etc. It can be understood that in addition to being the above-mentioned electronic product that can run the message queue in the embodiment of the present application, the terminal device can also be a server-side installed in the cloud. The server-side can be a server, or a server cluster composed of several servers, or a cloud computing service center. The server-side can be used to obtain data storage information in the message queue and rebalance the data in the message queue.

[0049] It can be understood that the terminal device can be any electronic product that can perform human-computer interaction through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device. The terminal device can also be a server installed on the cloud, and the server can receive remote instructions sent by the user, and the remote instructions are used to execute the rebalancing method of the embodiment of the present application.

[0050] In the embodiment of the present application, the message queue runs in the server, and the data in the message queue is stored in at least one first storage node 102 and at least one second storage node 103. The storage node refers to a plurality of storage spaces for storing data in the message queue. It is understandable that there may be multiple storage nodes in the embodiment of the present application, which is not limited here. For example, Figure 1 As shown, during the data storage process in a message queue, the following scenario may occur: the first storage node already stores a relatively large amount of data, and its storage space utilization has reached 95%, meaning that the first storage node's storage space is about to be exhausted. To expand capacity, a new storage node is added to the message queue, referred to as the second storage node. At this point, the utilization of the second storage node is 0%. Suppose the message queue needs to write two new files. Since there is only one newly added second storage node, and the number of second storage nodes is less than the number of files to be written, the old first storage node will still be used when writing the new files, and the idle capacity of the second storage node cannot be fully utilized, resulting in poor expansion. Furthermore, if the utilization of the first storage node drops from 95% to 100%, the first storage node will be unable to write data, resulting in a data write failure. The only way to successfully write the two new files is to expand the number of second storage nodes from one to two. However, if the number of expanded nodes is limited, data writes will fail, and the current imbalance will seriously affect the overall availability of the message queue. Therefore, the message queue needs to be rebalanced.

[0051] like Figure 1 As shown, the terminal device 101 obtains the storage space information of each source storage node in the message queue, wherein the data segments in the message queue are stored in each source storage node; the terminal device determines the data to be migrated in each source storage node based on the storage space information of the source storage node; and concurrently migrates the data to be migrated in each source storage node to the target storage node, wherein the target storage node is a storage node newly added to the message queue or a storage node whose space utilization in the message queue is lower than the storage threshold; finally, the terminal device clears the data to be migrated in each source storage node.

[0052] It is understandable that the embodiments of the present application Figure 1The example illustrates one of the application scenarios, that is, the terminal device actively rebalances the message queue without the user actively inputting relevant content. In other application scenarios, the user can actively input a control instruction to the terminal device, which requires the terminal device to rebalance the message sequence. The embodiment of this application does not limit the specific application scenario. Figure 1 The application scenarios are only for illustration.

[0053] Reference Figure 2 , Figure 2 A flowchart of a message queue rebalancing method provided in an embodiment of the present application, which can be applied to Figure 1 The terminal device 101 in FIG. Figure 2 The message queue rebalancing method of the embodiment of the present application can be executed by a terminal device, or can be executed on any device that has data processing capabilities and can run the message queue of the embodiment of the present application, for example, it can be executed on a personal computer. The method specifically includes but is not limited to steps S100-S400:

[0054] S100: Acquire storage space information of each source storage node in a message queue, wherein data segments in the message queue are stored in each source storage node;

[0055] Specifically, "source" refers to the origin. Since the embodiments of the present application require rebalancing the data in the message queue, the storage location of the data may change during the rebalancing process. Therefore, the storage node of the data in the message queue before rebalancing is called the source storage node, and the data in the message queue is stored in the form of segments in each source storage node. In the embodiments of the present application, the terminal device checks each source storage node in the message queue one by one and obtains the storage space information of the source storage node. The storage space information is used to indicate the amount of data accumulated in the source storage node.

[0056] S200: Determine the data to be migrated in each source storage node according to the storage space information of the source storage node;

[0057] Specifically, since the embodiment of the present application needs to rebalance the data in the message queue, during the rebalancing process, some data needs to be migrated from a source storage node with less remaining storage space to another storage node with more remaining storage space. Therefore, this portion of data that needs to change its storage location is referred to as to-be-migrated data. The embodiment of the present application determines the to-be-migrated data that needs to be migrated in each source storage node based on the storage space information of each source storage node obtained in step S100.

[0058] S300: Concurrently migrate the data to be migrated in each source storage node to a target storage node, where the target storage node is a storage node newly added to the message queue or a storage node whose space utilization in the message queue is lower than a storage threshold;

[0059] Specifically, in an embodiment of the present application, concurrent migration means that within the same time period, data to be migrated from several source storage nodes are being migrated. As can be seen from step S100, the data is stored in each source storage node in a segmented form, so the same source storage node can have one or more segments of data to be migrated, and multiple segments of data to be migrated can be migrated to the same target storage node, or they can be migrated to different target storage nodes respectively. Since the embodiment of the present application needs to rebalance the data in the message queue, that is, migrate the data from the storage node with higher space utilization to the storage node with lower space utilization, the target storage node is the storage node newly added to the message queue or the storage node in the message queue whose space utilization is lower than the storage threshold. The storage threshold can refer to the maximum space utilization of each storage node, or it can be the space utilization set in advance by the user, which is not limited here.

[0060] S400: Clear the data to be migrated in each source storage node;

[0061] Specifically, after migrating the data to be migrated from the source storage node to the target storage node through step S300, it is necessary to clear the data to be migrated in the source storage node to reduce the amount of data accumulated in the source storage node and release the storage pressure of the source storage node. In the embodiment of the present application, the operation of clearing the data to be migrated can be actively triggered by the user, or periodically triggered by the terminal device. For example, after completing the data migration, the user sends a data clearing instruction to the terminal device. The data clearing instruction refers to an instruction received by the terminal device to perform data clearing. When the terminal device is a smart phone, the data clearing instruction can be, for example, a click instruction, a long press instruction, or a voice instruction; when the terminal device is a personal computer, the data clearing instruction can be, for example, a command code entered by a console. The data clearing instruction is used to clear the data to be migrated in the source storage node.

[0062] Through steps S100-S400, the embodiment of the present application realizes rebalancing of the accumulated data of each storage node in the message queue.

[0063] The embodiment of the present application proposes a message queue rebalancing method, which first obtains the storage space information of each source storage node in the message queue, wherein the data in the message queue is stored in each source storage node in the form of data segments, and then determines the data to be migrated in each source storage node based on the storage space information of the source storage node, and then concurrently migrates the data to be migrated in each source storage node to the target storage node, and finally clears the data to be migrated in each source storage node. The embodiment of the present application refines the partition storage method in the traditional message queue into segmented storage, which can reduce the amount of data during the data migration process and improve the rebalancing efficiency of the message queue; in addition, the data to be migrated in the embodiment of the present application is in the form of segmented storage, so there is no master-slave partition division, and the content of any data segment can be migrated and balanced without restriction, which improves flexibility; at the same time, the embodiment of the present application can concurrently migrate the data to be migrated to the target storage node, which improves the data migration efficiency; in addition, the target storage node in the embodiment of the present application is a storage node newly added to the message queue or a storage node whose space utilization in the message queue is lower than the storage threshold, which can make full use of the node with less storage pressure in the message queue for rebalancing, thereby improving the utilization of the storage node.

[0064] Figure 3 The flowchart of obtaining the storage space information of the source storage node provided in the embodiment of the present application, in some embodiments, the step of obtaining the storage space information of each source storage node in the message queue in the above step S100 is as follows: Figure 3 As shown, the following steps S110-S120 are included:

[0065] S110, obtaining the space utilization rate of each source storage node in the message queue;

[0066] Specifically, in some embodiments, the storage space information includes space utilization and average utilization. Space utilization refers to the proportion of storage space occupied by data in the source storage node. For example, Figure 1 In the example above, the space utilization of the first storage node is 95%, while the space utilization of the newly added second storage node is 0% when no data is stored. Terminal devices can directly obtain disk utilization through code to determine the space utilization of each source storage node in the message queue. Source storage spaces with high space utilization have a large amount of data accumulated, so data rebalancing is necessary.

[0067] S120. Calculate the average utilization of all storage nodes based on the space utilization of each source storage node and the space utilization of the target storage node.

[0068] Specifically, the average utilization refers to the average of the space utilization of all storage nodes in the message queue. It is understandable that in the rebalancing process of the message queue of the embodiment of the present application, if there is a newly added storage node as the target storage node, then the average utilization of all existing source storage nodes and the newly added target storage node needs to be calculated when calculating the average utilization. For example, Figure 1 For example, if the existing first storage node is the source storage node and its space utilization is 95%, and the space utilization of the newly added second storage node is 0%, then the average utilization of the storage nodes in the message queue is (95% + 0%) / 2 = 47.5%. If no new storage nodes are added during the rebalancing process of the message queue, the average utilization of all storage nodes in the current message queue is calculated as the average utilization of the storage nodes.

[0069] Through steps S110 - S120 , the terminal device obtains the storage space information of each source storage node in the message queue, and determines the space utilization and average utilization of the storage node.

[0070] Figure 4 The flowchart of determining the data to be migrated provided in the embodiment of the present application, in some embodiments, the step of determining the data to be migrated in each source storage node according to the storage space information of the source storage node in the above step S200, such as Figure 4 As shown, the following steps S210-S220 are included:

[0071] S210: Generate a data migration instruction based on the space utilization and average utilization of each source storage node;

[0072] Specifically, a data migration instruction refers to an instruction received by a terminal device to migrate data from a current storage node to another storage node. When the terminal device is a smart phone, the data migration instruction may be, for example, a click instruction, a long press instruction, or a voice instruction; when the terminal device is a personal computer, the data migration instruction may be, for example, a command code entered by a console. It is understandable that the data migration instruction may also be triggered by the terminal device itself according to the data accumulation in the message queue. Among them, the data migration instruction includes a preset amount of data to be migrated. In other words, the user can pre-set the amount of data to be migrated in this rebalancing process as needed. It is understandable that since the space utilization of each source storage node is not necessarily the same, the amount of data required to be migrated by the data storage instruction may also be different for different source storage nodes.

[0073] S220: Determine the data to be migrated in each source storage node according to the preset amount of data to be migrated;

[0074] Specifically, since the data in the message queue is stored in the storage node in the form of segments, according to the preset amount of data to be migrated, the source storage node selects several segments of data in the storage space as the data to be migrated.

[0075] Through steps S210-S220, the embodiment of the present application determines the amount of data to be migrated in each source storage node according to the data migration instruction received by the terminal device, and determines the data to be migrated in each source storage node according to the amount of data to be migrated.

[0076] In some embodiments, the step of determining the data to be migrated in each source storage node based on the storage space information of the source storage node in step S200 may also include: determining the data segments in each source storage node that exceed the average utilization as data to be migrated based on the space utilization and average utilization of each source storage node.

[0077] Specifically, the space utilization rate of each source storage node is obtained by step S110, and the average utilization rate of all storage nodes in the message queue is calculated by step S120. Since the rebalancing method proposed in the embodiment of the present application is used to balance the space utilization rate of each storage node in the message queue, if the space utilization rate of a source storage node in the message queue is higher than the average utilization rate, it means that the data accumulation amount of the current source storage node is large, that is, the data segment that exceeds the average utilization rate in the current source storage node is determined as the data to be migrated. It can be understood that after migrating the data to be migrated in the current source storage node to other storage nodes, the space utilization rate of the current source storage node will be equal to or lower than the average utilization rate of all storage nodes in the message queue, and the space utilization rate of the storage node receiving the data to be migrated will increase, closer to the average utilization rate, thereby achieving the purpose of releasing the storage pressure of the current source storage node and balancing the data volume of each storage node.

[0078] It is understandable that if the current source storage node has a large amount of data to be migrated, these data to be migrated can be migrated to multiple target storage nodes for separate storage. For example, the data to be migrated on the current source storage node include data segment 1 and data segment 2. If data segment 1 is migrated to the first target storage node, the space utilization rate of the first target storage node reaches or is close to the average utilization rate. If data segment 2 is migrated to the first target storage node, the space utilization rate of the first target storage node will exceed the average utilization rate, which is inconsistent with the purpose of rebalancing the message queue data proposed in this application. Therefore, the embodiment of this application migrates data segment 1 to the first target storage node and data segment 2 to other target storage nodes, thereby achieving the purpose of balancing the space utilization rates of each storage node in the message queue.

[0079] It should be noted that the data in the message queue in the embodiment of the present application is stored in segments in the storage node. Figure 5 , Figure 5 Schematic diagram of data segmented storage in storage nodes provided by the embodiment of the present application. In the method mentioned above for determining data segments exceeding the average utilization rate as data to be migrated, for example, Figure 5 As shown, assuming that the current storage node stores 4 data segments 1-4, if the total data volume of data segment 1, data segment 2, data segment 3 and data segment 4 exceeds the average utilization of the current message queue, then assuming that the calculation starts from data segment 4, the total data volume of data segment 4, data segment 3 and data segment 2 is calculated. When the space utilization of this data volume is equal to or lower than the average utilization of the current message queue, data segment 1 is determined to be the data segment that exceeds the average utilization, and data segment 1 is determined to be the data to be migrated. In the subsequent data migration step, data segment 1 is migrated to other storage nodes, thereby reducing the space utilization of the current storage node and relieving the pressure on the current storage node.

[0080] Figure 6 The flowchart of concurrently migrating the data to be migrated to the target storage node provided in the embodiment of the present application, in some embodiments, the step of concurrently migrating the data to be migrated in each source storage node to the target storage node in step S300 is as follows: Figure 6 As shown, the following steps S310-S330 are included:

[0081] S310, adding the data to be migrated in each source storage node to a list to be migrated;

[0082] Specifically, the embodiment of the present application can traverse all source storage nodes in the message queue, determine the data to be migrated in each source storage node according to step S200, and add the data to be migrated in each source storage node to the list to be migrated.

[0083] S320: Start multiple parallel threads according to the list to be migrated;

[0084] Specifically, since data is stored in the storage node in a segmented form, multiple parallel threads can be started for the data to be migrated in the to-be-migrated list. Multiple data to be migrated are migrated concurrently within the same time period, thereby accelerating the migration process.

[0085] S330, migrating the data to be migrated in the corresponding source storage node to the target storage node according to the parallel thread;

[0086] Specifically, since the data is stored in the storage nodes in a segmented manner, after multiple parallel threads are started in step S320, the data to be migrated is migrated from the current source storage node to the target storage node in the corresponding thread. It will be appreciated that different data to be migrated from the same source storage node can be migrated to the same target storage node via the same process, or to different target storage nodes via different parallel processes.

[0087] It can be understood that by traversing each source storage node in the message queue through steps S310-S330, the data to be migrated in each source storage node can be concurrently migrated to the target storage node through parallel threads, thereby achieving the purpose of releasing the storage pressure of the storage node with higher space utilization, and achieving the purpose of balancing the space utilization of each storage node in the message queue.

[0088] Figure 7 The flowchart of migrating the data to be migrated to the target storage node concurrently according to the parallel threads provided in the embodiment of the present application, in some embodiments, the step of migrating the data to be migrated in the corresponding source storage node to the target storage node according to the parallel threads in the above step S330 is as follows: Figure 7 As shown, the following steps S331-S334 are included:

[0089] S331. Obtain the source data segment number of the data to be migrated and the source storage node number where the data to be migrated is located;

[0090] Specifically, in order to distinguish between multiple storage nodes in a message queue and multiple data segments in a storage node, the storage nodes can be numbered when the storage nodes are divided, and the data segments are also numbered when the data is written to the storage nodes. In this way, in the data rebalancing process of the message queue proposed in this application, it is necessary to obtain the source data segment number of the data to be migrated and the source storage node number where the data to be migrated is located. Among them, the source storage node number is the number of the storage node where the data to be migrated is currently located; and the source data segment number is the current data segment number of the data to be migrated. Since the data to be migrated needs to be migrated, the data segment number will also change after the data is migrated to the new storage node. Therefore, what is obtained is the current source data segment number of the data to be migrated.

[0091] S332. Obtain the target data segment number and target storage node number of the data to be migrated;

[0092] Specifically, as mentioned in step S331, after the data to be migrated is migrated, the storage node where the data to be migrated is located and the data segment number of the data to be migrated are changed. Therefore, when performing rebalancing, the embodiment of the present application needs to first obtain the target data segment number and target storage node number of the data to be migrated. The target data segment number is the new data segment number corresponding to the data to be migrated after migration, and the target storage node number is the number of the target storage node to which the data to be migrated is to be migrated.

[0093] S333. Obtain the data to be migrated from the source data segment number and the source storage node number according to the parallel thread;

[0094] Specifically, the data of the embodiment of the present application can be migrated concurrently according to multiple parallel threads. In the message queue, the storage node from which the data needs to be migrated is determined based on the obtained source storage node number, the data to be migrated is determined based on the obtained source data segment number, and the storage location of the data to be migrated is migrated according to the parallel threads.

[0095] S334, migrating the data to be migrated to the target storage node where the target data segment number and the target storage node number are located;

[0096] Specifically, since the embodiment of the present application is to migrate the data to be migrated from the source storage node to the target storage node, the target storage node to which the data to be migrated is determined based on the obtained target storage node number, and the new data segment number of the migrated data is determined based on the target storage segment number, thereby achieving the purpose of migrating the data to be migrated from the source storage node to the target storage node.

[0097] Through steps S331-S334, the data to be migrated is obtained according to the source data segment number and source storage node number of the data to be migrated, and through multiple parallel processes, the data to be migrated is migrated to the target storage node according to the target storage node number, and the target data segment number of the data migrated to the target storage node is determined.

[0098] In some embodiments, the list to be migrated mentioned in step S310 includes a list to be copied, which is used to start multiple parallel threads, and according to the parallel threads, the data to be migrated in the corresponding source storage node is copied to the target storage node where the target data segment number and the target storage node number are located. Specifically, in the process of migrating the data to be migrated from the source storage node to the target storage node, the data to be migrated in the source storage node is copied to obtain several copies, and the copies of the data to be migrated are copied to the target storage node through parallel threads. It can be understood that a certain amount of data redundancy can improve the reliability of data storage, so the same data to be migrated can be copied into multiple copies, and the copies are copied to different target storage nodes through different parallel threads to achieve concurrent migration of multiple data to be migrated.

[0099] Figure 8 The schematic diagram of data partition migration and data segment migration provided in the embodiment of the present application is as follows: Figure 8 The data partition migration scheme in the related art and the data segment migration scheme proposed in the embodiment of the present application are shown in the figure. The reference numeral 800 represents a storage node with high space utilization in the message queue. The storage node 800 needs to migrate out some data. The reference numerals 810 and 820 respectively represent two vacant storage nodes newly added to the message queue. The dotted box of reference numeral 830 is the data partition migration scheme in the related art. The reference numeral 840 represents the data storage area in the related art. Figure 8 As shown, since data migration is performed in partitions in the related art, data storage area 840 is copied entirely to storage node 820 through partition replication. However, if the data storage capacity of data storage area 840 is relatively large, copying and migrating the entire partition will take a long time and is not flexible enough. Secondly, since data is written to the data storage area sequentially, in order to ensure the order of data within the partition, a single data storage area cannot be copied and migrated concurrently, which will also affect the speed of data migration. In addition, in the related art, different data storage areas are divided into master and slave. When the master partition is copied and migrated, it will affect the normal writing of the client and affect the business process. The dotted box labeled 850 is the data segmentation migration scheme proposed in the embodiment of the present application, which stores the data in the data storage area 840 in the form of segments. Labels 850, 860 and 870 respectively represent the data segments that need to be migrated out of the data storage area 840. Since in the embodiment of the present application, the data is stored in the storage node in the form of segments, one or more data segments can be selectively copied for data migration during the data copy and migration process. Therefore, compared with the data migration scheme based on partitions in the related art, the scheme for migrating data segments based on segments proposed in the embodiment of the present application has smaller granularity. The data of the same partition can be copied to multiple target storage nodes, and the copying process of the data segments can be carried out concurrently, thereby speeding up the migration speed and facilitating the rapid completion of data migration. Moreover, the data migration based on data segments will have a lighter impact on the reading and writing of client data, and the effect of smooth migration can be achieved as much as possible, that is, the rebalancing of the accumulated data is completed as much as possible without the business perception. In addition, since there is no master-slave relationship between the replicated copies during data segment replication and migration, no matter which data segment is migrated, there is no need to select the master copy. This ensures that during the data migration process, the reading and writing of client data is not affected by the master copy migration. For example, Figure 8 As shown, data segment 850 can be migrated to storage node 810, and data segment 860 and data segment 870 can be concurrently copied to storage node 820, thereby completing the rebalancing of data accumulated in the storage nodes in the message queue.

[0100] In some embodiments, message queue rebalancing can be achieved through Pulsar messaging middleware. In related technologies, there are two typical methods for rebalancing accumulated data: Kafka's accumulation rebalancing solution and Pulsar's accumulation rebalancing solution. These two methods are briefly introduced below.

[0101] 1. In related technologies, message queues often use the Kafka messaging middleware to implement data rebalancing. Message queues store data in partitions, organized by topic. A topic can be thought of as a large file that can only be written to and cannot be deleted or modified. When a Kafka cluster needs to be expanded, data from existing topics in the message queue continues to be written to the old storage nodes. New storage nodes can only handle the load of the newly added topics. Therefore, even with the addition of more nodes, data migration is still required: data from storage nodes with large data volumes must be moved to storage nodes with smaller data volumes. Furthermore, because Kafka stores data in partitions (i.e., topics), migration requires migrating the entire partition to the new storage node. This approach is inflexible and slows down data migration. Furthermore, Kafka partitions are divided into master and slave partitions. Write operations can only be performed on the master partition and then synchronized to the slave partitions. Therefore, the migration of the master partition may affect client writes.

[0102] 2. In related technologies, message queues also use the Pulsar messaging middleware to implement data rebalancing. Pulsar uses a segmented storage method, which offers finer granularity than partitioned storage methods. Therefore, in scenarios with sufficient expansion resources, Pulsar can rebalance data by adjusting the storage node selection for each segment, eliminating the need for data migration. Pulsar first addresses this issue by addressing the source, avoiding unevenness. Specifically, when each segment is written, it selects replica storage nodes using a method such as round-robin. Secondly, when an imbalance in the message queue occurs, it implements a resource-aware placement strategy to prioritize storage nodes with more free space to ensure traffic balance across all storage nodes in the cluster. However, this Pulsar rebalancing method is based on scenarios with sufficient expansion resources. If the number of newly added nodes falls short of the agreed-upon number of replicas, new data will fail to be written successfully, severely impacting the overall availability of the message queue.

[0103] Based on the deficiencies of the solutions in the above two related technologies, the embodiment of the present application proposes to achieve rebalancing of the message queue through Pulsar message middleware and Zookeeper server cluster.

[0104] Figure 9 The composition diagram of the topic provided in the embodiment of the present application. In the above embodiment, it is mentioned that the topic can be regarded as a large file that can only be written but not deleted or modified. Since the data in the embodiment of the present application is migrated in a segmented form, that is, it is migrated in segments, the topic is divided into multiple segments, and the segments are written to the topic in sequence, that is, Figure 9 The order of data segments in the topic shown is segment1 to segment6.

[0105] Figure 10 This is a schematic diagram of the message queue rebalancing method provided in an embodiment of the present application. In the above embodiment, it is mentioned that segments are arranged in order in the topic. For example, Figure 10 Storage nodes 1-3 correspond to one topic each, and each topic includes four segments. The terminal device migrates segment1 from storage node 1 to the new node 4, migrates segment2 from storage node 2 to the new node 4, and migrates segment3 from storage node 3 to the new node 4. After the three segments are migrated to the new node 4, the metadata of segment1, segment2, and segment3 in Zookeeper are updated. When the user actively triggers or the terminal device periodically triggers the garbage collection thread, the metadata in Zookeeper is queried. If it is found that there are several segments in the old node whose corresponding metadata cannot be found in Zookeeper, the garbage collection thread will delete one or more segments.

[0106] The following describes how the embodiment of the present application specifically implements message queue rebalancing through Pulsar message middleware.

[0107] like Figure 11 As shown, Figure 11 A message queue rebalancing method based on a Pulsar environment is provided in an embodiment of the present application. The method can be applied to the above Figure 1 This is achieved by using the terminal device. Figure 11 The message queue rebalancing method can be implemented by the Pulsar message middleware in the terminal device. The method specifically includes but is not limited to steps S1100-S1130:

[0108] S1100: Data in the message queue is segmented and stored in each storage node according to the Pulsar message middleware;

[0109] Specifically, in the embodiment of the present application, the data storage of the Pulsar message middleware adopts a segmentation method, that is, the data in the topic is divided into several data segments and stored in each storage node.

[0110] S1110. Obtain data of each partition in the source storage node;

[0111] Specifically, through step S100, information about the stored data of each partition in the source storage space is obtained, including but not limited to the accumulated amount of data stored in each partition.

[0112] S1120: Determine multiple segments of data to be migrated from the data in each partition, or determine data in any partition as data to be migrated;

[0113] Specifically, through step S200, the data to be migrated is determined from the partitions of the source storage node. It is understandable that the embodiment of the present application needs to relieve the storage pressure of the storage node with a large amount of data accumulation, and there are multiple partitions in the storage node. Therefore, the data to be migrated can be determined from multiple partitions in the source storage node, that is, a portion of the data is migrated from different partitions of the current storage node respectively; or the data to be migrated can be determined from any partition in the source storage node, that is, a portion of the data is migrated from a specified partition of the current storage node, thereby achieving the purpose of relieving the storage pressure of the current storage node.

[0114] S1130, concurrently migrating the data to be migrated to multiple different target storage nodes;

[0115] Specifically, since the partitioned data in Pulsar can be migrated to the target storage node concurrently, the data to be migrated determined in step S1120 can be concurrently migrated from the current source storage node to the corresponding target storage node. In the embodiment of the present application, the partitioned data in Pulsar is migrated to the target storage node using a replication method, so multiple copies of the data to be migrated can be concurrently migrated to different target storage nodes, improving the redundancy of the data to ensure the reliability of data storage in the message queue and increasing the migration speed. In addition, since there is no master-slave distinction between the copies, the migration of the data segment copy does not involve re-election of the master, and therefore does not affect the client's data writing process. At the same time, the data segment migration solution with a smaller granularity is conducive to reducing business perception and achieving smooth data migration as much as possible.

[0116] Through steps S1100-S1130, the embodiment of the present application uses Pulsar message middleware to achieve rebalancing of the message queue, that is, to achieve the purpose of concurrently migrating the segmented stored data to be migrated to multiple different target storage nodes.

[0117] In addition, the embodiment of the present application also realizes the migration of data to be migrated through the Zookeeper terminal device cluster in conjunction with the Pulsar message middleware. In the above embodiment, the step of adding the data to be migrated in each source storage node to the list to be migrated in step S310 mentions that the list to be migrated includes a list to be copied, multiple parallel threads are started through the list to be copied, and the data to be migrated is copied to the target storage node through the copies of the parallel threads. And step S334 mentions migrating the data to be migrated to the target storage node where the target data segment number and the target storage node number are located. The target data segment number (also called segment id) and the target storage node number are called the metadata of the segment, and in the embodiment of the present application, the metadata of the segment is stored in Zookeeper, that is, when no data migration is performed, the metadata of the segment is consistent with the metadata stored in Zookeeper. After the copy of the data to be migrated is copied to the target storage node, the data segment number and storage node number of the data to be migrated copied in the target storage node are updated through the Zookeeper terminal device cluster, that is, the updated metadata is stored in Zookeeper.

[0118] Reference Figure 12 , Figure 12 The flowchart of an embodiment of the present application provides a method for updating metadata of data to be migrated in a target storage node through a Zookeeper terminal device cluster. The method specifically includes but is not limited to steps S1200-S1210:

[0119] S1200. Modify the data segment number of the data to be migrated in the Zookeeper terminal device cluster to the target data segment number of the current target storage node;

[0120] Specifically, the Zookeeper terminal device cluster stores the data segment number of the data to be migrated, that is, the source data segment number. Therefore, during the update, the source data segment number will be deleted in Zookeeper and replaced with the target data segment number of the current target storage node.

[0121] S1210. Modify the storage node number of the data to be migrated in the Zookeeper terminal device cluster to the target storage node number of the current target storage node;

[0122] Specifically, the Zookeeper terminal device cluster stores the storage node number of the data to be migrated, that is, the source storage node number. Therefore, during the update, the source storage node number will be deleted in Zookeeper and replaced with the target storage node number of the current target storage node.

[0123] Through steps S1200-S1210, the embodiment of the present application updates the metadata of the data replica to be migrated through the Zookeeper terminal device cluster, that is, updates the data segment number and storage node number of the data to be migrated in Zookeeper.

[0124] Figure 13 The flowchart of clearing the data to be migrated in each source storage node provided in the embodiment of the present application, in some embodiments, the step of clearing the data to be migrated in each source storage node in the above step S400 is as follows: Figure 13 As shown, the following steps S410-S420 are included:

[0125] S410: After the data to be migrated is concurrently migrated to the target storage node, a data clearing instruction is generated;

[0126] Specifically, a data purge instruction refers to an instruction received by a terminal device to purge data to be migrated from a source storage node. In embodiments of the present application, the data purge instruction is used to purge data to be migrated from a source storage node after the data to be migrated has been concurrently migrated to a target storage node. It should be noted that the data purge instruction can be triggered proactively by a user or periodically by a terminal device.

[0127] S420: Clear the data to be migrated in each source storage node according to the data clearing instruction.

[0128] Specifically, the embodiments of the present application traverse each source storage node in the message queue according to the data clearing instruction and clear the data to be migrated from the source storage node. It is understandable that the data to be migrated has already been migrated to the target storage node, and clearing the data to be migrated from the source storage node does not affect the storage status of the target storage node. After clearing the data to be migrated from the source storage node, the data accumulation in the source storage node decreases, the space utilization rate decreases, and the storage capacity between the various storage nodes in the message queue tends to be balanced.

[0129] Through steps S410-S420, after the data to be migrated is migrated, the data to be migrated in the source storage node is cleared according to the data clearing instruction, thereby releasing the storage pressure of the source storage node and balancing the storage capacity among the storage nodes in the message queue.

[0130] As mentioned in the above embodiment, in the embodiment of the present application, Zookeeper stores the metadata of the segment, so in some embodiments, the data to be migrated can be cleared by cooperating with the Zookeeper terminal device cluster. Figure 14 , Figure 14The flowchart of clearing the data to be migrated in each source storage node based on Zookeeper provided in the embodiment of the present application, in some embodiments, the step S420 above clears the data to be migrated in each source storage node according to the data clearing instruction, such as Figure 14 As shown, the following steps S421-S422 are included:

[0131] S421. Obtain metadata information of the data to be migrated in the Zookeeper terminal device cluster, where the metadata information includes a data segment number and a storage node number;

[0132] Specifically, as mentioned in the above embodiment, the metadata information of the data to be migrated is stored in the Zookeeper terminal device cluster. The metadata information includes the data segment number and storage node number of the data to be migrated.

[0133] S422: When the storage node number of the data to be migrated is different from the source storage node number, clear the data to be migrated on the corresponding source storage node.

[0134] Specifically, in steps S1000-S1010, the metadata information of the data to be migrated stored in the Zookeeper terminal device cluster is updated. When the terminal device receives a data clearing instruction and clears the data to be migrated in each source storage node, each source storage node traverses the segments stored in itself. When it is found that the metadata of the segment in the current source storage node is different from the metadata stored in the Zookeeper terminal device cluster, that is, the metadata corresponding to the segment cannot be found in the Zookeeper terminal device cluster, it means that the segment is the data to be migrated whose metadata has been updated in steps S1000-S1010. Since the data to be migrated has been copied to the corresponding target storage node, the segment in the current source storage node can be deleted. At this time, the terminal device marks the segment as invalid garbage data and waits for the garbage collection thread to delete the segment.

[0135] In summary, the message queue rebalancing method of the embodiment of the present application can obtain the storage space information of each source storage node in the message queue, wherein the data in the message queue is stored in each source storage node in the form of data segments, and then the data to be migrated in each source storage node is determined based on the storage space information of the source storage node, and then the data to be migrated in each source storage node is concurrently migrated to the target storage node, and finally the data to be migrated in each source storage node is cleared. The embodiment of the present application can concurrently migrate the segmented stored data to be migrated from the source storage node with a large amount of data accumulation to the target storage node, thereby improving the efficiency of data migration; in the case of a lack of expansion resources, it can make full use of the nodes with less storage pressure in the message queue for rebalancing, thereby improving the utilization rate of the storage nodes in the message queue.

[0136] Based on the message queue rebalancing method mentioned in the above embodiment, the following describes the process of migrating a single segment based on the Pulsar environment. Figure 15 , Figure 15 A flowchart of a single segment migration based on a Pulsar environment is provided in an embodiment of the present application. The method includes but is not limited to steps S1500-S1540.

[0137] S1500. Select the segno, Saddr, and Daddr to be copied.

[0138] Specifically, an embodiment of the present application selects the data segment number (segno), source storage node number (Saddr) and target storage node number (Daddr) of the segment that needs to be copied and migrated, so as to determine the data to be migrated based on segno, Saddr and Daddr.

[0139] Specifically, as mentioned in the above embodiment, through steps S100-S120, the embodiment of the present application determines the data to be migrated in the source storage node based on the storage space information of each source storage node in the message sequence, that is, determines the segment that needs to be copied and migrated in each source storage node. Exemplarily, for example, if the storage node number of the current source storage node is 1, then Saddr is 1, and the segno of the segment that needs to be migrated in the current node is 0001. The segment will be copied and migrated to the target storage node with the target storage node number of 2, then Daddr is 2, and so on. All source storage nodes are traversed to select the segno, Saddr, and Daddr of the segment that needs to be copied and migrated. It can be understood that Saddr and Daddr indicate the migration path of the segment. In parallel threads, segments of different segno are concurrently migrated from the storage node shown by Saddr to the storage node shown by Daddr.

[0140] S1510, concurrently read multiple copies of segno and obtain the first returned result;

[0141] Specifically, using the Pulsar concurrent data reading method, multiple copies are read according to segno, and the first returned result is obtained. The returned result refers to one of the copies of a single segment. For example, if the segno of the segment that needs to be migrated is 1, and the segment needs to be migrated to three different target storage nodes, then the segment needs to replicate copy 1, copy 2, and copy 3. The above three copies are read through segno 1, and the first returned copy is obtained. For example, the first returned copy is copy 2, which means that the reading of copy 2 is completed first, and copy 2 is operated first in subsequent steps. It can be understood that the above embodiment sorts the copies according to the order in which the copies are read. The embodiment of the present application can also sort according to the order of the copies themselves, that is, after completing the reading of copy 1, copy 2, and copy 3, subsequent operations are performed in the order of copy 1, copy 2, and copy 3. The embodiment of the present application is not limited to this.

[0142] It should be noted that replica 1, replica 2, and replica 3 do not indicate a master-slave relationship between replicas. The numerical labels are only used for sorting purposes.

[0143] S1520, write the read result into Daddr;

[0144] Specifically, the returned result, that is, one of the copies of the segment, is obtained, and the copy is written to the destination address Daddr to complete the copy migration of the segment.

[0145] S1530. In the Zookeeper server cluster, replace the metadata of segno from Saddr to Daddr.

[0146] Specifically, in the above step S1520, the embodiment of the present application writes a copy of the read segment into Daddr, that is, the segment has been successfully migrated into the target storage node referred to by Daddr, and then the Saddr of the metadata is updated to Daddr according to segno in the Zookeeper server cluster, that is, the storage node number of the segment is updated, indicating that the storage node of the segment has changed, and at the same time, the space utilization of the storage node shown by Daddr is improved.

[0147] S1540: Trigger a garbage collection thread to recycle the copy data of segno stored in Saddr.

[0148] Specifically, when all copies of a single segment that needs to be migrated have been migrated, the segment remaining in the source storage node (the node number of the source storage node is Saddr) needs to be cleared. After the replication migration of the segment copies is completed, the user or the terminal device periodically triggers the garbage collection thread to clear the copy data of the segno stored in Saddr. For example, for example, the segno of the segment is 0001, Saddr is 1, and a copy of the segment needs to be migrated to the target storage node with the storage node number Daddr being 2. Since the copy of the segment has completed the copy migration in step S1520, in step S1530, according to the segno being 0001, the metadata Saddr is updated to Daddr, that is, the Saddr of the segment with segno being 0001 is updated from 1 to 2. Then, in the garbage collection thread, all segments in the storage node with Saddr being 1 are traversed, and the metadata with segno being 0001 cannot be found in Zookeeper. Therefore, for the storage node with Saddr being 1, the data segment with segno being 0001 is judged to be an invalid data segment and is cleared by the garbage collection thread, thereby reducing the space utilization of the storage node with Saddr being 1 and releasing the storage pressure.

[0149] Through steps S1500-S15140, the embodiment of the present application migrates a single segment based on the Pulsar environment, determines the segment to be migrated through the obtained segno, determines the migration path of the segment through the obtained Saddr and Daddr, reads the copy of the segment, obtains the first returned copy and writes the copy to the target storage node indicated by Daddr according to Daddr, and modifies the metadata of the segment in Zookeeper; scans all segments in the source storage node through user triggering or periodic triggering of the terminal device. Since the metadata of the segment that has completed the copy migration has been updated, the corresponding metadata cannot be found in Zookeeper. Therefore, for the current source storage node, the segment that has completed the copy migration will be judged as an invalid data segment and will be cleared by the garbage collection thread, thereby utilizing the free storage space of the target storage node to release the pressure of the current source storage node.

[0150] Figure 16 A schematic diagram of the structure of a message queue rebalancing device provided in an embodiment of the present application. The rebalancing device 1600 can be applied to Figure 1The terminal device 101 shown, for example, can implement part or all of the functions of the terminal device 112 through hardware or a combination of hardware and software to implement the steps in the above rebalancing method. Figure 16 As shown, the rebalancing device may include:

[0151] An acquisition module 1610 is configured to acquire storage space information of each source storage node in a message queue, wherein data segments in the message queue are stored in each source storage node;

[0152] A concurrent migration module 1620 is configured to concurrently migrate the data to be migrated from each source storage node to a target storage node, wherein the target storage node is a storage node newly added to the message queue or a storage node whose space utilization in the message queue is lower than a storage threshold;

[0153] The clearing module 1630 is configured to clear the data to be migrated from each source storage node.

[0154] In summary, the rebalancing device provided by the embodiment of the present application can obtain the storage space information of each source storage node in the message queue, wherein the data in the message queue is stored in each source storage node in the form of data segments, and then the data to be migrated in each source storage node is determined based on the storage space information of the source storage node, and then the data to be migrated in each source storage node is concurrently migrated to the target storage node, and finally the data to be migrated in each source storage node is cleared. The embodiment of the present application can concurrently migrate the segmented stored data to be migrated from the source storage node with a large amount of data accumulation to the target storage node, thereby improving the efficiency of data migration; in the case of a lack of expansion resources, it can make full use of the nodes with less storage pressure in the message queue for rebalancing, thereby improving the utilization rate of the storage nodes in the message queue.

[0155] An embodiment of the present application further provides an electronic device, the device comprising a processor and a memory;

[0156] The memory stores a program;

[0157] The processor executes the program to perform Figure 1 The rebalancing method of the terminal device 101 is shown.

[0158] The electronic device has the function of running the message queue proposed in the embodiment of the present application, such as a personal computer (PC), a mobile phone, a smart phone, a personal digital assistant (PDA), a wearable device, a handheld computer PPC (Pocket PC), a tablet computer, etc.

[0159] In the embodiment of the present application, the processor included in the terminal device has the following functions:

[0160] Acquire storage space information of each source storage node in the message queue, wherein data segments in the message queue are stored in each source storage node;

[0161] Determine the data to be migrated in each source storage node based on the storage space information of the source storage node;

[0162] Concurrently migrate the data to be migrated in each source storage node to the target storage node, where the target storage node is a storage node newly added to the message queue or a storage node whose space utilization in the message queue is lower than the storage threshold;

[0163] Clear the data to be migrated from each source storage node.

[0164] The present application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 2 The message queue rebalancing method shown.

[0165] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0166] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art will be able to implement the present invention as set forth in the claims using ordinary skill without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0167] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0168] The logic and / or steps represented in the flowcharts or otherwise described herein, which may be considered, for example, as a queued list of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0169] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0170] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0171] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0172] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0173] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A message queue rebalancing method, characterized in that: include: Get the space utilization of each source storage node in the message queue; Calculating an average utilization rate of all storage nodes based on the space utilization rate of each source storage node and the space utilization rate of the target storage node, wherein the storage nodes include the source storage node and the target storage node, and the data segments in the message queue are stored in each source storage node; Determining the data to be migrated in each of the source storage nodes according to the storage space information of the source storage nodes; Concurrently migrating the data to be migrated in each of the source storage nodes to a target storage node, wherein the target storage node is a storage node newly added to the message queue or a storage node whose space utilization in the message queue is lower than a storage threshold; Clearing the data to be migrated from each of the source storage nodes; The step of determining the data to be migrated in each of the source storage nodes according to the storage space information of the source storage nodes includes: According to the space utilization rate of each source storage node and the average utilization rate, data segments in each source storage node that exceed the average utilization rate are determined as data to be migrated.

2. A message queue rebalancing method according to claim 1, characterized in that: The step of determining the data to be migrated in each of the source storage nodes according to the storage space information of the source storage nodes further includes: generating a data migration instruction according to the space utilization rate of each source storage node and the average utilization rate, wherein the data migration instruction includes a preset amount of data to be migrated; The data to be migrated in each of the source storage nodes is determined according to the preset amount of data to be migrated.

3. A message queue rebalancing method according to claim 1 or 2, characterized in that: The step of concurrently migrating the data to be migrated from each of the source storage nodes to the target storage node includes: Adding the data to be migrated in each source storage node to a list to be migrated; Start multiple parallel threads according to the list to be migrated; The data to be migrated in the source storage node is migrated to the target storage node according to the parallel thread.

4. A message queue rebalancing method according to claim 3, characterized in that: Migrating the data to be migrated in the source storage node to the target storage node according to the parallel thread includes: Obtaining the source data segment number of the data to be migrated and the source storage node number where the data to be migrated is located; Obtaining the target data segment number and target storage node number of the data to be migrated; Obtaining the data to be migrated from the source data segment number and the source storage node number according to the parallel thread; The data to be migrated is migrated to the target storage node where the target data segment number and the target storage node number are located.

5. A message queue rebalancing method according to claim 4, characterized in that: The to-be-migrated list includes a to-be-copied list, and the to-be-copied list is used to start multiple parallel threads and migrate the to-be-migrated data corresponding to the source storage node to the target storage node according to the parallel threads; The step of migrating the data to be migrated to the target storage node where the target data segment number and the target storage node number are located is specifically: The data to be migrated is copied to the target storage node where the target data segment number and the target storage node number are located.

6. A message queue rebalancing method according to claim 4, characterized in that: The method further comprises: According to the Pulsar message middleware, the data in the message queue is segmented and stored in each storage node; Get the data of each partition in the source storage node; Determining multiple segments of data to be migrated from the data in each partition, or determining data in any partition as data to be migrated; The data to be migrated is concurrently migrated to different target storage nodes.

7. A message queue rebalancing method according to claim 5, characterized in that: The method further comprises: The data segment number and storage node number of the data to be migrated copied in the target storage node are updated through the Zookeeper server cluster.

8. A message queue rebalancing method according to claim 7, characterized in that: The updating of the data segment number and storage node number of the data to be migrated obtained by copying in the target storage node through the Zookeeper server cluster includes: Modify the data segment number of the data to be migrated in the Zookeeper server cluster to the target data segment number of the current target storage node; The storage node number of the data to be migrated in the Zookeeper server cluster is modified to the target storage node number of the current target storage node.

9. A message queue rebalancing method according to claim 1, characterized in that: The clearing of the data to be migrated from each of the source storage nodes includes: After the data to be migrated is concurrently migrated to the target storage node, generating a data clearing instruction; According to the data clearing instruction, the data to be migrated in each of the source storage nodes is cleared.

10. A message queue rebalancing method according to claim 9, characterized in that: The clearing of the data to be migrated in each of the source storage nodes according to the data clearing instruction includes: According to the data clearing instruction, obtain metadata information of the data to be migrated in the Zookeeper server cluster, wherein the metadata information includes a data segment number and a storage node number; When the storage node number of the data to be migrated is different from the source storage node number, the data to be migrated on the corresponding source storage node is cleared.

11. A message queue rebalancing device, characterized in that: include: An acquisition module is used to obtain the space utilization of each source storage node in the message queue; Calculating an average utilization rate of all storage nodes based on the space utilization rate of each source storage node and the space utilization rate of the target storage node, wherein the storage nodes include the source storage node and the target storage node, and the data segments in the message queue are stored in each source storage node; a determination module, configured to determine, based on the space utilization rate of each source storage node and the average utilization rate, data segments in each source storage node that exceed the average utilization rate as data to be migrated; A concurrent migration module, configured to concurrently migrate the data to be migrated in each of the source storage nodes to a target storage node, wherein the target storage node is a storage node newly added to the message queue or a storage node whose space utilization in the message queue is lower than a storage threshold; The clearing module is used to clear the data to be migrated in each of the source storage nodes.

12. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 10.

Images