Method, system and equipment for distributing data import channels of database and medium

Abstract

The invention provides a data import channel distribution method, system and device of a database and a medium. The method, system and device are applied to a main node of a target database. The method comprises the following steps: monitoring a preset distributed coordination service, and obtaining change events including a node adding event, a node quitting event and a channel number change event; and based on the current node set of the target database and the data import channel set, generating a channel allocation scheme and sending the channel allocation scheme to the distributed coordination service, so that each node of the target database respectively monitors the distributed coordination service and obtains a corresponding channel list according to the channel allocation scheme to realize re-allocation of the data import channel. When the distribution scheme is generated, the real-time node state and the channel state of the distributed database cluster are directly associated, it is ensured that channel distribution is always matched with actual resources of the cluster, resource waste or overload is avoided, and the overall data importing efficiency can be greatly improved.

Description

Technical Field

[0001] This invention application relates to the field of distributed databases, and more particularly to a method, system, device, and medium for allocating data import channels for a database. Background Technology

[0002] Distributed (pure in-memory) database clusters achieve horizontal scaling and high availability by distributing data and computational load across multiple nodes. However, when importing streaming data into such clusters, one key challenge is how to effectively distribute the external data streams across different nodes (e.g., Kafka topic partitions). Currently, existing technologies primarily employ client-based allocation, specifically utilizing the Kafka ConsumerGroup's own rebalancing mechanism. This mechanism may cause a brief "Stop-the-World" pause for all consumers, resulting in significant processing latency in high-concurrency scenarios. Furthermore, this allocation strategy is decoupled from the internal state of the database cluster, impacting overall data import efficiency. Summary of the Invention

[0003] This invention application provides a method, system, device, and medium for allocating data import channels in a database, in order to solve the technical problem of how to improve the overall efficiency of data import.

[0004] To address the aforementioned technical problems, this invention provides a method for allocating data import channels in a database, applied to the master node of a target database; the target database is a distributed database cluster. The import channel allocation method includes: The system monitors a pre-defined distributed coordination service to obtain change events of the target database and the data import channel set; wherein, the change events include node addition events, node exit events, and channel number change events. Upon detecting the change event, a channel allocation scheme is generated based on the current node set and data import channel set of the target database; The channel allocation scheme is sent to the distributed coordination service so that each node of the target database listens to the distributed coordination service and obtains the corresponding channel list according to the channel allocation scheme it hears. The data import channels are reassigned based on the channel list.

[0005] As a preferred embodiment, each node of the target database listens to the distributed coordination service and obtains a corresponding channel list based on the channel allocation scheme it has listened to, including: Each node of the target database listens to the corresponding allocation path in the distributed coordination service to obtain allocation information; Based on the allocation information, the corresponding channel list is obtained.

[0006] As a preferred embodiment, the data import channel allocation method further includes: Each node in the target database compares the channel list with the currently processed data import channel to obtain a channel comparison result; wherein, the channel comparison result includes added channels and removed channels; Send a channel addition command to the local importer to enable the importer to start the new channel, thereby importing data into the target database; Each node of the target database sends a channel removal command to its local importer, causing the importer to stop processing the removed channel and release the computing resources corresponding to the removed channel.

[0007] As a preferred embodiment, the step of generating a channel allocation scheme based on the current node set and data import channel set of the target database includes: Calculate the load data of each node in the node set, and determine the node weight based on the load data; Based on the number of channels in the data import channel set and the number of nodes in the node set, calculate the theoretically allocated baseline value of the number of channels for each node; Obtain the hash value of the channel identifier of each data import channel in the data import channel set; and perform hash bucket pre-allocation based on the hash value and the channel quantity baseline value to obtain the initial allocation scheme; The initial allocation scheme is adjusted according to the node weights to obtain the channel allocation scheme.

[0008] As a preferred embodiment, the channel allocation scheme is represented by an allocation mapping table, which is an allocation relationship table with node identifiers as keys and channel lists as values.

[0009] This application also provides another method for allocating data import channels in a database, applied to a server. This method, prior to the pre-defined distributed coordination service described in any of the above embodiments, further includes: Temporary nodes are created under the pre-built node election directory in the distributed coordination service; The successfully constructed temporary node is designated as the master node.

[0010] This application also provides another method for allocating data import channels in a database, applied to a server. This method, prior to the pre-defined distributed coordination service described in any of the above embodiments, further includes: A directory structure is created in the distributed coordination service; wherein the directory structure includes a node registration directory, a node election directory, a channel registration directory, and a channel allocation directory; When an import task is detected, a data import channel is registered in the channel registration directory to obtain the data import channel set.

[0011] Accordingly, this application also provides a database data import channel allocation system, applied to the master node of a target database; the target database is a distributed database cluster; The import channel allocation system includes a monitoring module, a generation module, a sending module, and an allocation module; wherein, The monitoring module is used to monitor a preset distributed coordination service to obtain change events of the target database and the data import channel set; wherein, the change events include node addition events, node exit events, and channel number change events; The generation module is used to generate a channel allocation scheme based on the current node set and data import channel set of the target database when the change event is detected. The sending module is used to send the channel allocation scheme to the distributed coordination service, so that each node of the target database listens to the distributed coordination service and obtains the corresponding channel list according to the channel allocation scheme it listens to. The allocation module is used to reallocate data import channels according to the channel list.

[0012] As a preferred embodiment, each node of the target database listens to the distributed coordination service and obtains a corresponding channel list based on the channel allocation scheme it has listened to, including: Each node of the target database listens to the corresponding allocation path in the distributed coordination service to obtain allocation information; Based on the allocation information, the corresponding channel list is obtained.

[0013] As a preferred embodiment, the data import channel allocation system further includes a channel change module, which is used for: Each node in the target database is controlled to compare the channel list with the currently processed data import channel to obtain the channel comparison result; wherein, the channel comparison result includes added channels and removed channels; Send a channel addition command to the local importer to enable the importer to start the new channel, thereby importing data into the target database; Each node of the target database is controlled to send a channel removal command to its local importer, so that the importer stops processing the removed channel and releases the computing resources corresponding to the removed channel.

[0014] As a preferred embodiment, the generation module generates a channel allocation scheme based on the current node set and data import channel set of the target database, including: The generation module calculates the load data of each node in the node set and determines the node weight based on the load data; Based on the number of channels in the data import channel set and the number of nodes in the node set, calculate the theoretically allocated baseline value of the number of channels for each node; Obtain the hash value of the channel identifier of each data import channel in the data import channel set; and perform hash bucket pre-allocation based on the hash value and the channel quantity baseline value to obtain the initial allocation scheme; The initial allocation scheme is adjusted according to the node weights to obtain the channel allocation scheme.

[0015] As a preferred embodiment, the channel allocation scheme is represented by an allocation mapping table, which is an allocation relationship table with node identifiers as keys and channel lists as values.

[0016] This application also provides another database data import channel allocation system applied to a server. The data import channel allocation system includes a master node determination module, which is used before the pre-defined distributed coordination service described in any of the above embodiments: Temporary nodes are created under the pre-built node election directory in the distributed coordination service; The successfully constructed temporary node is designated as the master node.

[0017] This invention application also provides another database data import channel allocation system applied to a server. The data import channel allocation system further includes a channel registration module, which is used to monitor the preset distributed coordination service described in any of the above embodiments: A directory structure is created in the distributed coordination service; wherein the directory structure includes a node registration directory, a node election directory, a channel registration directory, and a channel allocation directory; When an import task is detected, a data import channel is registered in the channel registration directory to obtain the data import channel set.

[0018] Accordingly, this application also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the aforementioned database data import channel allocation method.

[0019] Accordingly, this application also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the aforementioned database data import channel allocation method.

[0020] Compared with the prior art, this invention application has the following beneficial effects: This invention provides a method, system, device, and medium for allocating data import channels in a database, applied to the master node of a target database; the target database is a distributed database cluster; the data import channel allocation method includes: monitoring a preset distributed coordination service to obtain change events of the target database and the data import channel set; wherein, the change events include node addition events, node exit events, and channel number change events; when the change event is detected, generating a channel allocation scheme based on the current node set and data import channel set of the target database; sending the channel allocation scheme to the distributed coordination service so that each node of the target database monitors the distributed coordination service and obtains a corresponding channel list according to the monitored channel allocation scheme; and reallocating data import channels according to the channel list. This invention application, based on the current node set and data import channel set of the target database, directly associates the real-time node status (node ​​join event and node leave event) and channel status (channel number change event) of the distributed database cluster when generating the allocation scheme. This ensures that the channel allocation always matches the actual resources of the cluster (e.g., when a node is added, an idle channel is allocated in a timely manner, and when a node leaves, its channel is quickly reclaimed and allocated to a non-master node that has not left), avoiding resource waste or overload. By reallocating and updating the channels through the data import channel allocation method of this scheme, the overall data import efficiency can be greatly improved.

[0021] In addition, all channels obtain change events and channel allocation schemes through a unified listening service. All channel allocation schemes are centrally distributed through this service, ensuring that the channel list obtained by each non-master node is unique and non-overlapping. This avoids conflicts where multiple non-master nodes compete for the same channel, eliminates duplicate data imports caused by duplicate channel allocations, and prevents data loss caused by non-master nodes not obtaining valid channels due to inconsistent allocations. This significantly improves the accuracy and reliability of data import. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating an embodiment of the database data import channel allocation method provided in this invention application.

[0023] Figure 2 This is a flowchart illustrating one embodiment of the database data import channel allocation method provided in this invention application.

[0024] Figure 3 This is a flowchart illustrating another embodiment of the data import channel allocation method for the database provided in this application.

[0025] Figure 4 This is a schematic diagram of an embodiment of the data import channel allocation system for the database provided in this application.

[0026] Figure 5 This is a schematic diagram of another embodiment of the data import channel allocation system for the database provided in this application.

[0027] Figure 6 This is a schematic diagram of another embodiment of the data import channel allocation system for the database provided in this application. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Example 1 Please refer to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the database data import channel allocation method provided in this invention application. Figure 1 The illustrated embodiment can be applied to the master node of the target database.

[0030] In this embodiment, the target database is a distributed (pure in-memory) database cluster. This distributed database cluster comprises multiple nodes, distributing data and computing load across these nodes to achieve horizontal scalability and high availability of the system.

[0031] Figure 1 The illustrated embodiment includes steps S101 to S104; each step is described in detail below: Step S101: Listen to the preset distributed coordination service to obtain change events of the target database and the data import channel set; wherein, the change events include node addition events, node exit events, and channel number change events.

[0032] In this embodiment, the nodes in the distributed database cluster can be divided into master nodes and non-master nodes. The master nodes can be used to listen for the aforementioned change events and determine the new channel set corresponding to each node by executing a preset allocation algorithm, thereby obtaining the channel allocation scheme described in step S102 below.

[0033] In some implementations, the main components involved include a Channel Assigner and a distributed coordination service (such as ZooKeeper). The Channel Assigner is a background service module running on each database node, responsible for coordinating and managing the import channels that the node should handle. The distributed coordination service is a highly available, shared, and hierarchical information storage space used for cluster node registration, master node election, and the publication and subscription of channel allocation information.

[0034] Before step S101, a directory structure can be created in the distributed coordination service; wherein, the directory structure includes a node registration directory, a node election directory, a channel registration directory, and a channel allocation directory; when an import task is detected, a data import channel is registered under the channel registration directory to obtain the data import channel set.

[0035] For example, the server can create a predefined directory structure in the distributed coordination service ZooKeeper, such as / importer / nodes (node ​​registration directory), / importer / channels (channel registration and allocation directory, in this example, the channel registration directory and the channel allocation directory are the same), and / importer / master (directory used for node election, used for electing the master node, i.e., determining the master node and the remaining nodes).

[0036] After the channel allocator on each node starts, the server can create a temporary node ZNode (zookeeper node) in the / nodes directory to represent itself and listen for changes in the child nodes of the / nodes directory.

[0037] Preferably, the server can construct ephemeral nodes under the pre-built node election directory / master in the distributed coordination service; the successfully constructed ephemeral node is determined as the master node. Based on ZooKeeper's session mechanism, it can be ensured that the node registration process is atomic (i.e., success or failure, no partial states) and unique, so only one ephemeral node will be successfully constructed, and this ephemeral node is determined as the master node.

[0038] In some implementations, when an import task is detected, multiple data import channels are registered in the channel registration directory to obtain the data import channel set. For example, when importing a Kafka topic, and the import is considered configured and started, the importer can register all the data import channels it needs to process with the local channel allocator, i.e., all partitions of the Kafka topic, and can register this channel information in the / channels / registered directory of ZooKeeper.

[0039] Understandable. Figure 1 In any of the embodiments shown, the master node is used as the execution subject. Before step S101, the steps of creating the directory structure, determining the master node, and registering the data import channel can be executed by the server; the step of registering the data import channel can also be executed by the channel allocator.

[0040] Step S102: When the change event is detected, a channel allocation scheme is generated based on the current node set and data import channel set of the target database.

[0041] In this embodiment, the change events mainly include node joining events, node leaving events, and channel number change events. When any of the above change events is detected, the master node generates a channel allocation scheme based on the latest node set and data import channel set.

[0042] In some preferred embodiments, such as Figure 2 As shown, step S102, which generates a channel allocation scheme based on the current node set and data import channel set of the target database, includes steps S201 to S204; each step is detailed below: Step S201: Calculate the load data of each node in the node set, and determine the node weight based on the load data; Step S202: Calculate the theoretically allocated baseline value of the number of channels for each node based on the number of channels in the data import channel set and the number of nodes in the node set. Step S203: Obtain the hash value of the channel identifier of each data import channel in the data import channel set; and perform hash bucket pre-allocation based on the hash value and the channel quantity baseline value to obtain the initial allocation scheme; Step S204: Adjust the initial allocation scheme according to the node weights to obtain the channel allocation scheme.

[0043] For example, before step S201, the master node can obtain a list of all currently active nodes through ZooKeeper, and read the heartbeat timestamp, node identifier, and node status flag of each node; in addition, it can obtain a set of data import channels to be allocated, and read the channel identifier, channel type (such as Kafka partition number), channel data volume statistics, and channel priority flag; at the same time, it can collect real-time load data of each node, such as CPU utilization, memory utilization, network I / O rate, and the number of channels currently being processed; in addition, it can read the sharding distribution mapping table of the database cluster to obtain the final storage location information of the target data corresponding to each channel, etc., to realize the collection and preprocessing of cluster status information.

[0044] Furthermore, step S201 can determine the node weight based on the load data. For example, the load data can quantify the node load through CPU utilization and memory utilization.

[0045] For example, set the CPU utilization threshold to 80% and the memory utilization threshold to 85%. The weight of nodes exceeding the threshold can be reduced proportionally, calculated using the formula: weight = 1 - max(0, (CPU utilization - 80%) / 20%, (Memory utilization - 85%) / 15%).

[0046] In some implementations, node weights can be further adjusted based on the time a node has been in the cluster. For example, the weight of a newly added node (running time < 10 minutes) can be reduced by 30% to ensure allocation stability. Nodes can be classified according to their weights as high-weight nodes, medium-weight nodes, and low-weight nodes. Optionally, the node weight can be a normalized weight, or a weight coefficient can be used and further normalized before subsequent steps.

[0047] Further, step S203 performs hash bucket pre-allocation based on the hash value and the baseline value of the number of channels to obtain an initial allocation scheme. This pre-allocation process aims to ensure that adjacent partitions of the same topic are allocated to different nodes as much as possible, avoiding data skew. When the initial allocation scheme is obtained, it can be adjusted according to the node weights to obtain the channel allocation scheme. For example, the number of high-weight nodes allocated can exceed the baseline value by 20%; the number of low-weight nodes allocated is limited to within 70% of the baseline value; and medium-weight nodes are allocated close to the baseline value.

[0048] Furthermore, the channel allocation scheme is represented by an allocation mapping table, which is an allocation relationship table with node identifiers as keys and channel lists as values.

[0049] Furthermore, the initial allocation scheme or channel allocation scheme can be further adjusted based on data affinity. For example, analyze the final target node for channel data storage: determine which database node each channel's data is primarily stored on based on database sharding rules; prioritize allocating channels to the node where the primary replica of its target data resides; when the primary replica node is overloaded, select the node where the backup replica of the target data resides; record each affinity adjustment to avoid over-concentration; for channels that cannot achieve affinity matching, assess their data transmission network overhead and prioritize the node with the lowest network latency, etc.

[0050] Furthermore, the currently running channels can be compared with the channel allocation scheme to identify channels that need to be stopped (channels removed from the node) and channels that need to be newly started (channels added to the node). The total number of channel migrations can then be calculated based on the number of removed and added channels. When the number of migrations exceeds 30% of the total number of channels, the migration can be performed in batches. Channels with small data volumes and low processing frequencies are migrated first. The allocation of channels that are currently processing large amounts of data remains unchanged until the current batch is completed.

[0051] Furthermore, when determining the channel allocation scheme, node failure scenarios should be considered. Channels should be prioritized based on data affinity and urgency, with critical channels being transferred first. Channels from failed nodes should be evenly distributed across multiple surviving nodes to avoid single-point overload. For nodes that have recovered and rejoined, instead of immediately allocating a large number of channels, a small number of lowest-priority channels should be allocated for warm-up operation. After a period of stable operation, such as 10 minutes, the number of allocated channels can be gradually increased. During channel allocation, it should be ensured that each channel is handled by exactly one node, and the difference in the number of allocated channels between nodes should not exceed 50% of the theoretical average. Additionally, any abnormal situations such as node overload or idleness should be checked.

[0052] In some preferred embodiments, to ensure the rationality and scientific nature of the channel allocation scheme, a quality score can also be applied, such as a load balancing score (the smaller the variance of the number of channels allocated to each node, the higher the score), a data affinity score (the higher the affinity matching ratio, the higher the score), and a migration cost score (the fewer channels that need to be migrated, the higher the score).

[0053] When the score is lower than the preset score threshold (e.g., 70 points), it can be optimized and adjusted again; the final allocation scheme is serialized into a standard format, written to the corresponding path of the distributed coordination service ZooKeeper, and a version number and timestamp are set for subsequent steps to call.

[0054] Step S103: Send the channel allocation scheme to the distributed coordination service so that each node of the target database listens to the distributed coordination service and obtains the corresponding channel list according to the channel allocation scheme it has listened to.

[0055] In this step, the master node sends the channel allocation scheme to the distributed coordination service. Each node in the target database (including the master node and non-master nodes) listens to the corresponding allocation path ( / channels / assigned / {my_node_id}) in the distributed coordination service to obtain the allocation information. Based on the allocation information, the corresponding channel list is obtained.

[0056] Step S104: Reassign the data import channels according to the channel list.

[0057] In this embodiment, each node can compare the channel list with the channel (list) it is currently processing to obtain the added and removed channels. At this time, the data import channels are redistributed based on the latest channel list.

[0058] In some preferred embodiments, such as Figure 3 As shown, the data import channel allocation method further includes steps S301 to S303; each step is described in detail below: Step S301: Each node in the target database compares the channel list with the currently processed data import channel to obtain a channel comparison result; wherein, the channel comparison result includes added channels and removed channels; In step S302, each node of the target database sends a channel addition instruction to its local importer, so that the importer starts the new channel, thereby realizing the import of data into the target database. In step S303, each node of the target database sends a channel removal command to its local importer, so that the importer stops processing the removed channel and releases the computing resources corresponding to the removed channel.

[0059] As an example, in some application instances, suppose a cluster consisting of 3 DB nodes (Node1, Node2, Node3) needs to import a Kafka topic named iot_data, which has 6 partitions (P0, P1, P2, P3, P4, P5).

[0060] The Kafka topics (import tasks) mentioned above typically specify detailed information about the Kafka topic, such as the topic name and the number of topic partitions, through configuration files (such as JSON or YAML format) or API interfaces. The system parses this configuration information upon startup.

[0061] The importer interacts with the Kafka cluster, for example, by querying the topic's partition information through Kafka's metadata API to obtain the exact list of channels (i.e., partitions). For instance, if the Kafka topic is named "iot_data", the importer will query Kafka to get the number of partitions for that topic (e.g., 6 partitions: P0 to P5) and generate a set of channels.

[0062] After the three nodes are started, their channel allocator modules create three ephemeral nodes, Node1, Node2, and Node3, respectively, under / importer / nodes / in ZooKeeper. Based on ZooKeeper's session mechanism, Node1 (or its channel allocator) becomes the master (allocator) node.

[0063] The administrator configured an import task for the iot_data topic in the cluster. The importer on each node discovered this task and registered the six partitions P0 to P5 with its respective channel allocator.

[0064] Ultimately, the master node Node1 learns from ZooKeeper that the channels to be allocated are {P0, P1, P2, P3, P4, P5}.

[0065] During the initial allocation, the master node Node1 detected 3 live nodes and 6 channels to be allocated.

[0066] Execute the preset uniform allocation algorithm to obtain the channel allocation scheme: Node1: {P0, P1}; Node2: {P2, P3}; Node3: {P4, P5}; Node1 writes this scheme to ZooKeeper: write "P0,P1" in / channels / assigned / node1, "P2,P3" in / channels / assigned / node2, and "P4,P5" in / channels / assigned / node3.

[0067] Node1, Node2, and Node3 (or their channel allocators) each listen for updates to their assigned path data. At this point, Node1 notifies the local importer to start consuming data from P0 and P1. Node2 notifies the local importer to start consuming data from P2 and P3. Node3 notifies the local importer to start consuming data from P4 and P5. Thus, the data import process is now running with a load balancer across the cluster.

[0068] In another application example, a new database node, Node4, joined the cluster and started successfully.

[0069] The Node4 channel allocator can be created in the / nodes directory of ZooKeeper.

[0070] The master node Node1 detects a node addition event in / nodes, which triggers a change event.

[0071] Node1 obtains a new list of nodes {Node1, Node2, Node3, Node4} and a list of channels {P0...P5}.

[0072] The master node recalculates the allocation scheme, with the allocation goal of minimizing channel migration. For example, Node1: {P0}; Node2: {P2}; Node3: {P4}; Node4: {P1, P3, P5}; As can be seen, to achieve system balancing, Node4 takes one partition from each older node. Node1 writes the new channel allocation scheme into ZooKeeper. Nodes 1, 2, and 3 detect the reduction in their respective channel lists and stop processing P1, P3, and P5. Node4 detects the new channel list assigned to it and starts processing P1, P3, and P5. The entire scaling process requires no manual intervention, achieving automatic load balancing.

[0073] In order to balance the system, Node4 takes away one partition from each old node. This allows new nodes to quickly acquire workloads and distributes the load sources as widely as possible to avoid excessive instantaneous load drops on a single old node.

[0074] The new node Node4 needs to take on the load from scratch. To achieve an initial balanced state, the master node calculates the minimum number of channels to migrate, rounded down (total number of channels / total number of new nodes). In this application example, it's floor(6 / 4) = 1. To achieve a better balance as quickly as possible, more channels may be migrated. In this application example, 3 channels were migrated, likely to bring the final distribution closer to the intermediate state of {1, 1, 1, 3}, before fine-tuning.

[0075] All existing nodes (Node1, Node2, Node3) currently holding channels are selected as contributing nodes. Then, channels are selected in a round-robin fashion: the master node traverses the list of contributing nodes in a predetermined order (e.g., sorted by node ID), and selects a channel from the list of channels currently managed by each node for migration.

[0076] Choose P1 from {P0, P1} of Node1.

[0077] Choose P3 from {P2, P3} of Node2.

[0078] Choose P5 from {P4, P5} in Node3.

[0079] Thus, we obtain Node4: {P1, P3, P5}.

[0080] In another application example, suppose Node2 fails and goes down due to a hardware problem.

[0081] The temporary node / nodes / node2 created by Node2 in ZooKeeper was automatically deleted due to session timeout.

[0082] The master node Node1 detects a node exit event in / nodes, which triggers a change event.

[0083] Node1 obtains the list of surviving nodes {Node1, Node3} and the channels {P2, P3} that need to be reallocated (originally belonging to Node2).

[0084] Recalculate the allocation scheme and assign the channels that need to be reallocated to the surviving nodes: Node1: {P0, P1, P2} Node3: {P4, P5, P3} Node1 writes the new scheme into ZooKeeper.

[0085] Node1 and Node3 detected the allocation change and took over the processing of P2 and P3 respectively. Data processing automatically resumed after a brief interruption, achieving high availability.

[0086] Accordingly, such as Figure 4 As shown, this invention application also provides a database data import channel allocation system 400, applied to the master node of a target database; the target database is a distributed database cluster; The import channel allocation system includes a monitoring module 401, a generation module 402, a sending module 403, and an allocation module 404; wherein, The monitoring module 401 is used to monitor a preset distributed coordination service to obtain change events of the target database and the data import channel set; wherein, the change events include node addition events, node exit events, and channel number change events; The generation module 402 is used to generate a channel allocation scheme based on the current node set and data import channel set of the target database when the change event is detected. The sending module 403 is used to send the channel allocation scheme to the distributed coordination service, so that each node of the target database listens to the distributed coordination service and obtains the corresponding channel list according to the channel allocation scheme it listens to. The allocation module 404 is used to reallocate the data import channels according to the channel list.

[0087] As a preferred embodiment, each node of the target database listens to the distributed coordination service and obtains a corresponding channel list based on the channel allocation scheme it has listened to, including: Each node of the target database listens to the corresponding allocation path in the distributed coordination service to obtain allocation information; Based on the allocation information, the corresponding channel list is obtained.

[0088] As a preferred embodiment, the data import channel allocation system 400 further includes a channel change module, which is used for: Each node in the target database is controlled to compare the channel list with the currently processed data import channel to obtain the channel comparison result; wherein, the channel comparison result includes added channels and removed channels; Send a channel addition command to the local importer to enable the importer to start the new channel, thereby importing data into the target database; Each node of the target database is controlled to send a channel removal command to its local importer, so that the importer stops processing the removed channel and releases the computing resources corresponding to the removed channel.

[0089] As a preferred embodiment, the generation module 402 generates a channel allocation scheme based on the current node set and data import channel set of the target database, including: The generation module 402 calculates the load data of each node in the node set and determines the node weight based on the load data; Based on the number of channels in the data import channel set and the number of nodes in the node set, calculate the theoretically allocated baseline value of the number of channels for each node; Obtain the hash value of the channel identifier of each data import channel in the data import channel set; and perform hash bucket pre-allocation based on the hash value and the channel quantity baseline value to obtain the initial allocation scheme; The initial allocation scheme is adjusted according to the node weights to obtain the channel allocation scheme.

[0090] As a preferred embodiment, the channel allocation scheme is represented by an allocation mapping table, which is an allocation relationship table with node identifiers as keys and channel lists as values.

[0091] like Figure 5 As shown, this application also provides another database data import channel allocation system 500, applied to a server. The data import channel allocation system includes a master node determination module 501, which is used before the pre-defined distributed coordination service described in any of the above embodiments: Temporary nodes are created under the pre-built node election directory in the distributed coordination service; The successfully constructed temporary node is designated as the master node.

[0092] like Figure 6 As shown, this application also provides another database data import channel allocation system 600, applied to a server. The data import channel allocation system further includes a channel registration module 601, which is used before the pre-defined distributed coordination service described in any of the above embodiments: A directory structure is created in the distributed coordination service; wherein the directory structure includes a node registration directory, a node election directory, a channel registration directory, and a channel allocation directory; When an import task is detected, a data import channel is registered in the channel registration directory to obtain the data import channel set.

[0093] Accordingly, this application also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the aforementioned database data import channel allocation method.

[0094] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal, connecting various parts of the terminal via various interfaces and lines.

[0095] The memory can be used to store the computer program. The processor implements various functions of the terminal by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0096] Accordingly, this application also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the aforementioned database data import channel allocation method.

[0097] If the data import channel allocation system module of the database is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0098] Compared with the prior art, this invention application has the following beneficial effects: This invention provides a method, system, device, and medium for allocating data import channels in a database, applied to the master node of a target database; the target database is a distributed database cluster; the data import channel allocation method includes: monitoring a preset distributed coordination service to obtain change events of the target database and the data import channel set; wherein, the change events include node addition events, node exit events, and channel number change events; when the change event is detected, generating a channel allocation scheme based on the current node set and data import channel set of the target database; sending the channel allocation scheme to the distributed coordination service so that each node of the target database monitors the distributed coordination service and obtains a corresponding channel list according to the monitored channel allocation scheme; and reallocating data import channels according to the channel list. This invention application, based on the current node set and data import channel set of the target database, directly associates the real-time node status (node ​​join event and node leave event) and channel status (channel number change event) of the distributed database cluster when generating the allocation scheme. This ensures that the channel allocation always matches the actual resources of the cluster (e.g., when a node is added, an idle channel is allocated in a timely manner, and when a node leaves, its channel is quickly reclaimed and allocated to a non-master node that has not left), avoiding resource waste or overload. By reallocating and updating the channels through the data import channel allocation method of this scheme, the overall data import efficiency can be greatly improved.

[0099] In addition, all channels obtain change events and channel allocation schemes through a unified listening service. All channel allocation schemes are centrally distributed through this service, ensuring that the channel list obtained by each non-master node is unique and non-overlapping. This avoids conflicts where multiple non-master nodes compete for the same channel, eliminates duplicate data imports caused by duplicate channel allocations, and prevents data loss caused by non-master nodes not obtaining valid channels due to inconsistent allocations. This significantly improves the accuracy and reliability of data import.

[0100] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for allocating data import channels in a database, characterized in that, Applied to the master node of the target database; The target database is a distributed database cluster; The import channel allocation method includes: The system monitors a pre-defined distributed coordination service to obtain change events of the target database and the data import channel set; wherein, the change events include node addition events, node exit events, and channel number change events. Upon detecting the change event, a channel allocation scheme is generated based on the current node set and data import channel set of the target database; The channel allocation scheme is sent to the distributed coordination service so that each node of the target database listens to the distributed coordination service and obtains the corresponding channel list according to the channel allocation scheme it hears. The data import channels are reassigned based on the channel list.

2. The database data import channel allocation method as described in claim 1, characterized in that, Each node of the target database listens to the distributed coordination service and obtains a corresponding channel list based on the channel allocation scheme it has listened to, including: Each node of the target database listens to the corresponding allocation path in the distributed coordination service to obtain allocation information; Based on the allocation information, the corresponding channel list is obtained.

3. The database data import channel allocation method as described in claim 2, characterized in that, The data import channel allocation method also includes: Each node in the target database compares the channel list with the currently processed data import channel to obtain a channel comparison result; wherein, the channel comparison result includes added channels and removed channels; Send a channel addition command to the local importer to enable the importer to start the new channel, thereby importing data into the target database; Each node of the target database sends a channel removal command to its local importer, causing the importer to stop processing the removed channel and release the computing resources corresponding to the removed channel.

4. The database data import channel allocation method as described in claim 1, characterized in that, The process of generating a channel allocation scheme based on the current node set and data import channel set of the target database includes: Calculate the load data of each node in the node set, and determine the node weight based on the load data; Based on the number of channels in the data import channel set and the number of nodes in the node set, calculate the theoretically allocated baseline value of the number of channels for each node; Obtain the hash value of the channel identifier of each data import channel in the data import channel set; and perform hash bucket pre-allocation based on the hash value and the channel quantity baseline value to obtain the initial allocation scheme; The initial allocation scheme is adjusted according to the node weights to obtain the channel allocation scheme.

5. The database data import channel allocation method as described in claim 4, characterized in that, The channel allocation scheme is represented by an allocation mapping table, which is an allocation relationship table with node identifiers as keys and channel lists as values.

6. A method for allocating data import channels in a database, characterized in that, Applied to a server, the data import channel allocation method, prior to the pre-defined distributed coordination service described in any one of claims 1 to 5, further includes: Temporary nodes are created under the pre-built node election directory in the distributed coordination service; The successfully constructed temporary node is designated as the master node.

7. A method for allocating data import channels in a database, characterized in that, Applied to a server, the data import channel allocation method, prior to the pre-defined distributed coordination service described in any one of claims 1 to 5, further includes: A directory structure is created in the distributed coordination service; wherein the directory structure includes a node registration directory, a node election directory, a channel registration directory, and a channel allocation directory; When an import task is detected, a data import channel is registered in the channel registration directory to obtain the data import channel set.

8. A database data import channel allocation system, characterized in that, Applied to the master node of the target database; the target database is a distributed database cluster; The import channel allocation system includes a monitoring module, a generation module, a sending module, and an allocation module; wherein, The monitoring module is used to monitor a preset distributed coordination service to obtain change events of the target database and the data import channel set; wherein, the change events include node addition events, node exit events, and channel number change events; The generation module is used to generate a channel allocation scheme based on the current node set and data import channel set of the target database when the change event is detected. The sending module is used to send the channel allocation scheme to the distributed coordination service, so that each node of the target database listens to the distributed coordination service and obtains the corresponding channel list according to the channel allocation scheme it listens to. The allocation module is used to reallocate the data import channels according to the channel list.

9. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements a database data import channel allocation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform a database data import channel allocation method as described in any one of claims 1 to 7.

Images