Cluster data merging method, merging service cluster, equipment and storage medium

By building a merging service cluster and utilizing the collaboration of task modules, load balancing modules and merging node modules, real-time data merging and synchronization of multi-source data clusters is achieved, solving the problem of the existing technology that it is impossible to merge and synchronize multi-cluster MongoDB data tables in real time, and improving the efficiency and accuracy of data merging.

CN120763246APending Publication Date: 2025-10-10DUXIAOMAN TECH (BEIJING) CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202510869823.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-26
Publication Date
2025-10-10

AI Technical Summary

Technical Problem

Existing database and table synchronization tools cannot achieve real-time merging and synchronization of multi-cluster MongoDB data tables, and cannot meet the real-time data merging needs in the data integration field.

Method used

Build a merge service cluster, including task module, load balancing module and merge node module. Monitor node status information through the load balancing module, the task module determines the target merge node and issues the merge task, and the merge node module performs data table merge and synchronization operations to achieve real-time merge and synchronization of multi-source data clusters.

Benefits of technology

It realizes the real-time data merging and synchronization of multi-source data clusters, meets the real-time data merging needs in the field of data integration, and improves the efficiency and accuracy of data merging.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120763246A_ABST
    Figure CN120763246A_ABST
Patent Text Reader

Abstract

The invention provides a cluster data merging method, a merging service cluster, equipment and a storage medium, and relates to the technical field of cluster data processing. The method comprises the following steps: acquiring node state information of each combined node module through a load balancing module, and feeding back the node state information to a task module; under the condition that a target merging task is received, determining a target merging node module through the task module based on the node state information and task demand information of the target merging task; issuing a target merging task to a target merging node module through a task module; determining at least two source data clusters and a single target data cluster indicated by the target merging task through a target merging node module; and merging the obtained data tables in the at least two source data clusters through a target merging node module, and synchronizing the merged data tables to a target data cluster. According to the method, the data merging and synchronizing function of the multi-source data cluster can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of cluster data processing, and in particular to a cluster data merging method, a merging service cluster, a device, and a storage medium. Background Art

[0002] MongoDB is an open-source, document-centric NoSQL database that uses the flexible JSON format to store data. In data aggregation or consolidation, there's a need to merge and synchronize data tables from multiple MongoDB clusters into a single data cluster. However, existing database-table synchronization tools only periodically synchronize data from multiple clusters to a new data cluster and cannot achieve real-time merging and synchronization of data tables. Summary of the Invention

[0003] This application provides a cluster data merging method, a merging service cluster, a device, and a storage medium, which can merge data tables in multiple data clusters into one data cluster, ensuring the real-time performance of data integration. The technical solution is as follows:

[0004] According to one aspect of the present application, a cluster data merging method is provided. The method is applied to a merged service cluster, wherein the merged service cluster includes a task module, a load balancing module, and at least one merge node module. The method includes:

[0005] Obtaining node status information of each of the merging node modules through the load balancing module, and feeding back the node status information to the task module;

[0006] In the case of receiving a target merge task, determining, by the task module, a target merge node module for executing the target merge task based on the node status information and the task requirement information of the target merge task;

[0007] Sending the target merging task to the target merging node module through the task module;

[0008] Determining, by the target merging node module based on the received target merging task, at least two source data clusters and a single target data cluster indicated by the target merging task;

[0009] The acquired data tables in the at least two source data clusters are merged by the target merging node module, and the merged data tables are synchronized to the target data cluster.

[0010] According to another aspect of the present application, a merged service cluster is provided, wherein the merged service cluster includes a task module, a load balancing module, and at least one merge node module:

[0011] The load balancing module is used to obtain node status information of each of the merging node modules and feed back the node status information to the task module;

[0012] The task module is configured to, upon receiving a target merge task, determine a target merge node module for executing the target merge task based on the node status information and the task requirement information of the target merge task;

[0013] The task module is further configured to send the target merging task to the target merging node module;

[0014] a target merging node module, configured to determine, based on the received target merging task, at least two source data clusters and a single target data cluster indicated by the target merging task;

[0015] The target merging node module is further configured to merge the acquired data tables in the at least two source data clusters, and synchronize the merged data tables to the target data cluster.

[0016] According to one aspect of the present application, an electronic device is provided, including: a processor and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the cluster data merging method described above.

[0017] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the cluster data merging method as described above.

[0018] According to another aspect of the present application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-mentioned cluster data merging method.

[0019] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0020] By building a merging service cluster consisting of a task module, a load balancing module, and a merging node module, the load balancing module monitors the node status information of each merging node module and feeds it back to the task module. Upon receiving a target merging task, the task module can determine the target merging node module to execute the target merging task based on the node status information and task requirements, and issue the target merging task to the target node module. The target merging node module then performs data merging and synchronization based on the target merging task. This collaboration between the task module, the load balancing module, and the merging node module enables data merging and synchronization in multi-source data clusters, meeting data merging needs in data integration or aggregation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A flow chart of a cluster data merging method according to an exemplary embodiment of the present application is shown;

[0023] Figure 2 A flow chart showing another cluster data merging method according to an exemplary embodiment of the present application is shown;

[0024] Figure 3 A flow chart showing another cluster data merging method according to an exemplary embodiment of the present application is shown;

[0025] Figure 4 This is a schematic diagram of the structure of a merged service cluster provided in an embodiment of the present application;

[0026] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present application is shown. DETAILED DESCRIPTION

[0027] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of protection of the present application.

[0028] It should be understood that the various steps described in the method embodiments of the present application can be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0029] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; and the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifiers of "one" and "a plurality of" mentioned in this application are illustrative and not restrictive. Those skilled in the art should understand that unless the context clearly indicates otherwise, they should be understood as "one or more". The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not used to limit the scope of these messages or information.

[0030] The following describes the solution of the present application with reference to the accompanying drawings, and illustrates the technical solution provided by the embodiments of the present application in detail through specific embodiments and their application scenarios.

[0031] MongoDB is an open-source, document-centric NoSQL database that uses the flexible JSON format to store data. In data aggregation or consolidation, there's a need to merge and synchronize data tables from multiple MongoDB clusters into a single data cluster. However, existing database-table synchronization tools only periodically synchronize data from multiple clusters to a new data cluster and cannot achieve real-time merging and synchronization of data tables.

[0032] In order to solve the problems existing in the prior art, the embodiment of the present application provides a merging service cluster, which provides real-time merging and synchronization services for the MongoDB cluster. Figure 1 , which shows a flow chart of a cluster data merging method according to an exemplary embodiment of the present application. This method is used to merge service clusters as an example for illustration. Figure 1 As shown, the method includes:

[0033] Step 101: Obtain node status information of each merging node module through the load balancing module, and feed back the node status information to the task module.

[0034] Step 102: upon receiving the target merging task, the task module determines a target merging node module for executing the target merging task based on the node status information and the task requirement information of the target merging task.

[0035] Step 103: Send the target merging task to the target merging node module through the task module.

[0036] Step 104 : Determine, by the target merging node module based on the received target merging task, at least two source data clusters and a single target data cluster indicated by the target merging task.

[0037] Step 105 : merging the acquired data tables in at least two source data clusters through a target merging node module, and synchronizing the merged data tables to the target data cluster.

[0038] The merge service cluster provided in the embodiment of the present application includes a task module, a load balancing module, and at least one merge node module. The task module is used to manage merge tasks, issue merge tasks to the merge node module, stop merge tasks, associate merge tasks, and so on; the load balancing module is used to monitor the node status information of each merge node module, such as load information, the task running status of the merge task, and so on, and feed the node status information back to the task module; the merge node module is used to execute the received merge tasks, read the data table from the source data cluster for merging, and write the merged data table to the new target data cluster.

[0039] Exemplarily, if the merged service cluster is a server, the task module, the load balancing module and at least one merge node module are different service processes running on the server; if the merged service cluster is a server cluster composed of multiple servers, the task module, the load balancing module and at least one merge node module are different servers in the server cluster, and different servers communicate with each other to realize the function of real-time merging and synchronizing data tables.

[0040] Optionally, the merging service cluster provides a front-end interface, and the user can configure relevant task information of the target merging task in the front-end interface, and call the merging service cluster to execute the target merging task.

[0041] During the operation of the merge service cluster, the load balancing module will communicate with each merge node module to obtain the node status information of each merge node module. The node status information of each merge node module may include: node ID, the number of currently running tasks, the maximum number of supported tasks, the current idle resources, the current load factor, the upper limit of load, the lower limit of load, whether automatic expansion is allowed, etc., and the node status information will be fed back to the task module. After the task module receives the target merge task, it will traverse the node status information of each current merge node module, and determine the target merge node module that executes the target merge task based on the node status information and the task requirement information of the target merge task. After determining the target merge node module that executes the target merge task, the task module will send the target merge task to the target merge node module. After receiving the target merge task, the corresponding target merge node module initializes the task information of the target merge task, for example, the target merge task indicates at least two source data clusters and a single target data cluster, where the at least two source data clusters are the sources of the data tables to be merged and synchronized, and the target data cluster is the storage cluster of the merged data tables; and obtains the data tables from the at least two source data clusters and merges the data tables, and synchronizes the merged data tables to the target data cluster to realize the data merging and synchronization function of multiple source data clusters.

[0042] Optionally, after obtaining the node status information of each merge node module, the load balancing module further filters out available merge node modules based on the node status information and feeds back the node status information of only available merge node modules to the task module. Available merge node modules are merge node modules whose current load factor is lower than a load threshold, whose current idle resources are higher than a resource threshold, or whose current number of running tasks is lower than the maximum number of supported tasks.

[0043] In the task module, the target merging tasks to be assigned constitute a task queue. The task module assigns a target merging node module to process each target merging task in the order of task priority based on the task priority of each target merging task in the task queue. The task priority order can be configured in advance by the user. For example, priority identifiers with multiple task priorities are provided in advance. The user can configure corresponding priority identifiers for the target merging tasks as needed. After receiving the target merging tasks, the task module will sort them according to the priority identifiers of each target merging task. Target merging tasks with the same priority identifier can be sorted in order of task reception time to generate a task queue.

[0044] Exemplarily, the currently unassigned target merging tasks are Task 1, Task 2, Task 3, Task 4, Task 5 and Task 6. The task priority of Task 1 is identified as level 3, the task priority of Task 2 is identified as level 4, the task priority of Task 3 is identified as level 1, the task priority of Task 4 is identified as level 2, the task priority of Task 5 is identified as level 5, and the task priority of Task 6 is identified as level 3. Task 1 and Task 6 have the same task priority, but the task reception time of Task 1 is earlier than that of Task 6. The order of the task queue is: Task 5>Task 2>Task 1>Task 6>Task 4>Task 3, that is, the target merging node module is assigned to Task 5 first. In summary, the embodiment of the present application provides a cluster data merging method: by constructing a merging service cluster including a task module, a load balancing module and a merging node module, the load balancing module monitors the node status information of each merging node module and feeds it back to the task module. After receiving the target merging task, the task module can determine the target merging node module that executes the target merging task based on the node status information and task requirement information and send the target merging task to it, and then the target merging node module performs the data merging synchronization operation based on the target merging task. Through the collaboration among the task module, load balancing module and merging node module, the data merging and synchronization function of multi-source data clusters can be realized to meet the data merging needs in the field of data integration or aggregation.

[0045] When performing a table retrieval and merge synchronization operation, there may be two data synchronization scenarios: full data synchronization and update data synchronization. The target merge node module needs to accurately distinguish between the two data synchronization scenarios to avoid repeated data merge and synchronization operations.

[0046] Please refer to Figure 2 , which shows a flow chart of another cluster data merging method according to an exemplary embodiment of the present application. This method is used to merge service clusters as an example for exemplary description. Figure 2 As shown, the method includes:

[0047] Step 201: Obtain node status information of each merging node module through the load balancing module, and feed back the node status information to the task module.

[0048] The node state information of each merged node module includes: a node ID (NodeID), a current running task number (CurrentTaskCount), a node maximum supportable task number (MaxTaskCapacity), a current idle resource (ResourceAvailable), a current load factor (LoadFactor), a load upper limit (MaxLoadThreshold), a load lower limit (MinLoadThreshold), and whether to allow automatic scaling (AutoScale). The node ID is information for uniquely identifying the merged node module; the current running task number is the number of merged tasks that the merged node module is currently executing; the node maximum supportable task number is the number of merged tasks that the merged node module can currently execute simultaneously, which is determined by the total resource quantity currently possessed by the merged node module; the current load factor is determined by the CPU usage and the memory usage of the merged node module, and the higher the CPU usage and / or the memory usage, the greater the current load factor; exceeding the load upper limit indicates that the merged node module is currently in high load, and being below the load lower limit indicates that the merged node module is currently in low load; and whether to allow automatic scaling indicates whether resources can be added to the merged node module when the load of the merged node module is high. It should be noted that the resources can be CPU, memory capacity, and the like.

[0049] In step 202, in the case of receiving a target merged task, the current load factor, the load upper limit, and the current idle resource of each merged node module are obtained from the node state information by the task module.

[0050] After the task module receives the node state information of each merged node module fed back by the load balancing module, the node state information is saved as a node list (NodeList). After the task module receives the target merged task, the node list can be traversed to determine a target merged node module for executing the target merged task.

[0051] Specifically, the merging task issuing strategy of the task module is: (1) traversing the NodeList, screening out the CandidateNodes (candidate merging node modules) whose LoadFactor (current load factor) is less than MaxLoadThreshold (upper limit of load) and ResourceAvailable (current available resource) can meet the demand of the merging task; (2) selecting the merging node module with the lowest LoadFactor (load factor) in the CandidateNodes to issue the task; (3) if the LoadFactor of all NodeList exceeds MaxLoadThreshold, triggering the creation of a new merging node module (NewNode); waiting for the NewNode to start and complete, joining the NodeList, and then reattempting task allocation.

[0052] Based on the above merging task issuing strategy, after receiving the target merging task, the task module first traverses the node state information in the NodeList to obtain the current load factor, upper limit of load and current idle resource of each merging node module, so as to screen the first candidate merging node module according to the information.

[0053] Step 203, screening the first candidate merging node module from each merging node module by the task module, whose current load factor is less than the upper limit of load and whose current idle resource meets the task demand information of the target merging task.

[0054] The first candidate merging node module is the merging node module whose current load factor is less than the upper limit of load and whose current idle resource meets the task demand of the target merging task. In a possible implementation, after obtaining the current load factor, upper limit of load and idle resource of each merging node module, the task module first compares the current load factor and the upper limit of load to screen the first merging node module whose current load factor is less than the upper limit of load, then compares the current idle resource of the first merging node module with the task demand information of the target merging task, and determines the first merging node module whose current idle resource meets the task demand information as the first candidate merging node module.

[0055] Among them, the task requirement information can be the CPU resources and memory resources required to execute the target merge task, and the current idle resources are the current idle CPU resources and current idle memory resources of the merge node module. When judging whether the current idle resources of the first merge node module meet the task requirements, it can be compared whether the current idle CPU resources of the first merge node module are greater than or equal to the CPU resources required for the target merge task, and whether the current idle memory resources of the first merge node module are greater than or equal to the memory resources required for the target merge task. If the current idle CPU resources of the first merge node module are greater than or equal to the CPU resources required for the target merge task, and the current memory resources of the first merge node module are greater than or equal to the memory resources required for the target merge task, then it is determined that the current idle resources of the first merge node module meet the task requirement information of the target merge task, and the first merge node module is determined as the first candidate merge node module.

[0056] For example, taking the case where the merged node modules include node 1, node 2, node 3, node 4, and node 5, the node status information of each merged node module may be as shown in Table 1.

[0057] Table 1

[0058]

[0059] As shown in Table 1, by comparing the current load factor and the load upper limit of each merge node module, the first merge node modules (node ​​1, node 2, node 3 and node 4) whose current load factor is less than the load upper limit are screened out; if the task requirement information of the target merge task indicates that the target merge task requires 45% of CPU resources and 30% of memory resources, then according to the current idle resources of each first merge node module, the first candidate merge node modules screened out are (node ​​1, node 2 and node 4).

[0060] Step 204: If there is a first candidate merge node module, the task module determines the first candidate merge node module with the smallest current load factor as the target merge node module.

[0061] After screening out the first candidate merge node modules, in order to allocate a better merge node module to the current target merge task, the current load factors of the various first candidate merge node modules are compared, and the merge node modules with lower loads are preferentially allocated to the target merge task for task execution. That is, the task module will determine the first candidate merge node module with the smallest current load factor as the target merge node module.

[0062] For example, taking the node status information in Table 1 as an example, after screening out the first candidate merge node (node ​​1, node 2 and node 4), the current load factors of node 1, node 2 and node 4 can also be compared. The current load factor of node 2 is the smallest, and node 2 is correspondingly determined as the target merge node module for executing the target merge task.

[0063] Step 205: If the first candidate merge node module does not exist, a new merge node module is created through the task module, and the new merge node module is determined as a target merge node module.

[0064] When determining the target merge node module for executing the target merge task, there may be two scenarios: the first is that there is a merge node module that can execute the target merge task among the merge node modules already created in the merge service cluster (corresponding to the scenario in which the first candidate merge node module exists in the above embodiment); the second is that there is no merge node module that can execute the target merge task among the merge node modules already created in the merge service cluster. For the second scenario, i.e., the scenario in which the first candidate merge node module does not exist, in order to ensure the smooth execution of the target merge task, the task module can create a new merge node module so that the new merge node module can be used as the target merge node module for executing the target merge task.

[0065] Considering the hardware cost of the merged service cluster, the configuration does not allow for unlimited addition of new merged node modules. In one possible implementation, a first quantity threshold is configured. If the first candidate merged node module does not exist, the number of nodes of the existing merged node modules in the merged service cluster is obtained, and the relationship between the number of nodes and the first quantity threshold is determined. If the number of nodes of the existing merged node modules is less than the first quantity threshold, it indicates that the current merged service cluster allows the creation of new merged node modules, and the new merged node modules are created through the task module.

[0066] Exemplarily, the first quantity threshold may be configured by business personnel according to cluster requirements, and the first quantity threshold may be 50.

[0067] Optionally, if the first candidate merge node module does not exist, and the number of existing merge node modules is greater than or equal to the first threshold, this indicates that the number of existing merge node modules in the current merge service cluster has reached the upper limit, and no new merge node modules can be added. To ensure that the target merge node module can still be assigned to the target merge task, the task module also obtains the expansion permissions of each merge node module from the node status information. If a second candidate merge node module with expansion permissions exists, the task module determines the second candidate merge node module with the lowest current load factor as the target merge node module.

[0068] For example, the merge node modules include nodes 6, 7, 8, and 9. The current load factors of the four merge node modules are all greater than the load limit and / or the current idle resources cannot meet the task requirement information of the target merge task, and the new merge node module cannot be created in the current merge service cluster. Then, it is necessary to determine the target merge node module from the existing nodes 6, 7, 8, and 9. Further determine whether nodes 6, 7, 8, and 9 have the capacity expansion authority. The capacity expansion authority indicates that CPU resources and memory resources can be added to the merge node module. If nodes 6 and 9 do not have the capacity expansion authority, nodes 7 and 8 have the capacity expansion authority, and the current load factor of node 7 is less than the current load factor of node 8, then node 7 is determined as the target merge node module.

[0069] Step 206: Send the target merging task to the target merging node module through the task module.

[0070] After determining the target merging node module, the task module will send the target merging task to the target merging node module so that the target merging node module can execute the target merging task.

[0071] Step 207 : Determine, by the target merging node module based on the received target merging task, at least two source data clusters and a single target data cluster indicated by the target merging task.

[0072] When the target merge node module receives the target merge task, it will initialize and obtain task-related information of the target merge task, such as at least two source data clusters and a single target data cluster indicated by the target merge task, that is, clearly merging both sides of the synchronized data.

[0073] Step 208: Load merge mark information from at least two source data clusters based on the cluster identifier of the target data cluster through the target merge node module.

[0074] When performing a data table acquisition merge synchronization operation, there may be two data synchronization scenarios: one is full data synchronization, which means that the data merge synchronization operation from the source data cluster to the target data cluster has not been performed before, and the full data needs to be synchronized and merged; the other is update data synchronization, which means that the data merge synchronization operation from the source data cluster to the target data cluster has been performed before, and only the updated data in the source data cluster after the last operation needs to be synchronized.

[0075] When the target merge node module executes the target merge task, in order to avoid repeated data merge synchronization operations, the target merge node module needs to accurately distinguish between the above two data synchronization scenarios. In one possible implementation, merge mark information is provided, and the merge mark information is used to mark the operation node position of the last data merge synchronization operation from the source data cluster to the target data cluster, and the merge mark information is associated with the cluster identifier of the target data cluster and stored in the source data cluster. After obtaining at least two source data clusters and a single target data cluster indicated by the target merge task, the target merge node module loads the merge mark information from the at least two source data clusters based on the cluster identifier of the target data cluster to see whether it can be loaded into the merge mark information to determine which data synchronization scenario it is.

[0076] Step 209 : If the merge mark information is loaded, the target merge node module obtains the updated data table from at least two source data clusters for merging, and synchronizes the merged updated data table to the target data cluster.

[0077] If a merge marker is loaded, it indicates that a data merge and synchronization operation has previously been performed between the source data cluster and the target data cluster. Only the updated data in the source data cluster since the last merge and synchronization operation needs to be synchronized. The corresponding target merge node module retrieves the updated data tables from at least two source data clusters, merges them, and synchronizes the merged updated data tables to the target data cluster. The updated data table is determined by the operation node position of the last data merge and synchronization operation indicated by the merge marker information.

[0078] Step 210 : If the merge mark information is not loaded, all data tables are obtained from at least two source data clusters through the target merge node module for merging, and all merged data tables are synchronized to the target data cluster.

[0079] If the merge marker information is not loaded, it means that the data merge and synchronization operation between the source data cluster and the target data cluster has not been performed before, and the full data needs to be merged synchronously. The corresponding target merge node module obtains all data tables from at least two source data clusters, merges them, and synchronizes all the merged data tables to the target data cluster.

[0080] Step 211 : After synchronizing the merged updated data table or all merged data tables to the target data cluster, update the merge mark information.

[0081] To avoid duplicate data merging and synchronization, after synchronizing the merged updated database or all merged databases to the target data cluster, the merge marker information needs to be updated so that the next time the data merging and synchronization operation is performed, the data tables that need to be merged and synchronized can be determined based on the updated merge marker information.

[0082] In this embodiment, by configuring the merge mark information, different data merge synchronization scenarios can be distinguished based on whether the merge mark information can be loaded, thereby avoiding repeated data merge synchronization operations; moreover, by comparing the node status information and the task requirement information of the target merge task, a better execution task node can be selected for the target merge task to ensure the real-time and smooth execution of the target merge task.

[0083] In order to save resources of the merged service cluster, the task module will also periodically recycle low-load nodes. Please refer to Figure 3 , which shows a flow chart of another cluster data merging method according to an exemplary embodiment of the present application. This method is used to merge service clusters as an example for exemplary description. Figure 3 As shown, the method includes:

[0084] Step 301: Obtain node status information of each merging node module through the load balancing module, and feed back the node status information to the task module.

[0085] Step 302: Obtain the current load factor, load lower limit and number of currently running tasks of each merge node module from the node status information through the task module.

[0086] Step 303 : The task module selects a third candidate merging node module from each merging node module, wherein the third candidate merging node module has a current load factor less than a load lower limit and a current number of running tasks less than a second number threshold within a preset time period.

[0087] Step 304: If a third candidate merge node module exists, the task module deletes the third candidate merge node module.

[0088] The task module is also configured with a strategy to recycle low-load nodes: if the LoadFactor (current load node) of some merge node modules is <MinLoadThreshold(负载下限),且运行任务数长期为0,则考虑销毁该合并节点模块,节约资源。

[0089] In one possible implementation, the task module obtains the current load factor, load lower limit, and current number of running tasks of each node module from the node status information, and screens out the merged node modules whose current load factor is less than the load lower limit and whose current number of running tasks is less than a second threshold value. If such a merged node module exists, it is further determined whether its current load factor is less than the load lower limit and its current number of running tasks is less than the second threshold value within a preset time period. If so, the merged node module is determined as the third candidate merged node module, indicating that it is a low-load node. The task module saves resources of the merged service cluster by deleting the third candidate merged node module.

[0090] For example, the second quantity threshold can be 1.

[0091] In this embodiment, the low-load nodes are screened by monitoring the current load factor, the load lower limit and the current running task number of the merge node module, and the cluster resources are saved by deleting the low-load nodes.

[0092] Please refer to Figure 4 , which is a structural diagram of a merged service cluster provided by an embodiment of the present application. For example, as shown in Figure 4 , the merged service cluster 400 includes a task module 401, a load balancing module 402 and at least one merge node module 403.

[0093] The load balancing module 402 is configured to acquire node state information of each merge node module and feed back the node state information to the task module.

[0094] The task module 401 is configured to, in a case where a target merge task is received, determine a target merge node module for executing the target merge task based on the node state information and task requirement information of the target merge task.

[0095] The task module 401 is further configured to issue the target merge task to the target merge node module.

[0096] The target merge node module is configured to determine at least two source data clusters and a single target data cluster indicated by the target merge task based on the received target merge task.

[0097] The target merge node module is further configured to merge data tables in the at least two source data clusters and synchronize the merged data tables to the target data cluster.

[0098] Optionally, the target merge node module is further configured to:

[0099] Load merge mark information from the at least two source data clusters based on a cluster identifier of the target data cluster;

[0100] If the merge mark information is loaded, acquire update data tables from the at least two source data clusters for merging, and synchronize the merged update data tables to the target data cluster.

[0101] If the merge mark information is not loaded, acquire all data tables from the at least two source data clusters for merging, and synchronize the merged all data tables to the target data cluster.

[0102] Optionally, the target merge node module is further configured to:

[0103] After synchronizing the merged updated data table or all the merged data tables to the target data cluster, the merge mark information is updated.

[0104] Optionally, the task module 401 is further configured to:

[0105] Acquire the current load factor, load upper limit and current idle resources of each of the merged node modules from the node status information;

[0106] Selecting a first candidate merging node module from each of the merging node modules, wherein the current load factor is less than the load upper limit and the current idle resources meet the task requirement information of the target merging task;

[0107] If the first candidate merge node module exists, determining the first candidate merge node module with the smallest current load factor as the target merge node module;

[0108] If the first candidate merge node module does not exist, a new merge node module is created, and the new merge node module is determined as the target merge node module.

[0109] Optionally, the task module 401 is further configured to:

[0110] If the first candidate merge node module does not exist and the number of nodes of the existing merge node module is less than a first number threshold, the newly added merge node module is created.

[0111] Optionally, the task module 401 is further configured to:

[0112] If the first candidate merge node module does not exist, and the number of nodes of the existing merge node modules is greater than or equal to the first number threshold, obtaining the expansion permission of each merge node module from the node status information;

[0113] If there is a second candidate merging node module with the expansion authority, the second candidate merging node module with the smallest current load factor is determined as the target merging node module.

[0114] Optionally, the task module 401 is further configured to:

[0115] Obtaining the current load factor, load lower limit and number of currently running tasks of each of the merge node modules from the node status information;

[0116] Selecting a third candidate merging node module from each of the merging node modules, wherein the current load factor is less than the load lower limit and the number of currently running tasks is less than the second number threshold within a preset time period;

[0117] If the third candidate merge node module exists, delete the third candidate merge node module.

[0118] It should be noted that the merged service cluster in the above embodiment can be implemented as an electronic device; or, the merged service cluster is a collection of electronic devices, and the task module, load balancing module and merge node module can all be implemented as electronic devices.

[0119] The exemplary embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, wherein the computer program, when executed by the at least one processor, causes the electronic device to perform the cluster data merging method according to the exemplary embodiments of the present application.

[0120] An exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the cluster data merging method according to an embodiment of the present application.

[0121] An exemplary embodiment of the present application further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to perform the cluster data merging method according to the embodiment of the present application.

[0122] refer to Figure 5 , a block diagram of an electronic device 500 that can serve as a server or client of the present application will now be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0123] like Figure 5As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0124] Multiple components within electronic device 500 are connected to I / O interface 505, including an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. Input unit 506 can be any type of device capable of inputting information into electronic device 500. Input unit 506 can receive input numeric or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 508 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 509 allows electronic device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0125] The computing unit 501 may be a variety of general and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, Figure 1 、 Figure 2 The illustrated method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. In some embodiments, the computing unit 501 may be configured to execute the computer program in any other suitable manner (e.g., by means of firmware). Figure 1 、 Figure 2 The method shown.

[0126] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0131] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

Claims

1. A cluster data merging method, characterized in that: The method is applied to a merged service cluster, wherein the merged service cluster includes a task module, a load balancing module, and at least one merge node module. The method includes: Obtaining node status information of each of the merging node modules through the load balancing module, and feeding back the node status information to the task module; In the case of receiving a target merge task, determining, by the task module, a target merge node module for executing the target merge task based on the node status information and the task requirement information of the target merge task; Sending the target merging task to the target merging node module through the task module; Determining, by the target merging node module based on the received target merging task, at least two source data clusters and a single target data cluster indicated by the target merging task; The acquired data tables in the at least two source data clusters are merged by the target merging node module, and the merged data tables are synchronized to the target data cluster.

2. The method according to claim 1, characterized in that The merging of the acquired data tables in the at least two source data clusters by the target merging node module, and synchronizing the merged data tables to the target data cluster, includes: Loading merge mark information from the at least two source data clusters based on the cluster identifier of the target data cluster by the target merge node module; If the merge mark information is loaded, the updated data table is obtained from the at least two source data clusters through the target merge node module to merge, and the merged updated data table is synchronized to the target data cluster; If the merge mark information is not loaded, all data tables are acquired from the at least two source data clusters through the target merge node module for merging, and all merged data tables are synchronized to the target data cluster.

3. The method according to claim 2, characterized in that The method further comprises: After synchronizing the merged updated data table or all the merged data tables to the target data cluster, the merge mark information is updated.

4. The method according to any one of claims 1 to 3, characterized in that: The determining, by the task module, a target merging node module for executing the target merging task based on the node status information and the task requirement information of the target merging task includes: Obtaining the current load factor, load upper limit and current idle resources of each of the merged node modules from the node status information through the task module; Screening, by the task module, from the respective merging node modules, a first candidate merging node module whose current load factor is less than the load upper limit and whose current idle resources meet the task requirement information of the target merging task; If the first candidate merge node module exists, determining the first candidate merge node module with the smallest current load factor as the target merge node module through the task module; If the first candidate merge node module does not exist, a new merge node module is created through the task module, and the new merge node module is determined as the target merge node module.

5. The method according to claim 4, characterized in that If the first candidate merge node module does not exist, creating a new merge node module through the task module includes: If the first candidate merge node module does not exist and the number of nodes of the existing merge node module is less than a first number threshold, the new merge node module is created by the task module.

6. The method according to claim 5, characterized in that The method further comprises: If the first candidate merge node module does not exist, and the number of nodes of the existing merge node modules is greater than or equal to the first number threshold, obtaining the expansion permission of each merge node module from the node status information through the task module; If there is a second candidate merging node module with the expansion authority, the second candidate merging node module with the smallest current load factor is determined as the target merging node module through the task module.

7. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Obtaining the current load factor, load lower limit and number of currently running tasks of each of the merged node modules from the node status information through the task module; Screening, by the task module, from the respective merging node modules a third candidate merging node module whose current load factor is less than the load lower limit and whose number of currently running tasks is less than a second quantity threshold within a preset time period; If the third candidate merge node module exists, the third candidate merge node module is deleted by the task module.

8. A merged service cluster, characterized in that: The merged service cluster includes a task module, a load balancing module and at least one merge node module: The load balancing module is used to obtain node status information of each of the merging node modules and feed back the node status information to the task module; The task module is configured to, upon receiving a target merge task, determine a target merge node module for executing the target merge task based on the node status information and the task requirement information of the target merge task; The task module is further configured to send the target merging task to the target merging node module; a target merging node module, configured to determine, based on the received target merging task, at least two source data clusters and a single target data cluster indicated by the target merging task; The target merging node module is further configured to merge the acquired data tables in the at least two source data clusters, and synchronize the merged data tables to the target data cluster.

9. An electronic device comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, enable the processor to perform the cluster data merging method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the cluster data merging method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Merging processing method and related equipment

    CN112711564A

  • Task scheduling method and device, equipment, storage medium and computer program product

    CN116541142A