Data merging processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a data merging processing method, device, electronic device and storage medium, the method comprising: when a merging trigger condition is met, according to the condition type of the merging trigger condition, determining the shard service node that matches the condition type among N shard service nodes as the target shard service node; according to the determination rule corresponding to the condition type of the merging trigger condition, determining the target shard data to be merged corresponding to the target shard service node from the multiple shard data stored by the target shard service node; sending a merging processing instruction to the target shard service node, so that the target shard service node performs a merging operation on the target shard data according to the merging processing instruction. The data merging processing method provided by the present disclosure is conducive to saving system resources, and by flexibly setting the merging trigger condition, it can make the execution timing and execution method of the merging operation more in line with business needs, and can adapt to a variety of application scenarios.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data merging and processing method, device, electronic device and storage medium. Background Art

[0002] In databases, especially in column-based database clusters, data synthesis (compact) operations are often required to merge data in data storage units, so as to achieve the purpose of improving data reading and writing efficiency, clearing invalid files, etc. Taking the Hbase database as an example, the following method is usually used to perform merge operations in Hbase: set a preset natural day cycle, and when the preset natural day cycle arrives, randomly select a time to perform merge operations on the shard data in all shard service nodes in Hbase. In this way, the merge operation can only be performed on all shard service nodes, resulting in a long execution time and a large occupation of system resources. In addition, the mode is single and can only be applied to application scenarios of scheduled execution. Summary of the invention

[0003] The present disclosure provides a data merging and processing method, device, electronic device and storage medium.

[0004] In a first aspect, the present disclosure provides a data merging processing method, comprising:

[0005] Applied to a column-based database cluster, the column-based database cluster includes N shard service nodes and management nodes, each shard service node stores multiple shard data; N is an integer greater than 1; the method is executed by the management node, and the method includes:

[0006] When a merge trigger condition is met, according to a condition type of the merge trigger condition, a shard service node among the N shard service nodes that matches the condition type is determined as a target shard service node;

[0007] Determine, according to a determination rule corresponding to the condition type of the merge trigger condition, the target shard data to be merged corresponding to the target shard service node from the multiple shard data stored by the target shard service node;

[0008] A merge processing instruction is sent to the target shard service node, so that the target shard service node performs a merge operation on the target shard data according to the merge processing instruction.

[0009] In a second aspect, the present disclosure provides a data merging processing device, comprising:

[0010] a confirmation unit, configured to, when a merge trigger condition is met, determine, according to a condition type of the merge trigger condition, a shard service node among the N shard service nodes that matches the condition type as a target shard service node;

[0011] The confirmation unit is further configured to determine, from the plurality of shard data stored by the target shard service node, the target shard data to be merged corresponding to the target shard service node according to a determination rule corresponding to the condition type of the merge trigger condition;

[0012] The sending unit is used to send a merge processing instruction to the target shard service node, so that the target shard service node performs a merge operation on the target shard data according to the merge processing instruction.

[0013] In a third aspect, the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor to enable the at least one processor to perform the above method.

[0014] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above method when executed by a processor / processing core.

[0015] The embodiments provided by the present disclosure set a variety of types of merge trigger conditions. For different types of merge trigger conditions, different ranges of shard service nodes are selected as target shard nodes. Then, according to a determination rule that matches the condition type of the merge trigger condition, the corresponding shard data in the target shard service node is determined as the target shard data. Thus, only the target shard data needs to be merged, rather than each shard data in all shard service nodes. This is beneficial for saving system resources. Moreover, by flexibly setting the merge trigger conditions, the execution timing and execution method of the merge operation can be more in line with business needs and can adapt to a variety of application scenarios.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 This is the architecture diagram of HBase;

[0019] Figure 2 A flowchart of a data merging processing method provided by an embodiment of the present disclosure;

[0020] Figure 3 A flowchart of the steps of setting node concurrency parameters in a data merging processing method provided in an embodiment of the present disclosure;

[0021] Figure 4 A flowchart of another data merging processing method provided by an embodiment of the present disclosure;

[0022] Figure 5 A block diagram of a data merging processing device provided in an embodiment of the present disclosure;

[0023] Figure 6 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0025] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0026] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0027] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0028] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0029] In order to improve the effectiveness of data merging and reduce the impact of data merging on the normal use of a database, an embodiment of the present application proposes a data merging processing method, which sets multiple types of merge trigger conditions, selects shard service nodes in different ranges as target shard nodes for different types of merge trigger conditions, and then determines the corresponding shard data in the target shard service node as the target shard data according to a determination rule that matches the condition type of the merge trigger condition, so that only the target shard data needs to be merged, rather than having to perform the merge operation on each shard data in all shard service nodes, which is beneficial to saving system resources, and by flexibly setting the merge trigger conditions, the execution timing and execution method of the merge operation can be more in line with business needs and can adapt to a variety of application scenarios.

[0030] According to the data merging processing method of the embodiment of the present application, it can be executed by an electronic device such as a terminal device or a server. The terminal device can be a vehicle-mounted device, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.; the server can be an independent physical server, a server cluster or a cloud server capable of cloud computing. The data merging processing method can be implemented specifically by a processor calling a computer-readable program instruction stored in a memory of an electronic device.

[0031] The data merging processing method according to the embodiment of the present application can be applied to various types of databases. For example, the data merging processing method according to the embodiment of the present application is applied to the column-based database cluster Hbase. Figure 1 , Figure 1 The following diagram shows the architecture of HBase. HBase contains at least three types of server nodes in the master-slave mode architecture:

[0032] (1) Region Server (RS), which contains multiple regions. RS is used to provide data read and write functions. When users operate data through the client, they interact directly with RS.

[0033] (2) The master node (HMaster) is used to manage multiple RSs and process the allocation of shard data in RSs and operations such as adding and deleting tables.

[0034] (3) State management node (Zookeeper). Zookeeper, as a distributed coordination service, records and manages the state of each node (including the master node and the distribution service node) in the entire column database cluster.

[0035] In Hbase, shard data is the data storage unit for Hbase's internal data management. One shard data is associated with multiple column data (HFiles). The essence of the merge operation is to merge multiple Hfiles with smaller data volumes into one Hfile with larger data volumes.

[0036] In addition, Hbase itself does not have the function of data replication and maintenance of data copies, but relies on the distributed file system (HDFS, Hadoop Distributed File System) to provide reliable and stable storage for HBase. Multiple data nodes (DataNode) are set in HDFS. RS stores the HFile stored in the shard data to the data node through the distributed file system client (DFS Client) to achieve persistence. Usually RS is deployed on the same server with some data nodes. RS reads data from the data node (i.e., local data node) deployed on the same server with it, with high read and write efficiency and low occupancy rate of network resources. Therefore, ideally, the Hfile in the shard data in an RS can be stored as much as possible on the data node deployed on the same server as the RS. In other words, the localization rate of RS is as high as possible. The merge operation will read the small Hfile data on the remote data node (i.e., the data node deployed on a different server from the RS) and rewrite it to the merged Hfile while merging the Hfile. The merged Hfile will store a copy on the local data node. In this way, the merge operation can quickly improve the localization rate of RS.

[0037] The localization rate of an RS is: the proportion of column data stored in the same storage structure as the RS among multiple column data associated with multiple shard data stored in the RS. The storage structure may be a server. Since the column data is stored in a data node, the localization rate of the RS can be obtained by determining the ratio of the number of column data associated with each shard data in the RS stored in a data node deployed in the same storage structure (e.g., a server) as the RS to the total number of column data associated with each shard data in the RS.

[0038] Figure 2 A flowchart of a data merging method provided by an embodiment of the present disclosure. Figure 2 The method is applied to a column-based database cluster, which includes N sharding service nodes and management nodes. Each sharding service node stores multiple sharding data, where N is an integer greater than 1. It should be noted that the method can be executed by a management node, which can be a master node in the column-based database cluster (e.g. Figure 1 master node in the Figure 1 Specifically, the method includes:

[0039] S1. When a merge trigger condition is met, according to the condition type of the merge trigger condition, a shard service node among N shard service nodes that matches the condition type is determined as a target shard service node.

[0040] The merge trigger condition is a condition that can trigger the merge operation. By setting multiple types of merge trigger operations, different ranges of RSs can be selected as the execution subjects of the merge operation according to different types of merge trigger operations. Among them, the condition types of the merge trigger condition include, for example: global node type and specified node type.

[0041] In the case where the condition type of the merge trigger condition is a global node type, the RSs that match the condition type of the merge trigger condition are all RSs in the column-based database cluster. In other words, the RSs that match the condition type of the merge trigger condition in S1 are determined as target RSs specifically as follows: all RSs are determined as target RSs. In the case where the condition type of the merge trigger condition is a specified node type, the RSs that match the condition type of the merge trigger condition are RSs that belong to the specified node in the column-based database cluster. In other words, the RSs that match the condition type of the merge trigger condition in S1 are determined as target RSs specifically as follows: the RSs that belong to the specified node are determined as target RSs.

[0042] The column database cluster is a cluster of RSs in a database (eg, Hbase). By setting the merging trigger conditions of the global node type and the merging trigger conditions of the specified node type, the execution scope of the merging operation can be controlled.

[0043] In some achievable ways, the condition types of the merge trigger conditions in S1 include: global node type and specified node type. For two different condition types, S1 may specifically include:

[0044] Sub-step a: When the condition type of the merge trigger condition is a global node type, in S1, according to the condition type of the merge trigger condition, the shard service nodes among the N shard service nodes that match the condition type are determined as the target shard service nodes, including: all shard service nodes are determined as the target shard service nodes, in other words, the RSs that match the condition type of the merge trigger condition are all RSs in the column database cluster.

[0045] Sub-step b: When the condition type of the merge trigger condition is a specified node type, in S1, according to the condition type of the merge trigger condition, the shard service node that matches the condition type among the N shard service nodes is determined as the target shard service node, including: determining the service node belonging to the specified node as the target shard service node, in other words, the RS that matches the condition type of the merge trigger condition is the RS belonging to the specified node in the column database cluster.

[0046] Among them, the designated nodes may include any one or more of the following nodes: shard service nodes in the whitelist; shard service nodes associated with the node cluster expansion operation; shard service nodes associated with the node cluster reduction operation; shard service nodes whose localization rate is lower than a preset threshold.

[0047] By setting the above two condition types, the execution scope of the merge operation can be controlled. When the condition type of the merge trigger condition is a global node type, the merge operation will be performed on the entire columnar database cluster, which is beneficial to improve the data reading and writing efficiency and localization rate of the entire cluster and maximize the release of storage space; when the condition type of the merge trigger condition is a specified node type, the merge operation is only performed on the specified node, which is beneficial to reduce unnecessary system resource usage and reduce the impact on other businesses. In addition, the localization rate and read and write efficiency of the specified node can be quickly improved to meet the business needs of the specified node.

[0048] In some examples, for the case of sub-step a, that is, the condition type of the merged trigger condition is a global node type, the merged trigger condition in S1 may specifically include: detecting that a preset period has arrived, and / or detecting that the current time belongs to a preset time period.

[0049] Specifically, a preset period can be set, and when it is detected that the preset period has arrived, all RSs are determined as target RSs. That is to say, a fixed merge operation is performed on the column-based database cluster in each preset period to optimize the data reading and writing efficiency of each RS in the column-based database cluster, clear redundant data (for example, invalid, expired, redundant data) in each RS, and improve the localization rate of RS. The preset period can be set specifically according to multiple factors, for example, at least one of the redundant data generation period and the data reading and writing efficiency reduction period.

[0050] Furthermore, in some cases, it is necessary to consider the system's business peak and off-peak periods to avoid performing merge operations during peak periods. Therefore, it is also possible to detect whether the time at which the merge operation is to be performed currently belongs to a preset period.

[0051] The preset time period includes a low-peak service period, and the low-peak service period is specifically a period when the total service volume of each RS is less than a preset service threshold. Specifically, the low-peak service period can be determined in one of the following ways:

[0052] Method 1: Set fixed off-peak hours.

[0053] Method 2: Dynamically monitor the traffic volume of each RS and set the period when the total traffic volume is less than the preset traffic threshold as the traffic off-peak period.

[0054] That is to say, based on the business volume of RSs in the system in each time period over a period of time, the time period with the smallest total business volume can be designated as a low-peak business period, for example: 0:00-6:00; the business volume of each RS can also be dynamically monitored. If the total business volume of RSs in a certain period of time is less than the preset business threshold, the period will be regarded as a low-peak business period.

[0055] S2. According to the determination rule corresponding to the condition type of the merge trigger condition, determine the target shard data to be merged corresponding to the target shard service node from the multiple shard data stored in the target shard service node.

[0056] Merge trigger conditions of different condition types correspond to different determination rules. Furthermore, among the shard data stored in the target RS, shard data of a corresponding range is selected according to the determination rule as execution data for executing the merge operation, thereby further controlling the execution scope of the merge operation.

[0057] In some examples, the determination rule indicates that the target shard data is determined based on the update time of each shard data in the target shard service node. Based on this, S2 includes: determining the target shard data to be merged corresponding to the target RS according to the update time of the target RS.

[0058] For the shard data in RS, it takes a certain amount of time to generate fragmented Hfiles, redundant data, or a decrease in localization rate, rather than instantaneously. Therefore, it is usually not necessary to perform a merge operation on the same shard data at a high frequency. Therefore, it is possible to determine whether the shard data needs to be merged based on the time interval between the latest update time of each shard data recorded by RS and the current time. Specifically, S2 includes:

[0059] Sub-step 1: Get the update time of each shard data in the target shard service node.

[0060] Sub-step 2: According to the update time of each shard data and the current time, the shard data that meets the merge execution condition is obtained from each shard data, and the obtained shard data is determined as the target shard data to be merged.

[0061] Among them, the time interval between the update time and the current time of the shard data that meets the merge execution condition is greater than the preset interval threshold. In other words, the merge execution condition includes: the time interval between the update time and the current time is greater than the preset interval threshold. If the time interval between the most recent update time of a shard data and the current time is greater than the preset interval threshold, it means that the shard data has not been merged for a long enough time, and it is determined as the target shard data; if the time interval between the most recent update time of a shard data and the current time is less than or equal to the preset interval threshold, it means that the shard data has just been merged recently, and there is no need to merge this time, so it is not determined as the target shard data.

[0062] The preset interval threshold may be the same as the preset period. In this way, it is possible to avoid too high a frequency of performing the merging operation on the fragmented data, thereby avoiding a waste of system resources.

[0063] It should be noted that the above sub-steps 1 and 2 can be executed after S1 if the merge trigger conditions of any conditional type are met. In other words, whether the target RS is all RSs or the target RS is the RS of a specified node, after determining the target RS, the target shard data in the target RS can be further determined according to the update time.

[0064] In some examples, for the case of sub-step b, that is, the condition type of the merge trigger condition is a specified node type, the merge trigger condition in S1 may specifically include: detecting that at least one shard service node among the N shard service nodes belongs to the specified node.

[0065] The designated node includes at least one of the following:

[0066] (1) Shard service nodes in the whitelist.

[0067] (2) Shard service nodes associated with the node cluster expansion operation.

[0068] (3) Shard service nodes associated with the node cluster scaling operation.

[0069] (4) Shard service nodes whose localization rate is lower than the preset threshold.

[0070] For (1), a whitelist can be set up, and at least one RS responsible for special services can be added to the whitelist. Special services, for example, services that generate a large amount of redundant data, can be merged regularly for maintenance of the RS in the whitelist. For (2), the RS associated with the node cluster expansion operation, that is, the RS obtained by the expansion operation, after obtaining an RS through the expansion operation, the localization rate of the RS is usually 0%, so it is necessary to perform a merge operation to quickly improve the localization rate of the RS. For (3), the RS associated with the node cluster reduction operation, that is, the RS related to the data migration of the original RS after the reduction, after performing a reduction operation on an RS, the shard data in the RS will be migrated to the data nodes corresponding to other RSs according to certain rules or randomly, so the localization rates of other RSs may change, so it is necessary to perform a merge operation to quickly improve the localization rates of these RSs. For (4), by directly detecting the localization rate of each RS, it can be more intuitive to determine which RSs need to perform a merge operation to quickly improve their own localization rates.

[0071] It should be noted that RS is usually deployed in the same server as some data nodes. RS has high read and write efficiency when reading data from data nodes (i.e. local data nodes) deployed in the same server as RS, and has low occupancy rate of network resources. Therefore, ideally, Hfiles in shard data in an RS can be stored as much as possible in data nodes deployed in the same server as RS. In other words, the localization rate of RS is as high as possible. When merging Hfiles, the merge operation will read the small Hfile data on the remote data node (i.e. data node deployed on a different server from the RS) and rewrite it to the merged Hfile. The merged Hfile will store a copy on the local data node. In this way, the merge operation can quickly improve the localization rate of RS.

[0072] Further, in the case of (4), whether each RS is a designated node can be detected in the following manner:

[0073] Sub-step 1: Dynamically detect the localization rate of each RS;

[0074] Sub-step 2: According to the detection result of the localization rate of each RS, the RS whose localization rate is lower than a preset threshold is determined as a designated node.

[0075] Further, in the above case, S2 includes: determining that all shard data in the shard service node belonging to the designated node is the target shard data.

[0076] Of course, in some examples, in S2, the target shard data in the RS belonging to the specified node can be further determined according to the node parameters corresponding to the specified node type, that is, S2 includes: obtaining the node parameters of the target RS belonging to the specified node, and determining the corresponding target shard data to be merged in the target shard service node according to the node parameters. Specifically, the node parameter can be a localization rate. Then, when the specified node is (1) a shard service node in the whitelist, (2) a shard service node associated with a node cluster expansion operation, or (3) a shard service node associated with a node cluster reduction operation, and the number of specified nodes is multiple, the determination rule corresponding to the specified node type indicates that the target shard data is determined according to the localization rate of each shard service node, then S2 includes:

[0077] Sub-step 1: Obtain the localization rate of each target shard service node among multiple target shard service nodes.

[0078] Sub-step 2: Determine the shard data in the target shard service node whose localization rate is lower than a preset threshold as the target shard data to be merged.

[0079] When multiple RSs are determined as designated nodes through preset nodes or by detecting expansion or reduction operations, the shard data in the target RS that needs to perform the merge operation can be further accurately screened out as the target shard data based on the localization rate of each target RS, thereby further avoiding waste of system resources.

[0080] S3. Send a merge processing instruction to the target shard service node, so that the target shard service node can perform a merge operation on the target shard data according to the merge processing instruction.

[0081] After determining the target shard data that needs to be merged, a merge processing instruction is sent to the target RS, so that the target shard service node performs the merge operation on the target shard data.

[0082] In some examples, in S3, the merge processing instruction sent to the target RS includes at least a node concurrency parameter, and the node concurrency parameter is used to instruct the target RS to create multiple concurrent merge tasks corresponding to the node concurrency parameter, wherein each concurrent merge task is used to perform the merge operation in parallel.

[0083] Among them, the node concurrency parameter represents the number of tasks that execute the merge operation in parallel in an RS. Usually, RS includes a huge number of shard data, so the number of target shard data is usually large. If the number of tasks that execute the merge operation is not limited, the business concurrency will increase, which will have a greater impact on the processing of other businesses. Therefore, in the embodiment of the present disclosure, a node concurrency parameter is set, and multiple concurrent merge tasks are generated according to the node concurrency parameter. That is to say, the number of concurrent merge tasks that execute the merge operation is limited by the node concurrency parameter to avoid too many tasks executing the merge operation, and there are no other resources to handle businesses other than the merge operation.

[0084] In some examples, concurrent merge tasks can be created by threads or processes.

[0085] For some examples, see Figure 3 , Figure 3 The flowchart of setting the node concurrency parameter in the embodiment of the present disclosure is shown. The node concurrency parameter is determined by the following method:

[0086] S01. Obtain parameter values ​​of concurrent reference parameters corresponding to a target shard service node.

[0087] S02. According to the obtained parameter value of the concurrency reference parameter, set the node concurrency parameter of the target shard service node.

[0088] The concurrent reference parameters include at least one of the following: system performance parameters, bandwidth parameters, CPU data parameters, and disk input / output parameters.

[0089] The concurrency reference parameters are parameters associated with the concurrency of the database system. Specifically, the system performance parameters may be the total number of threads in the system, the CPU data parameters may be the total amount of parallel processing loads that the CPU can handle, and the disk input / output parameters may be the number of input interfaces and the number of output interfaces of the disk. By setting the node concurrency parameters with reference to these concurrency reference parameters, it is possible to avoid excessive occupation of system resources by the merge operation, thereby affecting other services.

[0090] Furthermore, the node concurrency parameters can be determined in two specific ways:

[0091] Method 1: Static determination method.

[0092] In the first method, S01 is to obtain various parameter values ​​in advance, that is, S01 includes: obtaining the parameter value of the concurrency reference parameter corresponding to the target shard service node, in this case, it also includes S03: writing the set node concurrency parameter into the configuration file. Then in S3, the merge processing instruction is generated according to the following method:

[0093] Read the node concurrency parameters from the configuration file and add them to the merge processing instructions.

[0094] Method 2: Dynamic determination method.

[0095] In the second method, S02 is to dynamically obtain various parameter values, that is, S01 includes: dynamically obtaining the parameter value of the concurrency reference parameter corresponding to the target shard service node, and S02 includes: dynamically calculating the node concurrency parameter of the target shard service node according to the obtained parameter value of the concurrency reference parameter.

[0096] Among them, the first method can directly read the concurrent reference parameters, thereby saving network resources and system resources, and the second method can set the concurrent reference parameters in real time according to the current system status.

[0097] In some examples, the method provided by the embodiments of the present disclosure is applicable to a column-based database cluster. When the number of target shard service nodes included in the column-based database cluster is multiple, setting the node concurrency parameters of the target shard service nodes includes:

[0098] Get the cluster concurrency parameter corresponding to the columnar database cluster, and set the node concurrency parameter of each target shard service node in combination with the cluster concurrency parameter, so that the cumulative result of the node concurrency parameter of each target shard service node in a node cluster is not greater than the cluster concurrency parameter.

[0099] The cluster concurrency parameter indicates the number of tasks that execute the merge operation in parallel in each RS in the column-based database cluster. Multiple RSs form a column-based database cluster and process services collaboratively. Therefore, in addition to considering the concurrency of a single RS, the concurrency of the node cluster can also be considered. By setting the cluster concurrency parameter, the total number of concurrent merge tasks that execute the merge operation in each RS is limited to avoid too many tasks executing the merge operation in the node cluster, and there are no other resources to handle services other than the merge operation.

[0100] In the data merge processing method provided by the embodiment of the present disclosure, since the number of concurrent merge tasks is limited by the node concurrency parameters and / or cluster concurrency parameters in the above steps, that is, a fixed number of concurrent merge tasks are allocated to the merge operation tasks, therefore, in order to improve the efficiency of concurrent merge tasks and realize parallelization, after S2 and before S3, the method may include: according to the number of target shard data, allocating corresponding target shard data to the concurrent merge tasks.

[0101] Specifically, in some examples, after S2 and before sending the merge processing instruction to the target shard service node in S3, the method further includes:

[0102] Sub-step 1: Divide the target shard data to be merged into multiple sub-data, and allocate corresponding sub-data to each concurrent merge task.

[0103] Sub-step 2: carrying the data identification information of the sub-data allocated to each concurrent merging task in the merging processing instruction.

[0104] In this case, the merge processing instruction in S3 further includes: data identification information of the sub-data allocated to each concurrent merge task.

[0105] Dividing the data to be processed in the target shard data into multiple sub-data and distributing them to each concurrent merge task for parallel processing can increase the efficiency of concurrent processing. In order to facilitate matching concurrent merge tasks with sub-data and distinguish the source of sub-data, the data identification information of the sub-data can be carried in the merge processing instruction. The data identification information of a sub-data can be the corresponding identifier of the shard data to which the sub-data belongs in the metadata table, so that the data identification information of the sub-data belonging to a shard data is the same.

[0106] Based on the control of task allocation by this method, the task processing status may be further detected. Specifically, in some examples, after S2, after sending the merge processing instruction to the target shard service node in S3, the method further includes:

[0107] Sub-step 1: Obtain the task processing status of each concurrent merge task in the target shard service node.

[0108] Sub-step 2: Send a task control instruction to the target shard service node according to the task processing status of each concurrent merge task, so that the target shard service node can control the concurrent merge task according to the task control instruction.

[0109] The task processing status includes one of the following: completed status, paused status, failed status, abnormal status, etc.

[0110] After the merge processing instruction is sent to the RS, the RS allocates concurrent merge tasks to perform the merge operation. The management node (e.g. Figure 1 Client or Figure 1 The master node HMaster in the system needs to obtain the task processing status of each concurrent merge task so as to control each concurrent merge task according to the task processing status.

[0111] If the task processing status of a concurrent merge task is completed, sub-step 2 includes:

[0112] The sub-data are reallocated for the concurrent merge tasks whose task processing status is the completed status; the task control instruction is used to control the concurrent merge tasks whose task processing status is the completed status to perform merge processing on the reallocated sub-data.

[0113] Then the task control instruction in sub-step 2 is used to control the concurrent merging task in the completed state to perform merging processing on the reallocated sub-data.

[0114] In this case, when the sub-data allocated to each concurrent merge task contained in the merge processing instruction is only part of the data in the target shard data (that is, all the data in the non-target shard data), it is necessary to further allocate new sub-data for the processed concurrent merge tasks until all the sub-data contained in the target shard data are processed.

[0115] If the task processing status of a concurrent merge task is a failure status or an abnormal status, sub-step 2 includes:

[0116] Send a task creation instruction to the target RS corresponding to the concurrent merge task whose task processing status is a failed state or an abnormal state, so that the RS can recreate a new concurrent merge task; assign the sub-data matching the concurrent merge task whose task processing status is a failed state or an abnormal state to the new concurrent merge task.

[0117] If the task processing status of a concurrent merge task is in the paused state, sub-step 2 includes:

[0118] A task start instruction is sent to a target RS corresponding to a concurrent merge task whose task processing status is a failure state or an abnormal state, so that the RS controls the concurrent merge task to restart.

[0119] In some examples, the data merging processing method provided by the embodiments of the present disclosure further includes: when the time interval corresponding to the task waiting time arrives, executing the above sub-step one: obtaining the task processing status of each concurrent merging task in the target shard service node.

[0120] The task waiting time is also the thread / process waiting time, which is equivalent to the cycle of querying the task processing status. In some examples, the task waiting time can be written into the above configuration file.

[0121] By dynamically monitoring the task processing status, corresponding task control can be performed according to the real-time processing status of each concurrent merge task, thereby improving the system's fault tolerance mechanism and robustness; and the method provided in the embodiment of the present disclosure performs real-time detection and data allocation operations on each concurrent merge task, which can improve the flexibility of the merge operation.

[0122] According to the data merging processing method of the embodiment of the present disclosure, by setting a plurality of types of merging trigger conditions, different ranges of shard service nodes are selected as target shard nodes for different types of merging trigger conditions, and then the corresponding shard data in the target shard service node is determined as the target shard data according to the determination rule matching the condition type of the merging trigger condition, so that only the merging operation needs to be performed on the target shard data, instead of having to perform the merging operation on each shard data in all shard service nodes, which is beneficial to saving system resources, and by flexibly setting the merging trigger conditions, the execution timing and execution method of the merging operation can be more in line with business needs and can adapt to a variety of application scenarios.

[0123] It should be noted that, taking the method provided in the embodiment of the present disclosure as applied to Hbase as an example, the execution subject of the method can be any one of the master node, the client, and multiple RS nodes, which is not limited here. The target RS receives the merge processing instruction as the execution subject of the merge operation, and the target shard data is the execution object of the merge operation.

[0124] Further, a specific example is used to illustrate the data merging processing method provided in the embodiment of the present application, see Figure 4 , Figure 4 A flowchart of a data merging method is provided for an embodiment of the present application. The method includes:

[0125] S100: When a merge trigger condition is met, according to the condition type of the merge trigger condition, a shard service node matching the condition type among the N shard service nodes is determined as a target shard service node.

[0126] Specifically, the condition types of the merged trigger conditions include, for example, a global node type and a specified node type. Based on these two types, S100 includes two sub-steps corresponding to these two types:

[0127] When the condition type of the merge trigger condition is a global node type, S100 includes S111: determining all shard service nodes as target shard service nodes. In other words, the RSs matching the condition type of the merge trigger condition are all RSs in the column database cluster.

[0128] When the condition type of the merge trigger condition is a specified node type, S100 includes S121: determining a service node belonging to the specified node as a target shard service node. In other words, the RS that matches the condition type of the merge trigger condition is an RS belonging to the specified node in the node cluster.

[0129] The designated node includes at least one of the following:

[0130] (1) Shard service nodes in the whitelist.

[0131] (2) Shard service nodes associated with the node cluster expansion operation.

[0132] (3) Shard service nodes associated with the node cluster scaling operation.

[0133] (4) Shard service nodes whose localization rate is lower than the preset threshold.

[0134] By setting the above two condition types, the execution scope of the merge operation can be controlled. When the condition type of the merge trigger condition is a global node type, the merge operation will be performed on the entire columnar database cluster, which is beneficial to improve the data reading and writing efficiency and localization rate of the entire cluster and maximize the release of storage space; when the condition type of the merge trigger condition is a specified node type, the merge operation is only performed on the specified node, which is beneficial to reduce unnecessary system resource usage and reduce the impact on other businesses. In addition, the localization rate and read and write efficiency of the specified node can be quickly improved to meet the business needs of the specified node.

[0135] S200. According to a determination rule corresponding to the condition type of the merge trigger condition, determine the target shard data to be merged corresponding to the target shard service node from the multiple shard data stored in the target shard service node.

[0136] Based on the above S111 and S121, S200 also includes two sub-steps:

[0137] In the case of S111, the determination rule indicates that the target shard data is determined based on the update time of each shard data in the target shard service node, then S200 includes:

[0138] S211: Obtain the update time of each shard data in the target shard service node.

[0139] S212: According to the update time of each shard data and the current time, the shard data satisfying the merge execution condition is obtained from each shard data, and the obtained shard data is determined as the target shard data to be merged.

[0140] Among them, the time interval between the update time and the current time of the shard data that meets the merge execution condition is greater than the preset interval threshold. In other words, the merge execution condition includes: the time interval between the update time and the current time is greater than the preset interval threshold. If the time interval between the most recent update time of a shard data and the current time is greater than the preset interval threshold, it means that the shard data has not been merged for a long enough time, and it is determined as the target shard data; if the time interval between the most recent update time of a shard data and the current time is less than or equal to the preset interval threshold, it means that the shard data has just been merged recently, and there is no need to merge this time, so it is not determined as the target shard data.

[0141] The preset interval threshold may be the same as the preset period. In this way, it is possible to avoid too high a frequency of performing the merging operation on the fragmented data, thereby avoiding a waste of system resources.

[0142] In the case of S121, the determination rule indicates that the target shard data is determined according to the localization rate of each shard service node, then S200 includes:

[0143] S221: Obtain the localization rate of each target shard service node among multiple target shard service nodes.

[0144] S222: Determine the shard data in the target shard service node whose localization rate is lower than a preset threshold as the target shard data to be merged.

[0145] When multiple RSs are determined as designated nodes through preset nodes or by detecting expansion or reduction operations, the shard data in the target RS that needs to perform the merge operation can be further accurately screened out as the target shard data based on the localization rate of each target RS, thereby further avoiding waste of system resources.

[0146] S300: Send a merge processing instruction to a target shard service node, so that the target shard service node performs a merge operation on the target shard data according to the merge processing instruction.

[0147] After determining the target shard data that needs to be merged, a merge processing instruction is sent to the target RS, so that the target shard service node performs the merge operation on the target shard data.

[0148] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0149] In addition, the present disclosure also provides a data merging processing device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any data merging processing method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0150] Figure 5 A block diagram of a data merging processing device provided in an embodiment of the present disclosure.

[0151] Reference Figure 5 The present disclosure provides a data merging processing device, the data merging processing device comprising:

[0152] The confirmation unit 101 is used to determine, when a merge trigger condition is met, a shard service node matching the condition type as a target shard service node according to the condition type of the merge trigger condition;

[0153] The confirmation unit 101 is further configured to determine the target shard data to be merged corresponding to the target shard service node according to the condition type of the merge trigger condition;

[0154] The sending unit 102 is used to send a merge processing instruction to the target shard service node, so that the target shard service node performs a merge operation on the target shard data according to the merge processing instruction.

[0155] In some examples, the condition type of the combined trigger condition includes: a global node type and a specified node type; the confirmation unit 101 is specifically used to:

[0156] In the case where the condition type of the merge trigger condition is a global node type, according to the condition type of the merge trigger condition, determining a shard service node matching the condition type among the N shard service nodes as a target shard service node includes: determining all the N shard service nodes as the target shard service nodes;

[0157] When the condition type of the merge trigger condition is a specified node type, based on the condition type of the merge trigger condition, determining a shard service node among N shard service nodes that matches the condition type as a target shard service node includes: determining a shard service node belonging to the specified node as a target shard service node.

[0158] In some examples, the condition type of the merged trigger condition is a global node type; the merged trigger condition includes: detecting that a preset period has arrived; and / or detecting that the current time belongs to a preset time period; wherein the preset time period includes a low-peak business period.

[0159] In some examples, the determination rule indicates that the target shard data is determined based on the update time of each shard data in the target shard service node; the confirmation unit 101 is specifically used to:

[0160] Get the update time of each shard data in the target shard service node;

[0161] According to the update time of each shard data and the current time, the shard data that meets the merge execution condition is obtained from each shard data, and the obtained shard data is determined as the target shard data to be merged;

[0162] Among them, the time interval between the update time of the shard data that meets the merge execution condition and the current time is greater than the preset interval threshold.

[0163] In some examples, the condition type of the merge trigger condition is a specified node type; the merge trigger condition includes: detecting that at least one shard service node among the N shard service nodes belongs to the specified node; wherein,

[0164] The specified node includes at least one of the following:

[0165] Shard service nodes in the whitelist;

[0166] Shard service nodes associated with the node cluster expansion operation;

[0167] The shard service node associated with the node cluster scaling operation;

[0168] A shard service node whose localization rate is lower than a preset threshold; wherein one shard data is associated with multiple column data; the localization rate of a shard service node represents the proportion of column data stored in the same storage structure as the shard service node among the column data associated with the multiple shard data stored by the shard service node.

[0169] In some examples, when the designated node is one of a shard service node in a whitelist, a shard service node associated with a node cluster expansion operation, and a shard service node associated with a node cluster reduction operation, and the number of designated nodes is multiple; if the number of target shard service nodes is multiple, the determination rule indicates that the target shard data is determined according to the localization rate of each shard service node, then the confirmation unit 101 is specifically used to:

[0170] Obtaining a localization rate of each target shard service node among multiple target shard service nodes;

[0171] The shard data in the target shard service node whose localization rate is lower than a preset threshold is determined as the target shard data to be merged.

[0172] In some examples, the merge processing instruction includes a node concurrency parameter, and the node concurrency parameter is used to instruct the target shard service node to create multiple concurrent merge tasks corresponding to the node concurrency parameter; wherein each concurrent merge task is used to perform the merge operation in parallel.

[0173] In some examples, the node concurrency parameter in the confirmation unit 101 is determined by:

[0174] Get the parameter value of the concurrency reference parameter corresponding to the target shard service node;

[0175] According to the obtained parameter value of the concurrency reference parameter, set the node concurrency parameter of the target shard service node;

[0176] The concurrent reference parameters include at least one of the following: system performance parameters, bandwidth parameters, CPU data parameters, and disk input / output parameters.

[0177] In some examples, if there are multiple target shard service nodes, in the confirmation unit 101, the node concurrency parameters of the target shard service nodes are set, including:

[0178] Get the cluster concurrency parameter corresponding to the columnar database cluster, and set the node concurrency parameter of each target shard service node in combination with the cluster concurrency parameter, so that the cumulative result of the node concurrency parameter of each target shard service node in a node cluster is not greater than the cluster concurrency parameter.

[0179] In some examples, the apparatus further includes a sub-data allocation unit, the sub-data allocation unit being configured to:

[0180] The merge divides the target shard data to be merged into multiple sub-data, and allocates corresponding sub-data to each concurrent merge task;

[0181] The data identification information of the sub-data allocated to each concurrent merging task is carried in the merging processing instruction.

[0182] In some examples, the apparatus further includes a task control unit, the task control unit being configured to:

[0183] Get the task processing status of each concurrent merge task in the target shard service node;

[0184] According to the task processing status of each concurrent merge task, a task control instruction is sent to the target shard service node, so that the target shard service node can control the concurrent merge task according to the task control instruction.

[0185] In some examples, when the sub-data allocated to each concurrent merge task is part of the target shard data, the task control unit is specifically used to:

[0186] The sub-data are reallocated for the concurrent merge tasks whose task processing status is the completed status; the task control instruction is used to control the concurrent merge tasks whose task processing status is the completed status to perform merge processing on the reallocated sub-data.

[0187] The data merging processing device of the embodiment of the present disclosure sets multiple types of merging trigger conditions in the confirmation unit 101. For different types of merging trigger conditions, different ranges of shard service nodes are selected as target shard nodes. Then, according to the determination rule matching the condition type of the merging trigger condition, the corresponding shard data in the target shard service node is determined as the target shard data. The sending module 101 sends a merging processing instruction to the target shard service node, so that the target shard service node only needs to perform a merging operation on the target shard data, instead of having to perform a merging operation on each shard data in all shard service nodes, which is beneficial to saving system resources. Moreover, by flexibly setting the merging trigger conditions, the execution timing and execution method of the merging operation can be more in line with business needs and can adapt to various application scenarios.

[0188] Figure 6 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0189] Reference Figure 6 The embodiment of the present disclosure provides an electronic device, which includes: at least one processor 501; at least one memory 502, and one or more I / O interfaces 503, which are connected between the processor 501 and the memory 502; wherein the memory 502 stores one or more computer programs that can be executed by the at least one processor 501, and the one or more computer programs are executed by the at least one processor 501. The above-mentioned data merging processing method is applied to a column-based database cluster, and the column-based database cluster includes N sharding service nodes and management nodes, and each sharding service node stores multiple sharding data; N is an integer greater than 1; the method is executed by the management node, and the method includes:

[0190] When the merge trigger condition is met, according to the condition type of the merge trigger condition, a shard service node matching the condition type among the N shard service nodes is determined as a target shard service node;

[0191] According to the determination rule corresponding to the condition type of the merge trigger condition, determine the target shard data to be merged corresponding to the target shard service node from the multiple shard data stored in the target shard service node;

[0192] A merge processing instruction is sent to the target shard service node, so that the target shard service node performs a merge operation on the target shard data according to the merge processing instruction.

[0193] In some examples, the condition types of the merged trigger conditions include: a global node type and a specified node type;

[0194] In the case where the condition type of the merge trigger condition is a global node type, according to the condition type of the merge trigger condition, determining a shard service node matching the condition type among the N shard service nodes as a target shard service node includes: determining all the N shard service nodes as the target shard service nodes;

[0195] When the condition type of the merge trigger condition is a specified node type, based on the condition type of the merge trigger condition, determining a shard service node among N shard service nodes that matches the condition type as a target shard service node includes: determining a shard service node belonging to the specified node as a target shard service node.

[0196] In some examples, the condition type of the merged trigger condition is a global node type; the merged trigger condition includes: detecting that a preset period has arrived; and / or detecting that the current time belongs to a preset time period; wherein the preset time period includes a low-peak business period.

[0197] In some examples, the determination rule indicates determining the target shard data based on the update time of each shard data in the target shard service node; and determining the target shard data to be merged corresponding to the target shard service node from the multiple shard data stored in the target shard service node according to the determination rule corresponding to the condition type of the merge trigger condition, including:

[0198] Get the update time of each shard data in the target shard service node;

[0199] According to the update time of each shard data and the current time, the shard data that meets the merge execution condition is obtained from each shard data, and the obtained shard data is determined as the target shard data to be merged;

[0200] Among them, the time interval between the update time of the shard data that meets the merge execution condition and the current time is greater than the preset interval threshold.

[0201] In some examples, the condition type of the merge trigger condition is a specified node type; the merge trigger condition includes: detecting that at least one shard service node among the N shard service nodes belongs to the specified node; wherein,

[0202] The specified node includes at least one of the following:

[0203] Shard service nodes in the whitelist;

[0204] Shard service nodes associated with the node cluster expansion operation;

[0205] The shard service node associated with the node cluster scaling operation;

[0206] A shard service node whose localization rate is lower than a preset threshold; wherein one shard data is associated with multiple column data; the localization rate of a shard service node represents the proportion of column data stored in the same storage structure as the shard service node among the column data associated with the multiple shard data stored by the shard service node.

[0207] In some examples, when the designated node is one of a shard service node in a whitelist, a shard service node associated with a node cluster expansion operation, and a shard service node associated with a node cluster reduction operation, and the number of designated nodes is multiple; if the number of target shard service nodes is multiple, and the determination rule indicates that the target shard data is determined according to the localization rate of each shard service node, then according to the determination rule corresponding to the condition type of the merge trigger condition, the target shard data to be merged corresponding to the target shard service node is determined from the multiple shard data stored by the target shard service node, including:

[0208] Obtaining a localization rate of each target shard service node among multiple target shard service nodes;

[0209] The shard data in the target shard service node whose localization rate is lower than a preset threshold is determined as the target shard data to be merged.

[0210] In some examples, the merge processing instruction includes a node concurrency parameter, and the node concurrency parameter is used to instruct the target shard service node to create multiple concurrent merge tasks corresponding to the node concurrency parameter; wherein each concurrent merge task is used to perform the merge operation in parallel.

[0211] In some examples, the node concurrency parameter is determined by:

[0212] Get the parameter value of the concurrency reference parameter corresponding to the target shard service node;

[0213] According to the obtained parameter value of the concurrency reference parameter, set the node concurrency parameter of the target shard service node;

[0214] The concurrent reference parameters include at least one of the following: system performance parameters, bandwidth parameters, CPU data parameters, and disk input / output parameters.

[0215] In some examples, if there are multiple target shard service nodes, setting the node concurrency parameters of the target shard service nodes includes:

[0216] Get the cluster concurrency parameter corresponding to the columnar database cluster, and set the node concurrency parameter of each target shard service node in combination with the cluster concurrency parameter, so that the cumulative result of the node concurrency parameter of each target shard service node in a node cluster is not greater than the cluster concurrency parameter.

[0217] In some examples, before sending the merge processing instruction to the target shard service node, the method further includes:

[0218] Divide the target shard data to be merged into multiple sub-data, and assign corresponding sub-data to each concurrent merge task;

[0219] The data identification information of the sub-data allocated to each concurrent merging task is carried in the merging processing instruction.

[0220] In some examples, after sending the merge processing instruction to the target shard service node, the method further includes:

[0221] Get the task processing status of each concurrent merge task in the target shard service node;

[0222] According to the task processing status of each concurrent merge task, a task control instruction is sent to the target shard service node, so that the target shard service node can control the concurrent merge task according to the task control instruction.

[0223] In some examples, when the sub-data allocated to each concurrent merge task is part of the target shard data, sending the task control instruction to the target shard service node according to the task processing status of each concurrent merge task includes:

[0224] The sub-data are reallocated for the concurrent merge tasks whose task processing status is the completed status; the task control instruction is used to control the concurrent merge tasks whose task processing status is the completed status to perform merge processing on the reallocated sub-data.

[0225] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above-mentioned data merging processing method when executed by a processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0226] The embodiment of the present disclosure also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned data merging processing method.

[0227] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0228] As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0229] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0230] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0231] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0232] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0233] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0234] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0235] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0236] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A data merging processing method, characterized in that: Applied to a column-based database cluster, the column-based database cluster includes N shard service nodes and management nodes, each shard service node stores multiple shard data; N is an integer greater than 1; The method is performed by the management node, and the method includes: In the case where a merge trigger condition is met, according to a condition type of the merge trigger condition, determining a shard service node among the N shard service nodes that matches the condition type as a target shard service node; the condition type includes a specified node type; the target shard service node includes a shard service node belonging to a specified node, and a localization rate of the target shard service node is lower than a preset threshold; Determine, according to a determination rule corresponding to the condition type of the merge trigger condition, the target shard data to be merged corresponding to the target shard service node from the multiple shard data stored by the target shard service node; A merge processing instruction is sent to the target shard service node, so that the target shard service node performs a merge operation on the target shard data according to the merge processing instruction.

2. The data merging processing method according to claim 1, characterized in that: The condition type of the merge trigger condition also includes: global node type; Wherein, in the case where the condition type of the merge trigger condition is a global node type, determining, according to the condition type of the merge trigger condition, a shard service node among the N shard service nodes that matches the condition type as a target shard service node includes: determining all of the N shard service nodes as the target shard service nodes; In the case where the condition type of the merge trigger condition is a specified node type, determining the shard service node that matches the condition type among the N shard service nodes as the target shard service node based on the condition type of the merge trigger condition includes: determining the shard service node belonging to the specified node as the target shard service node.

3. The data merging processing method according to claim 2, characterized in that: The condition type of the merge trigger condition is a global node type; The merging triggering condition includes: detecting that a preset period has arrived; and / or detecting that the current time belongs to a preset time period; wherein the preset time period includes a low-peak service period.

4. The data merging method according to any one of claims 1 to 3, characterized in that: The determination rule corresponding to the condition type of the merge trigger condition indicates that the target shard data is determined based on the update time of each shard data in the target shard service node; the determination rule corresponding to the condition type of the merge trigger condition determines the target shard data to be merged corresponding to the target shard service node from the multiple shard data stored in the target shard service node according to the determination rule, including: Obtain the update time of each shard data in the target shard service node; According to the update time of each shard data and the current time, the shard data satisfying the merge execution condition is obtained from each shard data, and the obtained shard data is determined as the target shard data to be merged; Among them, the time interval between the update time of the fragment data that meets the merge execution condition and the current time is greater than a preset interval threshold.

5. The data merging method according to claim 2, characterized in that: The condition type of the merge trigger condition is a designated node type; the merge trigger condition includes: detecting that at least one of the N shard service nodes belongs to a designated node; wherein, The designated node includes at least one of the following: Shard service nodes in the whitelist; Shard service nodes associated with the node cluster expansion operation; The shard service node associated with the node cluster scaling operation; Shard service nodes whose localization rate is lower than the preset threshold; Among them, one shard data is associated with multiple column data; the localization rate of any shard service node represents the proportion of column data stored in the same storage structure as any shard service node among the column data associated with the multiple shard data stored by any shard service node.

6. The data merging method according to claim 5, characterized in that: When the designated node is one of a shard service node in a whitelist, a shard service node associated with a node cluster expansion operation, and a shard service node associated with a node cluster reduction operation, and the number of the designated nodes is multiple; If there are multiple target shard service nodes, and the determination rule indicates that the target shard data is determined according to the localization rate of each shard service node, then the determination rule corresponding to the condition type of the merge trigger condition determines the target shard data to be merged corresponding to the target shard service node from the multiple shard data stored by the target shard service node, including: Obtaining a localization rate of each target shard service node among multiple target shard service nodes; The shard data in the target shard service node whose localization rate is lower than a preset threshold is determined as the target shard data to be merged.

7. The data merging method according to claim 1, characterized in that: The merge processing instruction includes a node concurrency parameter, and the node concurrency parameter is used to instruct the target shard service node to create multiple concurrent merge tasks corresponding to the node concurrency parameter; wherein each concurrent merge task is used to perform the merge operation in parallel.

8. The data merging method according to claim 7, characterized in that: The node concurrency parameter is determined in the following way: Obtaining a parameter value of a concurrent reference parameter corresponding to the target sharding service node; According to the obtained parameter value of the concurrency reference parameter, the node concurrency parameter of the target shard service node is set; The concurrent reference parameters include at least one of the following: system performance parameters, bandwidth parameters, central processing unit data parameters, and disk input / output parameters.

9. The data merging method according to claim 8, characterized in that: If the number of the target shard service nodes is multiple, the step of setting the node concurrency parameters of the target shard service nodes includes: Obtain a cluster concurrency parameter corresponding to the columnar database cluster, and set the node concurrency parameter of each target shard service node in combination with the cluster concurrency parameter, so that the cumulative result of the node concurrency parameters of each target shard service node in a node cluster is not greater than the cluster concurrency parameter.

10. The data merging method according to claim 9, characterized in that: Before sending the merge processing instruction to the target shard service node, the method further includes: Divide the target shard data to be merged into multiple sub-data, and assign corresponding sub-data to each concurrent merging task; The data identification information of the sub-data allocated to each concurrent merging task is carried in the merging processing instruction.

11. The data merging method according to claim 10, characterized in that: After sending the merge processing instruction to the target shard service node, the method further includes: Obtaining the task processing status of each concurrent merge task in the target shard service node; According to the task processing status of each concurrent merged task, a task control instruction is sent to the target shard service node, so that the target shard service node controls the concurrent merged task according to the task control instruction.

12. The data merging processing method according to claim 11, characterized in that: In a case where the sub-data allocated to each concurrent merge task is part of the target shard data, sending a task control instruction to the target shard service node according to the task processing status of each concurrent merge task includes: The sub-data are reallocated for the concurrent merge tasks whose task processing status is the completed status; the task control instruction is used to control the concurrent merge tasks whose task processing status is the completed status to perform merge processing on the reallocated sub-data.

13. A data merging processing device, characterized in that: A management node applied to a column-based database cluster, wherein the column-based database cluster includes N shard service nodes and management nodes, and each shard service node stores multiple shard data; N is an integer greater than 1; The data merging and processing device comprises: A confirmation unit, configured to, when a merge trigger condition is met, determine, according to a condition type of the merge trigger condition, a shard service node among the N shard service nodes that matches the condition type as a target shard service node; the condition type includes a specified node type; the target shard service node includes a shard service node belonging to a specified node, and a localization rate of the target shard service node is lower than a preset threshold; The confirmation unit is further configured to determine, from the plurality of shard data stored by the target shard service node, the target shard data to be merged corresponding to the target shard service node according to a determination rule corresponding to the condition type of the merge trigger condition; The sending unit is used to send a merge processing instruction to the target shard service node, so that the target shard service node performs a merge operation on the target shard data according to the merge processing instruction.

14. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 12.

Images