Data partition merging method, electronic device, storage medium, and computer program product

Abstract

This application discloses a data partitioning and merging method, electronic device, storage medium, and computer program product. The method includes: determining a load factor based on resource indicators corresponding to the node where the database system resides in a target task scheduling round, wherein the load factor characterizes the load level of the node in the target task scheduling round; adjusting the task admission threshold corresponding to the target task scheduling round based on the load factor; obtaining a task evaluation result for each data merging task in the target task scheduling round, and in response to at least one data merging task having a task evaluation result greater than or equal to the task admission threshold, adding the at least one data merging task to a task queue corresponding to the target task scheduling round; and executing each data merging task included in the task queue.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data partitioning and merging method, electronic device, storage medium, and computer program product. Background Technology

[0002] With the rapid development of big data technology, columnar databases have been widely used in data analysis scenarios due to their high data compression ratio and query performance. In the MergeTree engine of columnar databases such as ClickHouse, data is stored in partitions, relying on background merging to aggregate small partitions into larger ones, thereby reducing the number of files, improving query efficiency, and promptly reclaiming space occupied by expired data. However, if the merging strategy does not consider system performance during the data partition merging process, it will affect the overall system stability, and consequently, the efficiency of the database system in merging, storage, and querying. Therefore, how to optimize data merging to improve the overall performance of the database system is a technical problem that urgently needs to be solved. Summary of the Invention

[0003] This application provides a data partitioning and merging method, electronic device, storage medium, and computer program product, which can solve the problem of how to optimize data merging to improve the overall performance of a database system.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, a data partitioning and merging method is provided, the method comprising: determining a load factor based on resource indicators corresponding to the node where the database system resides in a target task scheduling round, wherein the load factor is used to characterize the load level of the node in the target task scheduling round; adjusting the task admission threshold corresponding to the target task scheduling round according to the load factor; obtaining the task evaluation result corresponding to each data merging task in the target task scheduling round, and in response to at least one data merging task having a task evaluation result greater than or equal to the task admission threshold, adding the at least one data merging task to the task queue corresponding to the target task scheduling round; and executing each data merging task included in the task queue.

[0005] In a second aspect, an electronic device is provided, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0006] Thirdly, a computer-readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0007] Fourthly, a computer program product is provided, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the steps of the method described in the first aspect.

[0008] In this embodiment, a load factor is determined based on the resource indicators of the node where the database system resides in the target task scheduling round. The load factor characterizes the load level of the node in the target task scheduling round. The task admission threshold corresponding to the target task scheduling round is adjusted according to the load factor. The task evaluation result of each data merging task in the target task scheduling round is obtained. In response to the task evaluation result of at least one data merging task being greater than the task admission threshold, at least one data merging task is added to the task queue corresponding to the target task scheduling round. Each data merging task included in the task queue is executed. This optimizes the data merging strategy. That is, by evaluating the node load status, data merging tasks are scheduled differently. This effectively improves the overall performance of the database system based on the dynamic balance and optimization of node resource consumption and data partition merging efficiency.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0011] Figure 1 A flowchart illustrating a data partition merging method provided in an embodiment of this application is shown; Figure 2 A flowchart illustrating another data partition merging method provided in an embodiment of this application is shown; Figure 3 This paper illustrates a schematic diagram of the architecture of a data partitioning and merging system provided in an embodiment of this application. Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0012] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0013] Figure 1 The illustration shows one of the flowcharts of a data partition merging method provided in this application embodiment. This data partition merging method can be executed by an electronic device, which can be a terminal or a network-side device. The method may include the following steps: S110: Determine the load factor based on the resource indicators corresponding to the node where the database system is located in the target task scheduling round.

[0014] The load factor is used to characterize the load level of a node in the target task scheduling round.

[0015] In some embodiments, the node where the database system resides refers to a physical machine, virtual machine, or container that performs data processing (merging, querying, storing, updating, etc.).

[0016] In some embodiments, the target task scheduling round can be the current task scheduling round or a future task scheduling round. When the target task scheduling round is the current task scheduling round, the load factor is used to characterize the load level of the node in the current task scheduling round. When the target task scheduling round is a future task scheduling round, the load factor is used to characterize the predicted load level of the node in the future task scheduling round.

[0017] In some embodiments, the resource metrics may include, but are not limited to, CPU utilization, memory usage, disk I / O, network bandwidth, and current task queue length.

[0018] In this step, at the start of each round of task scheduling or when planning each round of task scheduling, the resource indicators corresponding to the node in that round are first obtained. Then, the busy / idle level, i.e., the load level, of the node is assessed based on the resource indicators. It is understandable that when performing data partition merging, the merging operation is essentially a data redistribution process, which may trigger a large amount of data migration, index rebuilding, and resource consumption (such as a large amount of disk read / write, memory sorting, and CPU computational overhead). If the performance of the nodes hosting the database system is not considered before merging, then in scenarios where the overall node resources are limited, the data merging task will intensify the competition for CPU, memory, and I / O resources, thereby crowding out computing resources for data storage and querying, leading to system response delays. In scenarios where the overall node resources are sufficient, the data merging task may be too conservative, resulting in a large amount of available computing resources being idle, thus causing low data merging efficiency and delayed storage space reclamation. Therefore, before data merging, it is necessary to obtain the node's load factor and, based on the actual load level of the node, reasonably set the data merging task to achieve efficient data merging while ensuring system stability.

[0019] S120: Adjust the task admission threshold corresponding to the target task scheduling round based on the load factor.

[0020] The task admission threshold is used to filter data merging tasks that can be scheduled in the target task scheduling round.

[0021] In this step, the lower limit of the task threshold for participation in scheduling is adjusted based on the load factor corresponding to the node, thereby dynamically configuring the task admission threshold for the target task scheduling round. It is understandable that different data merging tasks differ in data size, computational complexity, and I / O overhead. When a node load is high, the task admission threshold can be increased to prioritize lightweight or high-priority data merging tasks, thus preventing resource exhaustion caused by heavyweight tasks. Conversely, when a node load is low, the task admission threshold can be decreased to execute high-consumption, high-throughput data merging tasks, thus avoiding resource waste. Therefore, adjusting the task admission threshold for the target task scheduling round based on the load factor enables differentiated scheduling of data merging tasks.

[0022] S130: Obtain the task evaluation result of each data merging task in the target task scheduling round, and in response to the task evaluation result of at least one data merging task being greater than or equal to the task admission threshold, add at least one data merging task to the task queue corresponding to the target task scheduling round.

[0023] This data merging task comprises multiple groups, each containing two data partitions. It should be noted that these data partitions can be the smallest available data partitions or the result of merging multiple smallest data partitions. The task evaluation result measures the contribution of the data merging task to the overall system performance improvement; a higher evaluation result indicates a greater contribution, and a lower result indicates a smaller contribution. This task queue stores the data merging tasks that need to be scheduled for execution in the target task scheduling rounds.

[0024] In this step, for each data merging task, the task evaluation result corresponding to the target task scheduling round is obtained. If the task evaluation result is greater than or equal to the task admission threshold, the data merging task is added to the task queue corresponding to the target task scheduling round. In this way, by filtering the schedulable data merging tasks, a balance between maximizing merging efficiency and making full use of resources can be achieved while ensuring the stable operation of the system.

[0025] S140: Execute each data merging task included in the task queue.

[0026] In this step, each data merging task stored in the task queue is retrieved sequentially and executed in turn. Each data merging task includes a data partition to be merged. In some embodiments, the data merging task may further include merging rules, and each data merging task merges the data partitions to be merged according to the corresponding merging rules. In some embodiments, the merging rules may include deduplication strategies, sorting methods, and merge conflict resolution logic, etc.

[0027] In this embodiment, a load factor is determined based on the resource indicators of the node where the database system resides in the target task scheduling round. The load factor characterizes the load level of the node in the target task scheduling round. The task admission threshold corresponding to the target task scheduling round is adjusted according to the load factor. The task evaluation result of each data merging task in the target task scheduling round is obtained. In response to the task evaluation result of at least one data merging task being less than the task admission threshold, at least one data merging task is added to the task queue corresponding to the target task scheduling round. Each data merging task included in the task queue is executed. This optimizes the data merging strategy. That is, by evaluating the node load status, data merging tasks are scheduled differently. This effectively improves the overall performance of the database system based on the dynamic balance and optimization of node resource consumption and data partition merging efficiency.

[0028] In some embodiments, resource metrics include central processing unit (CPU) utilization R. cpu Disk I / O utilization Rio Memory utilization rate R mem At least one of the following; determine the load factor based on the resource indicators corresponding to the node where the database system is located in the target task scheduling round, including: for each utilization rate included in the resource indicators, calculate the available capacity corresponding to the utilization rate; determine the load factor based on the target available capacity and the average available capacity, wherein the target available capacity is the minimum available capacity among the available capacities corresponding to each item, and the average available capacity is the average of the sum of the available capacities corresponding to each item.

[0029] Among these, CPU utilization refers to the proportion of time the CPU is in a non-idle state within a certain period; disk I / O utilization refers to the proportion of time the disk is in an active service state within a certain period; and memory utilization refers to the proportion of memory actually used to the total physical memory. In some embodiments, these resource metrics may also include network bandwidth, current task queue length, etc.

[0030] In this embodiment, the available capacity corresponding to each resource indicator is first determined, then the minimum available capacity is selected from all available capacities as the target available capacity, the average available capacity is obtained by adding all available capacities together, and then the load factor is determined based on the target available capacity and the average available capacity.

[0031] For example, suppose resource metrics include CPU utilization R cpu Disk I / O utilization R io Memory utilization rate R mem So, CPU utilization R cpu The corresponding available capacity can be M cpu = max(0, 1 - R cpu Disk I / O utilization R ( / 100) io The corresponding available capacity can be M io = max(0, 1 - R io / 100), memory utilization R mem The corresponding available capacity can be M mem = max(0, 1 - R mem / 100). It's understandable that if the CPU utilization is R... cpu Therefore, the available CPU capacity is: M cpu =max(0, 1 - R cpu / 100): When R cpu Increase, 1 - R cpu / 100 decreases accordingly, M cpu Decrease; when R cpu When M is 0, cpuA value of 1 indicates that the CPU is completely idle; when R... cpu When M is ≥100 cpu A value of 0 indicates that the CPU has no available capacity. If the disk I / O utilization is R... io Therefore, the available disk I / O capacity is: M io = max(0, 1 - R io / 100): When R io Increase, 1 - R io / 100 decreases accordingly, M io Decrease; when R io When M is 0, io A value of 1 indicates that disk I / O is completely idle; when R... io When M is ≥100 io A value of 0 indicates that there is no available disk I / O capacity. If the memory utilization rate is R... mem Therefore, the available memory capacity is: M mem = max(0, 1 - R mem / 100): When R mem Increase, 1 -R mem / 100 decreases accordingly, M mem Decrease; when R mem When M is 0, mem A value of 1 indicates that the disk MEM is completely idle; when R mem When M is ≥100 mem A value of 0 indicates that there is no available memory. If the CPU utilization R... cpu >Disk I / O utilization R io >Memory utilization R mem Therefore, available CPU capacity < available disk I / O capacity < available memory capacity, thus min(M cpu M io M mem The result is the available CPU capacity M. cpu In other words, the higher the utilization rate, the smaller the available capacity.

[0032] In some embodiments, determining the load factor based on the target available capacity and the average available capacity may include: performing a weighted summation of the target available capacity and the average available capacity to obtain the load factor. For example, assume that resource metrics include CPU utilization R. cpu Disk I / O utilization R io Memory utilization rate R mem Therefore, the load factor can be determined using the following formula: M load =α×min(M cpu M io M mem )+(1 α)×mean(M cpu M io M mem ) Among them, M load α is the load factor; α is the extreme value adjustment coefficient (0≤α≤1), used to balance the effects of the weakest link effect and the average load; M cpu M io M mem These represent the available capacity of CPU, I / O, and memory, respectively; the min() function is used to capture bottleneck resources; and the mean() function is used to reflect the overall average load level.

[0033] In this embodiment, the target available capacity is used to identify the bottleneck resources of a node. For example, if the available capacity corresponding to disk I / O utilization is higher than the other two values, then the target available capacity can be the available capacity corresponding to disk I / O utilization. In this case, it can be determined that the node is in an "I / O bottleneck" state. In other words, the target available capacity indicates the actual limit of the current node's available computing power. In the scheduling strategy, using minimum resources as a constraint can effectively prevent system avalanche or task failure caused by local resource overload.

[0034] In some embodiments, adjusting the task admission threshold corresponding to the target task scheduling round based on the load factor includes: (1) In response to the node’s data partition merging strategy being in normal mode and the load factor being greater than the action trigger threshold of the hysteresis state machine, the data partition merging strategy is switched from normal mode to suppression mode, and the task admission threshold is increased. The task admission threshold corresponding to normal mode is the first initial value. Normal mode is used to indicate that the parameter value of the data partition merging strategy is kept at the second initial value, and suppression mode is used to indicate that the parameter value of the data partition merging strategy is lower than the second initial value. The data partition merging strategy operates in two modes: Normal mode and Suppression mode. Normal mode represents the standard state of the data partition merging strategy, where the strategy parameters are set to the second initial value, resulting in relatively high resource consumption. Suppression mode represents a degraded state where the strategy parameters are set below the second initial value to reduce node load. The second initial value is the parameter setting for the data partition merging strategy in Normal mode. For example, this parameter may include, but is not limited to, the number of concurrent merge operations, the minimum number of merged partitions, the maximum number of merged partitions, the I / O rate, and the merge cooldown time. A hysteresis state machine is a state transition mechanism: switching from Normal mode to Suppression mode requires the load factor to be greater than the action trigger threshold, and switching back from Suppression mode to Normal mode requires the load factor to be less than the recovery trigger threshold, thus avoiding frequent fluctuations between the two states. The first initial value is the default value for Normal mode. In Suppression mode, this first initial value needs to be increased.

[0035] In this embodiment, a hysteresis state machine mechanism is introduced: when the node's data partition merging strategy is in normal mode, if the load factor is greater than the action trigger threshold of the hysteresis state machine, it indicates that the current node is under heavy load and needs to switch from normal mode to suppression mode, while increasing the task admission threshold. In suppression mode, the data merging intensity needs to be weakened; therefore, the parameter value of the data partition merging strategy is lower than the second initial value. Simultaneously, the task admission threshold is increased to raise the task admission threshold, thereby ensuring node stability.

[0036] (2) In response to the data partition merging strategy being in suppression mode and the load factor being less than the recovery trigger threshold of the hysteresis state machine, the data partition merging strategy is restored from suppression mode to normal mode, and the task admission threshold is restored to the first initial value; If the load factor is less than the recovery trigger threshold of the hysteresis state machine when the data partition merging strategy of the node is in suppression mode, it means that the current node load is small and needs to be switched from suppression mode to normal mode. The task admission threshold should be restored to the first initial value to reduce the task admission threshold and thus improve the node's resource utilization.

[0037] (3) In response to the load factor being between the recovery trigger threshold and the action trigger threshold, the node is instructed to maintain the current mode, wherein the current mode is the normal mode or the suppression mode.

[0038] Understandably, if the load factor is between the recovery trigger threshold and the action trigger threshold, the current mode will remain unchanged to filter out instantaneous load spikes and prevent strategy oscillations.

[0039] In some embodiments, obtaining the task evaluation result corresponding to each data merging task in the target task scheduling round includes: for each data merging task, determining the task evaluation result corresponding to the data merging task based on the merging evaluation result of each group of partitions to be merged included in the data merging task, wherein each group of partitions to be merged includes two data partitions to be merged, and the merging evaluation result is determined based on the load factor and the target evaluation value corresponding to the two data partitions to be merged, wherein the target evaluation value includes at least one of the following: data volume difference evaluation value, index range overlap evaluation value, data heat evaluation value, and data timeliness evaluation value.

[0040] Among them, the data volume difference assessment value is used to measure the difference in data volume between the two data partitions to be merged, the index range overlap assessment value is used to measure the degree of overlap of the index ranges of the two data partitions to be merged, the data popularity assessment value is used to measure the access frequency of the two data partitions to be merged, and the data timeliness assessment value is used to measure the timeliness of the data in the two data partitions to be merged.

[0041] In some embodiments, the merge evaluation result for each group of partitions to be merged is the product of the weighted sum of the target evaluation values ​​and the load factor. For example, assuming the target evaluation values ​​include data volume difference evaluation values, index range overlap evaluation values, data popularity evaluation values, and data timeliness evaluation values, then the merge evaluation result can be determined by the following formula: P group =(W1×S size +W2×S overlap +W3×S temp +W4×S age )×M load Among them, P group This represents the merge evaluation results for a set of partitions to be merged; S size S is the evaluation value for the difference in data volume. overlap S is the index range overlap evaluation value; temp This is the data popularity assessment value; S age W1 represents the data timeliness assessment value; W1, W2, W3, and W4 are the weight coefficients for the corresponding dimensions, and satisfy W1 + W2 + W3 + W4 = 1; M load This is the load adjustment factor.

[0042] Assuming the two data partitions to be merged are A and B, then: S size = (1 - (Size_A + Size_B) / (2 × Max_Size)), where Max_Size is the upper limit threshold for the data partition size; S overlap =(Overlap percentage of primary indexes of A and B / 100), where the overlap percentage of primary indexes of A and B can be determined based on the overlap granule number. When calculating the overlap granule number, the merge sort process of the two data partitions can be simulated in memory, calculating the intersection of the two index fingerprint sequences in the key space. For example, the quantification formula for the overlap granule number can be: Overlapscore = ∑(GranuleA∩GranuleB), where GranuleA is the granule number of data partition A, and GranuleB is the granule number of data partition B; S temp Time_A represents the hourly popularity of data partition A, and Time_B represents the hourly popularity of data partition B; S age=((1 - log(Age_A+1) / log(100)) + (1 - log(Age_B+1) / log(Max_age)) ) / 2, where Age_A represents the duration of data partition A, Age_B represents the duration of data partition B, and Max_age is the maximum duration of existence (e.g., 100).

[0043] In this embodiment, for each data merging task, after obtaining the merging evaluation results of each group of partitions to be merged, the task evaluation result corresponding to the data merging task can be determined based on the merging evaluation results of each group of partitions to be merged. In some embodiments, the task evaluation result is obtained by quantifying the merging evaluation results, such as by weighted averaging of all merging evaluation results and using the average as the task evaluation result, or by using the maximum evaluation result among all merging evaluation results as the task evaluation result corresponding to the data merging task. In some embodiments, if a data merging task includes only one group of partitions to be merged, then the merging evaluation results of that group can be used as the task evaluation result corresponding to the data merging task to which it belongs.

[0044] In some embodiments, each data merging task included in the task queue includes: in response to the completion of the target data merging task, determining the target task benefit corresponding to the target data merging task based on the change in the state of the node before and after the execution of the target data merging task, wherein the change in state includes at least one of data query performance degradation, storage space change, and reduction in the number of data partitions; determining the target benefit trend corresponding to the target data merging task based on the target task benefit and historical benefit trend, wherein the historical benefit trend is the task benefit trend corresponding to the previous data merging task; in response to the target benefit trend being less than the historical benefit trend, adjusting the weight parameters of the target feature dimension, wherein the target feature dimension includes at least one feature dimension used to determine the task evaluation result.

[0045] The data query performance difference indicates the objective improvement in query performance after the merge, which can be indicated by metrics such as query response time, data scan time, and index scan time. The storage space change indicates the actual difference in disk space usage before and after the merge, and the reduction in the number of data partitions indicates the actual reduction in the number of files in the data partitions before and after the merge. In some embodiments, the state change may also include, but is not limited to, changes in I / O throughput, differences in data page / block continuity, and differences in buffer hit rate. The target task benefit indicates the magnitude of the optimization gain of the overall system performance brought about by this merge operation. The target benefit trend characterizes the continuous change in the system performance optimization gain brought about by continuous merge tasks.

[0046] In some embodiments, obtaining the state change of a node before and after the execution of the target data merging task may include: obtaining first state information of the node before the execution of the target data merging task and second state information after the execution of the target data merging task; and determining the state change based on the first state information and the second state information.

[0047] In some embodiments, determining the target task benefit corresponding to the target data merging task based on the state change of a node before and after the execution of the target data merging task may include: weighted summing of the items included in the state change to obtain the target task benefit. For example, the target task benefit can be expressed by the following formula: R actual (T i )=w q ×ΔQ actual +w s ×ΔS actual +w n ×ΔN actual Where, ΔQ actual Indicates poor data query performance, ΔS actual ΔN represents the change in storage space. actual w represents the amount by which the number of data partitions is reduced. q w s w n These are the normalization coefficients for each item.

[0048] In some embodiments, determining the target revenue trend corresponding to the target data merging task based on the target task revenue and historical revenue trend includes: processing the target task revenue and historical revenue trend based on a decay factor to obtain the target revenue trend.

[0049] The attenuation factor is used to smooth current and historical returns to identify long-term trends in returns. In some embodiments, processing the target task return and historical return trends based on the attenuation factor to obtain the target return trend may include: weighting the target task return and historical return trends using the attenuation factor to obtain the target return trend.

[0050] For example, the target return trend can be expressed as: E t =λ×R actual (T t )+(1 λ)×E t 1, where E t R represents the trend of the target revenue corresponding to the target data merging task. actual (T tE represents the benefit of the target task. t 1 represents the historical return trend, and λ is the smoothing coefficient.

[0051] In this embodiment of the application, if the target return trend is less than the historical return trend (i.e., E) t <E t 1) This indicates that the optimization effect of the tasks selected under the current weighting strategy is declining, requiring the triggering of a weight update mechanism to reduce the weight of feature dimensions that currently account for a high proportion of the evaluation results of high-scoring tasks. In other words, if a decline in returns occurs, it means that the feature dimensions with high scores can no longer bring high returns, so their weights need to be reduced, allowing other feature dimensions to have a chance to dominate the scheduling.

[0052] In some embodiments, adjusting the weight parameters used to determine the task evaluation result includes: taking the feature dimensions whose evaluation values ​​are higher than a preset threshold in the task evaluation result as target feature dimensions; and for each target feature dimension, determining the updated weight parameters corresponding to the target feature dimension based on the adjustment step size, target return trend, historical return trend, the evaluation value corresponding to the target feature dimension, and the weight parameters before the update corresponding to the target feature dimension.

[0053] The adjustment step size refers to the distance or magnitude by which the weights move along the gradient direction.

[0054] In this embodiment, if the target return trend is less than the historical return trend, it indicates that the optimization effect of the task selected under the current weighting strategy is declining, and a weight update mechanism needs to be triggered to reduce the weight of the feature dimensions that account for a high proportion in the current high task evaluation results. Therefore, the feature dimensions whose evaluation values ​​are higher than a preset threshold in the task evaluation results are taken as target feature dimensions, and the weight parameters of the target feature dimensions are adjusted. In some embodiments, the feature dimensions used to determine the task evaluation results may include at least one of data volume, index range, data popularity, and data timeliness.

[0055] In some embodiments, determining the updated weight parameters for the target feature dimension based on the adjustment step size, target return trend, historical return trend, evaluation value corresponding to the target feature dimension, and the weight parameters before the update for the target feature dimension may include: determining the updated weight for the target feature dimension as the difference between the weight before the update and a first value, wherein the first value is the product of the adjustment step size, a second value, and the evaluation value corresponding to the target feature dimension, and the second value is the absolute value of the difference between the target return trend and the historical return trend. For example, taking the weight W3 corresponding to data popularity as an example, its update rule is as follows: w 3new =w 3old η×∣Et E t 1 | × S temp w 3new For the updated weights, w 3old The weights before the update; η×∣E t E t 1 | × S temp Let η be the first value, η be the adjustment step size, and |E be the value. t E t 1| represents the second value, which is the absolute value of the difference between the target return trend and the historical return trend, S temp The evaluation value corresponding to the data popularity is the data popularity evaluation value.

[0056] In this embodiment of the application, the mechanism of penalty for declining revenue can automatically suppress feature weights that no longer generate high revenue, so that the scheduling strategy can dynamically adapt to changes in business load.

[0057] Figure 2 This document illustrates a flowchart of another data partitioning and merging method provided in an embodiment of this application, as shown below. Figure 2 As shown, the following steps may be included: S201: Determine the load factor based on the resource indicators corresponding to the node where the database system is located in the target task scheduling round.

[0058] S202: Obtain the task evaluation result corresponding to each data merging task in the target task scheduling round.

[0059] S203: Determine whether the node's data partition merging strategy is in normal mode or suppressed mode.

[0060] If it is in suppression mode, proceed to S204; if it is in normal mode, proceed to S207.

[0061] The task admission threshold corresponding to the normal mode is the first initial value. The normal mode is used to indicate that the parameter value of the data partition merging strategy is kept at the second initial value, and the suppression mode is used to indicate that the parameter value of the data partition merging strategy is lower than the second initial value.

[0062] S204: Determine whether the load factor is greater than the action trigger threshold of the hysteresis state machine.

[0063] If yes, proceed to S205; otherwise, proceed to S210.

[0064] S205: Switch the data partition merging strategy from normal mode to suppression mode.

[0065] S206: Increase the task admission threshold.

[0066] S207: Determine whether the load factor is less than the recovery trigger threshold of the hysteresis state machine.

[0067] If yes, proceed to S208; otherwise, proceed to S210.

[0068] S208: Restore the data partition merging strategy from suppressed mode to normal mode.

[0069] S209: Restore the task admission threshold to the first initial value.

[0070] S210: Maintain current mode.

[0071] S211: Iterate through each data merging task.

[0072] S212: Get the data merging task currently being traversed.

[0073] S213: Determine whether the task evaluation result of the data merging task being traversed is greater than or equal to the task admission threshold.

[0074] If yes, proceed to S214; otherwise, proceed to S215.

[0075] S214: Add the data merging task currently being traversed to the task queue corresponding to the target task scheduling round.

[0076] S215: Mission suspended.

[0077] S216: Waiting for the next task scheduling round.

[0078] S217: Determine whether the merging of all data tasks has been completed.

[0079] If not, proceed to S218; if yes, proceed to S219.

[0080] S218: Obtain the data merging task for the next iteration.

[0081] S219: Iterate through each data merging task included in the task queue.

[0082] S220: Obtain the data merging task currently being traversed and use it as the target data merging task.

[0083] S221: Process target data merging tasks through the MergeTree engine.

[0084] S222: Determine the target task benefit corresponding to the target data merging task based on the change in the node's state before and after the target data merging task is executed.

[0085] Among them, the change in status includes at least one of the following: poor data query performance, change in storage space, and reduction in the number of data partitions.

[0086] S223: Determine the target revenue trend corresponding to the target data merging task based on the target task revenue and historical revenue trends.

[0087] Among them, the historical return trend is the return trend corresponding to the previous data merging task.

[0088] S224: Determine whether the target return trend is less than the historical return trend.

[0089] If yes, proceed to S225; otherwise, proceed to S226.

[0090] S225: Adjust the weight parameters of the target feature dimension.

[0091] The target feature dimension includes at least one feature dimension used to determine the task evaluation result.

[0092] S226: Maintain the current weights.

[0093] S227: Determine whether the task queue, including all data merging tasks, has been completed.

[0094] If not, proceed to S228; if yes, end.

[0095] S228: Obtain the data merging task for the next iteration and use it as the target data merging task.

[0096] Figure 3 The diagram illustrates the architecture of a data partitioning and merging system provided in this application, including a management and scheduling layer, an adaptive optimization controller, an adaptive evolutionary loop, and a storage and execution layer.

[0097] In this embodiment, the management and scheduling layer includes: a policy configuration center, which manages global scheduling parameters (such as weight coefficients, hysteresis thresholds, etc.) and supports static configuration and ZooKeeper dynamic hot updates; and a task scheduler, which maintains a priority queue of tasks to be merged, receives tasks approved by the admission module, sorts them by score, and sends them to the storage engine.

[0098] The adaptive optimization controller includes: a metadata acquisition module, used to periodically extract partition metadata and real-time system indicators from the underlying system table and distribute them to subsequent modules; a load sensing module, used to calculate a three-dimensional load vector based on system indicators and output a comprehensive load factor as the basis for dynamic adjustment; a feature scoring module, used to calculate a basic score by combining partition features and weights and introducing the load factor to obtain the final scheduling score; and a dynamic admission module, used to maintain normal / suppression modes based on a hysteresis state machine and decide whether a task is admitted based on the load and threshold.

[0099] The adaptive evolutionary loop includes: a feedback evolution module, which monitors the actual benefits after task execution, smooths the process using the EWMA algorithm, and updates the weights of the scoring module in reverse.

[0100] The storage and execution layer includes: the first system table, the ClickHouse system table, which provides metadata information about data partitions (size, number of layers, status, etc.); the second system table, the ClickHouse system table, which provides real-time performance metrics of the instance (CPU, IO, memory, etc.); and the MergeTree storage engine, which receives merge commands and performs the actual data partition merging, deduplication, and sorting operations.

[0101] In this embodiment, by introducing a load factor, the background merging task can perceive the CPU and I / O load in real time and automatically achieve peak shaving and valley filling. It proactively avoids resource allocation during peak business periods, preventing query timeouts or write blocking caused by background merging and significantly reducing the risk of system avalanche. Simultaneously, it significantly optimizes query performance and resource utilization efficiency: through multi-dimensional weighted scoring (covering overlap, hotness, etc.), it prioritizes merging the data most valuable to the current business query, reducing unnecessary I / O overhead and ensuring fast response times for queries on hot data. Furthermore, by pre-setting different weighting strategy modes, it can adapt to various complex scenarios ranging from "write-intensive" to "read-intensive," thus providing the database system with refined merging capabilities.

[0102] like Figure 4 As shown, this application embodiment also provides an electronic device 400, including a processor 410 and a memory 420. The memory 420 stores a program or instructions that can run on the processor 410. When the program or instructions are executed by the processor 410, they implement the various processes of the above-described data partition merging method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0103] This application also provides a computer-readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described data partitioning and merging method embodiments and achieve the same technical effects. To avoid repetition, these will not be described again here.

[0104] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described data partitioning and merging method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0105] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0106] This application also provides a computer program / program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to implement the various processes of the above-described data partitioning and merging method, and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0107] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other implementations under the guidance of this application without departing from the spirit and scope of the claims. All of these implementations are within the protection scope of this application.

Claims

1. A data partitioning and merging method, characterized in that, The method includes: Based on the resource indicators corresponding to the node where the database system is located in the target task scheduling round, the load factor is determined, wherein the load factor is used to characterize the load level of the node in the target task scheduling round; Adjust the task admission threshold corresponding to the target task scheduling round based on the load factor; Obtain the task evaluation result corresponding to each data merging task in the target task scheduling round, and in response to the task evaluation result corresponding to at least one of the data merging tasks being greater than or equal to the task admission threshold, add the at least one of the data merging tasks to the task queue corresponding to the target task scheduling round. Execute each of the data merging tasks included in the task queue.

2. The method according to claim 1, characterized in that, The resource metrics include at least one of the following: CPU utilization, disk I / O utilization, and memory utilization. The step of determining the load factor based on the resource indicators corresponding to the node where the database system resides in the target task scheduling round includes: For each of the resource metrics, including the utilization rate, the available capacity corresponding to the utilization rate is calculated. The load factor is determined based on the target available capacity and the average available capacity, wherein the target available capacity is the minimum available capacity among the available capacities corresponding to each item, and the average available capacity is the average of the sum of the available capacities corresponding to each item.

3. The method according to claim 1, characterized in that, The step of adjusting the task admission threshold corresponding to the target task scheduling round based on the load factor includes: In response to the node's data partition merging strategy being in normal mode and the load factor being greater than the action trigger threshold of the hysteresis state machine, the data partition merging strategy is switched from normal mode to suppression mode, and the task admission threshold is increased. The task admission threshold corresponding to the normal mode is a first initial value. The normal mode is used to indicate that the parameter value of the data partition merging strategy remains at a second initial value, and the suppression mode is used to indicate that the parameter value of the data partition merging strategy is lower than the second initial value. In response to the data partition merging strategy being in the suppression mode and the load factor being less than the recovery trigger threshold of the hysteresis state machine, the data partition merging strategy is restored from the suppression mode to the normal mode, and the task admission threshold is restored to the first initial value; In response to the load factor being between the recovery trigger threshold and the action trigger threshold, the node is instructed to maintain the current mode, wherein the current mode is either the normal mode or the suppression mode.

4. The method according to claim 1, characterized in that, The step of obtaining the task evaluation result of each data merging task corresponding to the target task scheduling round includes: For each data merging task, based on the merging evaluation results of each group of partitions to be merged included in the data merging task, the task evaluation result corresponding to the data merging task is determined. Each group of partitions to be merged includes two data partitions to be merged. The merging evaluation result is determined based on the load factor and the target evaluation values ​​corresponding to the two data partitions to be merged. The target evaluation value includes at least one of the following: data volume difference evaluation value, index range overlap evaluation value, data popularity evaluation value, and data timeliness evaluation value.

5. The method according to claim 1, characterized in that, The execution of each of the data merging tasks included in the task queue includes: Obtain the target data merging task from the task queue and execute the target data merging task; In response to the completion of the target data merging task, the target task benefit corresponding to the target data merging task is determined based on the state change of the node before and after the execution of the target data merging task. The state change includes at least one of the following: data query performance degradation, storage space change, and reduction in the number of data partitions. Based on the target task revenue and historical revenue trend, the target revenue trend corresponding to the target data merging task is determined, wherein the historical revenue trend is the task revenue trend corresponding to the previous data merging task. In response to the target return trend being less than the historical return trend, the weight parameters of the target feature dimension are adjusted, wherein the target feature dimension includes at least one feature dimension used to determine the task evaluation result.

6. The method according to claim 5, characterized in that, The step of determining the target revenue trend corresponding to the target data merging task based on the target task revenue and historical revenue trends includes: The target return trend is obtained by processing the target task return and the historical return trend based on the decay factor.

7. The method according to claim 5, characterized in that, The adjustment is used to determine the weight parameters of the task evaluation result, including: The feature dimension whose evaluation value in the task evaluation result is higher than a preset threshold is taken as the target feature dimension; For each target feature dimension, the updated weight parameters corresponding to the target feature dimension are determined based on the adjustment step size, the target return trend, the historical return trend, the evaluation value corresponding to the target feature dimension, and the weight parameters before the update corresponding to the target feature dimension.

8. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method as claimed in any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-7.

Images