Method for dynamically dividing data into buckets

Abstract

The invention provides a method for dynamically dividing data into buckets. The bucket redistribution operation mode is dynamically determined by obtaining the bucket distribution heat parameter of the service data and the load parameter of the computing node in real time. And based on joint judgment of the popularity and the load, triggering a bucket splitting or merging operation, generating a target bucket scheme and storing an incremental snapshot, thereby ensuring that the affiliation relationship of the business data in the bucket adjustment process is clear and traceable. The incremental snapshot provides a core basis for lossless recovery of a subsequent calculation state by recording bucket metadata, calculating offset and intermediate result data. Furthermore, by predicting an expected load parameter of a computing node and combining a target bucket distribution scheme, the bucket distribution is dynamically distributed to a load matching node, and the problem of resource distribution stiffness of static bucket distribution is solved. The computing node recovers the business data processing flow based on the incremental snapshot, so that repeated calculation or loss of data caused by bucket adjustment is avoided, and the real-time data processing efficiency, the resource utilization rate and the business continuity are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of data management, and in particular to a method for dynamic data bucketing. Background Technology

[0002] With the rapid development of the digital economy, real-time data processing has become one of the core capabilities supporting backend business operations, and is widely used in scenarios such as real-time recommendations in e-commerce, real-time risk control in finance, real-time tracking of logistics, and real-time message push on social platforms. Due to the massive amount of user data, real-time computing frameworks are needed to complete data cleaning, feature extraction, and rule matching operations in a short period of time to enable relevant data display and applications.

[0003] The current real-time processing workflow includes data acquisition, bucketing and distribution, node computation, and result output. First, upstream business data is acquired using data acquisition tools. Then, based on a preset bucketing strategy, the data is distributed to different computing nodes. After processing, the computing nodes write the results to downstream storage or push them to the business system. The bucketing strategy is crucial to processing efficiency; reasonable bucketing ensures even data distribution, avoids overloading individual computing nodes, and guarantees real-time performance. However, some feasible implementations suffer from issues such as delayed response to data skew, low resource utilization, redundant data computation, and reliance on manual intervention, leading to poor efficiency and effectiveness, and ultimately, unsatisfactory data processing results. Therefore, a dynamic data bucketing method is needed to address these problems. Summary of the Invention

[0004] The purpose of this application is to at least address one of the aforementioned technical deficiencies, particularly the poor efficiency and effectiveness of data bucketing in the prior art.

[0005] Firstly, this application provides a method for dynamic data bucketing, the method comprising:

[0006] Obtain business data to determine the data heat parameters of each bucket and the current load parameters of each computing node;

[0007] The re-splitting operation method is determined based on the data heat parameters and the current load parameters;

[0008] According to the re-bucketing operation method, the business data is re-bucketed to obtain the target bucketing scheme and generate an incremental snapshot;

[0009] The target bucketing scheme is used to indicate the ownership relationship between the business data and each bucket after re-bucketing, and the incremental snapshot includes bucket metadata, calculated offset and intermediate result data;

[0010] Calculate the expected load parameters for each computing node, and allocate each bucket to each computing node based on the expected load parameters according to the target bucketing scheme;

[0011] Based on the incremental snapshot, the business data corresponding to the buckets on each computing node is obtained through each computing node.

[0012] As an optional implementation, determining the re-splitting operation method based on the data heat parameter and the current load parameter includes:

[0013] When the data heat parameter of any bucket exceeds the first data threshold, or the current load parameter of the corresponding computing node exceeds the first load threshold, a bucket splitting operation is triggered to determine the first target sub-bucket and the data splitting method, and to update the data routing according to the first target sub-bucket and the data splitting method.

[0014] If the data heat parameter of any bucket is continuously lower than the second data threshold within a preset number of statistical periods, a bucket merging operation is triggered. Multiple second target sub-buckets to be merged and the data merging method are determined, and the target merged bucket after merging is determined. Data routing is updated according to each second target sub-bucket, the target merged bucket, and the data merging method.

[0015] As an optional implementation, the data heat parameter includes the number of records per second per bucket, and the data acquisition method for the data heat parameter includes:

[0016] The business data and its corresponding bucket are acquired in real time. The number of data entries and bytes in the corresponding bucket are accumulated by a counter to determine the number of records per second of the bucket within a preset statistical period.

[0017] Furthermore, the first data threshold is determined based on the average and standard deviation of the number of records per second for all buckets, and the second data threshold is determined based on the product of the average number of records per second for the corresponding bucket and a preset coefficient. The first data threshold and the second data threshold are updated according to a preset update cycle.

[0018] Furthermore, the calculation method for the current load parameters includes:

[0019] Obtain the original parameters of each computing node;

[0020] The raw parameters include processor utilization, memory utilization, network input / output rate, and disk input / output rate.

[0021] The current load parameters are calculated using a load scoring model.

[0022] The load scoring model is used to perform a weighted summation of the original parameters.

[0023] As an optional implementation, the bucket splitting operation includes:

[0024] The number of splits is determined based on the ratio of the data heat parameter to the first data threshold.

[0025] If the original bucket is a hash bucket type, the original hash range is split into multiple consecutive sub-ranges and assigned to each of the first target sub-buckets. If the original bucket is a time window bucket type, the original time window is split into multiple consecutive sub-windows and assigned to each of the first target sub-buckets.

[0026] The identifiers of each of the first target sub-buckets are numbered consecutively based on the original bucket identifiers.

[0027] As an optional implementation, the bucket merging operation includes:

[0028] Determine the continuity of the identifiers of each second target sub-bucket, and prioritize matching and merging second target sub-buckets with continuous identifiers to generate the target merged bucket and its corresponding identifier;

[0029] If there are no consecutive buckets, calculate the absolute value of the difference between the sum of the data heat parameters in each of the second target sub-buckets and the second data threshold, and select the bucket combination with the smallest absolute value of the difference for merging to generate the target merged bucket and its corresponding identifier.

[0030] As an optional implementation, the step of calculating the expected load parameters of each computing node and allocating each bucket to each computing node based on the expected load parameters according to the target bucketing scheme includes:

[0031] The initial bucketing method is determined based on the data heat parameters of each bucket and the current load parameters of each computing node;

[0032] The initial bucketing method is used to indicate the matching method of negative correlation between the data heat parameter and the current load parameter;

[0033] Based on the initial bucketing method, the expected load parameters of each computing node are calculated according to the current load parameters and data heat processing capability parameters of each computing node, as well as the data heat parameters corresponding to the buckets to be allocated to each computing node in the initial bucketing method.

[0034] Based on the initial bucketing method, a computing node with an expected load parameter lower than the second load threshold is selected as the first target node for each bucket. If there are multiple first target nodes that meet the conditions, a node with a network input / output rate lower than a preset latency threshold is selected as the first target node, until each bucket in the target bucketing scheme is matched to a first target node.

[0035] As an optional implementation, the upper limit of the number of bucket matching for each computing node is determined based on the number of processor cores of the corresponding computing node, and the method further includes:

[0036] Continuously monitor the current load parameters of each computing node;

[0037] If the current load parameter of any computing node exceeds the preset third load threshold, a bucket transfer operation is triggered, and the buckets on the corresponding computing node whose data heat parameter is lower than the third data threshold are selected and migrated to the second target node.

[0038] Wherein, the current load parameter of the second target node is lower than the third load threshold, and the number of bucket matching has not reached the upper limit.

[0039] Secondly, this application provides a data dynamic bucketing device, comprising:

[0040] The acquisition module is used to acquire business data and determine the data heat parameters of each bucket and the current load parameters of each computing node.

[0041] The processing module is used to determine the re-splitting operation mode based on the data heat parameters and the current load parameters;

[0042] The processing module is further configured to re-bucket the business data according to the re-bucketing operation method, obtain the target bucketing scheme, and generate an incremental snapshot;

[0043] The target bucketing scheme is used to indicate the ownership relationship between the business data and each bucket after re-bucketing, and the incremental snapshot includes bucket metadata, calculated offset and intermediate result data;

[0044] The processing module is also used to calculate the expected load parameters of each computing node, and allocate each bucket to each computing node based on the expected load parameters according to the target bucketing scheme;

[0045] The processing module is further configured to obtain the business data corresponding to the buckets on each computing node based on the incremental snapshot and through each computing node.

[0046] Thirdly, this application provides a computer device including one or more processors and a memory storing computer-readable instructions that, when executed by the one or more processors, perform the steps of the method described in the first aspect.

[0047] Fourthly, this application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method described in the first aspect.

[0048] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0049] Based on any of the above embodiments, this application systematically solves the data skew problem in real-time data processing by constructing a closed-loop process of "data heat monitoring → node load monitoring → dynamic bucketing adjustment → collaborative node allocation → incremental snapshot switching". The data heat monitoring layer determines the bucketing status based on a dynamic threshold algorithm (such as mean ± standard deviation), replacing fixed thresholds to adapt to business fluctuations; the node load monitoring layer quantifies the real-time load of nodes through multi-dimensional resource indicator weighted scoring. A dual-dimensional linkage judgment mechanism (such as triggering adjustments for both bucket overheating and node overload) avoids the risk of misjudgment based on a single dimension, improving response accuracy. The dynamic bucketing adjustment layer adopts a splitting strategy based on hash range or time window, and an adjacent merging strategy to ensure uniform data distribution and compliance with business logic. The collaborative allocation layer dynamically schedules buckets to the optimal nodes based on a load prediction model, maximizing resource utilization. The incremental snapshot layer supports millisecond-level computational state recovery through lightweight triples (metadata, offset, intermediate results), avoiding the resource overhead and redundant calculations of full snapshots. Ultimately, the system ensures business continuity while significantly improving data processing timeliness and resource utilization efficiency, adapting to the needs of high-concurrency scenarios. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart illustrating a data dynamic bucketing method provided in one embodiment of this application;

[0052] Figure 2 A schematic diagram of the system architecture corresponding to a dynamic data bucketing method provided in one embodiment of this application;

[0053] Figure 3 This is a schematic diagram illustrating the process of splitting or merging buckets in a data dynamic bucketing method provided in one embodiment of this application;

[0054] Figure 4 A flowchart illustrating the data dynamic bucketing method provided in one embodiment of this application in a real-world application scenario;

[0055] Figure 5 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0057] With the rapid development of the digital economy, real-time data processing technology has become one of the core capabilities supporting backend business operations, and is widely used in scenarios such as real-time recommendations in e-commerce, real-time risk control in finance, real-time tracking of logistics, and real-time message push on social platforms. Taking e-commerce platforms as an example, during peak promotional periods, the amount of user behavior data (clicks, add-to-cart, orders) generated per second can reach millions. Real-time computing frameworks (such as Apache Flink and Spark Streaming) are needed to complete data cleaning, feature extraction, and rule matching within seconds to achieve functions such as real-time inventory updates or personalized recommendation pop-ups. In the financial sector, real-time risk control systems need to analyze user transfer and consumption data within 100ms to identify abnormal transactions (such as fraudulent transactions from other locations) and prevent financial losses.

[0058] In some feasible implementations, the real-time data processing flow follows the logic of "data acquisition - bucketing distribution - node computation - result output": First, upstream business data is acquired through data acquisition tools (such as Flink CDC, Kafka); then, the data is distributed to different computing nodes based on a preset bucketing strategy (such as hash bucketing, time window bucketing); after the computing nodes complete the data processing, the results are written to downstream storage (such as Redis, HBase) or pushed to the business system. Among these, the bucketing strategy is crucial to processing efficiency; reasonable bucketing can achieve even data distribution, avoid overloading individual computing nodes, and ensure real-time performance.

[0059] Currently, the mainstream solutions for real-time data bucketing and compute node scheduling in the industry mainly include the following two categories:

[0060] Static bucketing + fixed node allocation scheme:

[0061] This approach is the default implementation for real-time computing frameworks. Its core logic is "predefined bucketing rules, binding computing nodes." The specific implementation is as follows:

[0062] Bucketing Strategy: Before a task starts, bucketing rules are set based on business fields (such as user ID, order ID) or time dimensions (such as a 1-minute window). For example, in a real-time e-commerce order processing scenario, "user ID hash modulo" bucketing is used, where the user ID is modulo 100 to generate 100 buckets, and each bucket corresponds to a fixed computing node (such as bucket 0-24 corresponding to node A, bucket 25-49 corresponding to node B); in a time window scenario, data is divided into buckets every 5 minutes, and each bucket is assigned to a fixed node for processing.

[0063] Node scheduling: Buckets and compute nodes have a static "one-to-one" or "many-to-one" binding relationship, and the bucketing rules and node allocation are not adjusted during task execution. For example, in a Flink task, the bucketing field is specified through the keyBy operator, and the underlying mechanism uses the Slot mechanism to bind the bucket to a fixed TaskManager (compute node).

[0064] This solution is widely used in scenarios where data distribution is relatively uniform, such as e-commerce user behavior analysis during non-major promotional periods and routine transaction data processing.

[0065] Manual intervention in dynamic bin sorting scheme:

[0066] To address the issue of data skew in static bucketing schemes, an optimization approach combining manual monitoring and adjustment can be adopted. The core logic is to monitor data skew in real time and manually adjust the bucketing. The specific implementation is as follows:

[0067] Skew monitoring: The data volume and compute node load of each bucket are monitored through the metric system of the real-time computing framework (such as FlinkMetrics or SparkUI). For example, in the console, operations personnel can view the "Records Per Second (RPS)" of each bucket and the "CPU utilization" and "memory usage" of the nodes. When the RPS of a bucket exceeds that of other buckets by more than 10 times, or the CPU utilization of a node continues to exceed 90%, it is determined to be data skew.

[0068] Manual adjustments: Operations personnel manually modify bucketing rules or adjust node resources. For example, when user behavior data causes the RPS of bucket 3 in the "User ID hash bucket" to suddenly increase to 500,000 / second (while other buckets are only 50,000 / second), operations personnel need to pause the current task, split bucket 3 into "Bucket 3-1" and "Bucket 3-2" (using the "User ID hash + last digit range" rule, such as last digits 0-4 going to 3-1, 5-9 going to 3-2), reallocate nodes, and restart the task; if the node load is too high, it is also necessary to manually expand the computing nodes (such as increasing the number of TaskManagers).

[0069] This solution is primarily applied to scenarios where data skew occurs frequently but business operations can tolerate short-term interruptions, such as online scenarios associated with live streaming or trending events.

[0070] Disadvantages of the static bucketing + fixed node allocation scheme:

[0071] Lack of data skew handling capability: This solution relies on the presupposition of "uniform data distribution." When sudden data skew occurs (such as a 20-fold surge in orders for a popular product during an e-commerce promotion, or a large customer's bulk transfer data in a financial system), the computing nodes bound to that bucket will be instantly overloaded—CPU utilization will soar to over 95%, increasing the risk of memory overflow, causing data processing latency to extend from the normal 500ms to over 5 minutes, or even triggering node crashes, affecting business continuity. For example, during a "Double 11" shopping festival on a certain e-commerce platform in 2024, orders for a popular mobile phone caused the corresponding computing node in the bucket to crash, resulting in a 15-minute delay in inventory updates and an "overselling" problem.

[0072] Low resource utilization: Static binding between buckets and nodes leads to unbalanced resource allocation. For example, in a real-time risk control system, buckets 1-10 are bound to node A (CPU utilization is only 30%), and bucket 11 is bound to node B (CPU utilization is 90%). The idle resources of node A cannot be utilized by bucket 11, while node B is overloaded, resulting in an overall resource utilization of less than 50%.

[0073] Poor scalability: When the amount of business data increases and the number of buckets needs to be increased, the current task needs to be paused, the bucketing rules need to be redefined and restarted, resulting in business interruption (usually the interruption time is 5-10 minutes), which cannot meet the real-time business needs of "7×24 hours uninterrupted" (such as real-time financial transfer monitoring, real-time medical data analysis).

[0074] Disadvantages of manually intervened dynamic bucketing schemes:

[0075] Response Delay: There is a significant time lag between "detecting a skew" and "completing the adjustment." Operations personnel need to regularly check the monitoring panel (usually every 5 minutes). After detecting a skew, they need to analyze the cause (e.g., determine the skewed bucket, identify the skew type), formulate an adjustment plan (e.g., split the bucket, expand the node), and execute the adjustment operations (pause the task, modify the configuration, restart the task). The entire process takes at least 10-20 minutes. During this period, the processing delay of the skewed bucket continues to increase, potentially leading to business failures.

[0076] Risk of data duplication: When manually adjusting buckets, the task needs to be paused and the calculation offset reset (such as Flink's Checkpoint). After restarting, the task will recalculate from the last Checkpoint, causing data processed before the adjustment to be counted repeatedly. For example, after manually splitting buckets, restarting the task may cause 1,000 reconciled transactions to be counted repeatedly, resulting in a "reconciliation amount doubled" error, which requires subsequent manual verification and correction, increasing operational costs.

[0077] Relying on professional operation and maintenance capabilities: Adjusting bucketing rules requires familiarity with the underlying logic of real-time computing frameworks (such as Flink's keyBy operator principle and Spark's Shuffle mechanism), and also requires developing reasonable solutions based on business data characteristics (such as user ID distribution and time window patterns). Ordinary operation and maintenance personnel are unlikely to be competent in this task. For example, because operation and maintenance personnel are not familiar with the "difference between range bucketing and hash bucketing," they may use incorrect rules when splitting skewed buckets, resulting in more uneven data distribution and exacerbating the skew problem.

[0078] To address the problems of "delayed response to data skew, low resource utilization, redundant data calculation, and reliance on manual intervention" in existing technologies, this application aims to provide a dynamic bucketing and computing node collaborative processing system for real-time data skew, specifically including:

[0079] It enables automated bucketing adjustment based on two dimensions: "data popularity" and "node load," without manual intervention, reducing the time for data skew identification and processing from minutes to seconds.

[0080] Establish a collaborative scheduling mechanism between buckets and computing nodes, and dynamically allocate buckets according to node load to improve resource utilization;

[0081] Design an incremental snapshot mechanism to achieve seamless switching during bucket adjustments, avoid duplicate data calculations, and ensure data consistency;

[0082] It is compatible with mainstream real-time computing frameworks (such as Flink and Spark Streaming), has good compatibility and scalability, and meets the real-time business needs of different industries.

[0083] In summary, the technical concept of this application lies in systematically solving the data skew problem in real-time data processing by constructing a closed-loop process of "data heat monitoring → node load monitoring → dynamic bucketing adjustment → collaborative node allocation → incremental snapshot switching". The data heat monitoring layer determines the bucketing status based on a dynamic threshold algorithm (such as mean ± standard deviation), replacing fixed thresholds to adapt to business fluctuations. The node load monitoring layer quantifies the real-time load of nodes through multi-dimensional resource indicator weighted scoring. A dual-dimensional linkage judgment mechanism (such as triggering adjustments for both bucket overheating and node overload) avoids the risk of misjudgment based on a single dimension, improving response accuracy. The dynamic bucketing adjustment layer adopts a splitting strategy based on hash range or time window, and an adjacent merging strategy to ensure uniform data distribution and compliance with business logic. The collaborative allocation layer dynamically schedules buckets to the optimal nodes based on a load prediction model, maximizing resource utilization. The incremental snapshot layer supports millisecond-level computational state recovery through lightweight triples (metadata, offset, intermediate results), avoiding the resource overhead and redundant calculations of full snapshots. Ultimately, the system ensures business continuity while significantly improving data processing timeliness and resource utilization efficiency, adapting to the needs of high-concurrency scenarios.

[0084] The methods provided in this application will be described in detail below based on the corresponding implementation methods in some practical application scenarios.

[0085] Figure 1 This is a flowchart illustrating a data dynamic bucketing method provided in one embodiment of this application, as shown below. Figure 1 As shown, this application provides a method for dynamic data bucketing, the method comprising:

[0086] S101. Obtain business data and determine the data heat parameters of each bucket and the current load parameters of each computing node;

[0087] The data heat parameter is used to indicate the flow of business data in each bucket and the real-time data status in the current data bucket. The current load parameter is used to indicate the real-time performance indicators of the computing nodes that support each bucket. In a specific re-bucketing task, the actual situation of each bucket and each node must be taken into account.

[0088] S102. Determine the re-dividing operation mode based on the data heat parameters and the current load parameters;

[0089] In this application, re-bucketing mainly includes splitting and merging buckets, which depends on the data heat parameters of the bucket itself and the load parameters of the computing nodes that carry the data buckets. It comprehensively considers the expected data volume and node performance, thereby improving the parallelism of data processing. In fact, the need for splitting and merging may occur simultaneously, or only one of them may occur. It is also possible that the current bucketing scheme has already matched the overall performance of the system and there is no need to make further adjustments. In this case, the re-bucketing operation can be understood as maintaining the original bucketing method.

[0090] S103. According to the re-bucketing operation method, the business data is re-bucketed to obtain the target bucketing scheme and generate an incremental snapshot;

[0091] The target bucketing scheme is used to indicate the ownership relationship between the business data and each bucket after re-bucketing, and the incremental snapshot includes bucket metadata, calculated offset and intermediate result data;

[0092] Incremental snapshots can be used to quickly recover data during bucket resharding. After bucket resharding is complete, each bucket can retrieve the corresponding data from the snapshot path based on the incremental snapshot. For details, please refer to the subsequent description of implementation methods based on the system architecture.

[0093] S104. Calculate the expected load parameters of each computing node, and allocate each bucket to each computing node based on the expected load parameters according to the target bucketing scheme.

[0094] The re-sharding method provided in this application first determines the target sharding scheme, that is, the correspondence between data and shards. Then, it can perform pre-matching between shards and compute nodes. Based on the pre-matching, it calculates the expected load parameters. Then, based on the expected load parameters and the number of shards that the compute nodes can handle, it allocates the load parameters. After the allocation is completed, it can also continuously monitor relevant parameter indicators and perform real-time sharding to optimize the matching scheme between shards and compute nodes.

[0095] S105. Based on the incremental snapshot, obtain the business data corresponding to the bucket on each computing node through each computing node.

[0096] Please see Figure 2 , Figure 2 This is a schematic diagram of the system architecture corresponding to the dynamic data bucketing method provided in one embodiment of this application. This application provides a system architecture to realize the corresponding functions, including seven major modules: data access layer, data heat monitoring layer, node load monitoring layer, dynamic bucketing scheduling layer, incremental snapshot layer, calculation execution layer, and result output layer. Each module works together to realize dynamic bucketing of real-time data and collaborative processing of nodes.

[0097] Specifically, the data access layer is responsible for receiving upstream business data, supporting mainstream data sources such as Kafka, RabbitMQ, and FlinkCDC, and uniformly converting the data into a "key-value pair" format (the key is the bucket association field, such as user ID; the value is the business data content, such as order information), and pushing it to the dynamic bucket scheduling layer and the data heat monitoring layer.

[0098] The data heat monitoring layer is used to count the amount of data (heat) in each bucket in real time, providing a basis for bucket adjustment.

[0099] The node load monitoring layer can collect the hardware resources and operating status (load) of computing nodes, providing a basis for bucket allocation.

[0100] The dynamic bucket scheduling layer is the core module of the system. It receives heat and load data, executes bucket splitting or merging logic, and allocates buckets to the optimal computing nodes.

[0101] The incremental snapshot layer saves intermediate calculation results before bucketing adjustments and restores the calculation based on the snapshot after adjustments, thus avoiding duplicate calculations.

[0102] The computation execution layer consists of multiple computation nodes (such as FlinkTaskManager and SparkExecutor), which receive bucketed data and perform real-time computations (such as data cleaning, feature extraction, and rule matching).

[0103] The output layer pushes the calculation results to downstream storage (Redis, HBase) or business systems (such as real-time recommendation engines and risk control decision systems), supporting synchronous or asynchronous output modes.

[0104] In other embodiments, this application will provide a detailed description based on the system corresponding to this architecture and in conjunction with the application scenario.

[0105] This implementation dynamically determines the re-bucketing operation mode by acquiring real-time bucketing popularity parameters and compute node load parameters of business data, achieving two-dimensional coordinated control of data distribution and resource load. Based on the joint determination of popularity and load, bucket splitting or merging operations are triggered, generating a target bucketing scheme and saving incremental snapshots to ensure that the ownership relationship of business data is clear and traceable during bucketing adjustments. Incremental snapshots record bucket metadata, calculation offsets, and intermediate result data, providing a core basis for lossless recovery of subsequent calculation states. Furthermore, by predicting the expected load parameters of compute nodes and combining them with the target bucketing scheme, buckets are dynamically allocated to load-matching nodes, solving the rigidity of resource allocation in static bucketing. Finally, compute nodes restore the business data processing flow based on incremental snapshots, avoiding data duplication or loss caused by bucketing adjustments, thereby significantly improving the efficiency of real-time data processing, resource utilization, and business continuity.

[0106] As an optional implementation, determining the re-splitting operation method based on the data heat parameter and the current load parameter includes:

[0107] When the data heat parameter of any bucket exceeds the first data threshold, or the current load parameter of the corresponding computing node exceeds the first load threshold, a bucket splitting operation is triggered to determine the first target sub-bucket and the data splitting method, and to update the data routing according to the first target sub-bucket and the data splitting method.

[0108] If the data heat parameter of any bucket is continuously lower than the second data threshold within a preset number of statistical periods, a bucket merging operation is triggered. Multiple second target sub-buckets to be merged and the data merging method are determined, and the target merged bucket after merging is determined. Data routing is updated according to each second target sub-bucket, the target merged bucket, and the data merging method.

[0109] This implementation establishes a precise response mechanism for dynamic bucket adjustment by setting rules to trigger splitting operations when bucket popularity exceeds the limit or nodes are overloaded, and to trigger merging operations when low popularity persists. During splitting operations, the number of sub-buckets is determined based on a popularity threshold ratio, and a continuous splitting strategy based on hash ranges or time windows is used to ensure uniform data distribution and consistency with business logic. During merging operations, buckets with consecutive identifiers are merged first, or bucket combinations with a sum of popularity closest to the threshold are selected to avoid secondary data skew after merging. Seamless switching of bucket affiliation is achieved by updating data routing, ensuring data integrity while reducing node resource waste caused by unreasonable bucket granularity.

[0110] As an optional implementation, the data heat parameter includes the number of records per second per bucket, and the data acquisition method for the data heat parameter includes:

[0111] The business data and its corresponding bucket are acquired in real time. The number of data entries and bytes in the corresponding bucket are accumulated by a counter to determine the number of records per second of the bucket within a preset statistical period.

[0112] Furthermore, the first data threshold is determined based on the average and standard deviation of the number of records per second for all buckets, and the second data threshold is determined based on the product of the average number of records per second for the corresponding bucket and a preset coefficient. The first data threshold and the second data threshold are updated according to a preset update cycle.

[0113] Furthermore, the calculation method for the current load parameters includes:

[0114] Obtain the original parameters of each computing node;

[0115] The raw parameters include processor utilization, memory utilization, network input / output rate, and disk input / output rate.

[0116] The current load parameters are calculated using a load scoring model.

[0117] The load scoring model is used to perform a weighted summation of the original parameters.

[0118] Specifically, the implementation of the data popularity parameter is achieved through a data popularity monitoring layer. The core function of this layer is to calculate the "data popularity value" of each bucket in real time and quantify the data volume of each bucket. The specific implementation is as follows:

[0119] (1) Monitoring indicators:

[0120] Key metric: Records per second (RPS), which is the number of data entries flowing into a bucket per unit of time. The calculation formula is: RPS = Number of data entries in a bucket within the statistical period / Statistical period (seconds).

[0121] Auxiliary metric: Bucket data size (MB / s), used to distinguish between "small number of records with large volume" (such as social data containing image URLs) and "large number of records with small volume" (such as log data containing only IDs). The calculation formula is: Data size = total number of bytes of data in the bucket within the statistical period / statistical period (seconds).

[0122] (2) Data acquisition mechanism:

[0123] Data collection frequency: The "high-frequency and lightweight" data collection mode is adopted, and the statistical period is set to 1 second (which can be dynamically adjusted through configuration) to avoid the data collection operation consuming too many system resources;

[0124] Data collection method: The heat statistics code is embedded in the "bucket preprocessor" of the data access layer (for example, Flink, implemented through a custom MapFunction). For each piece of data received, the bucket to which it belongs is determined according to the key (such as user ID). The number of data entries and bytes in that bucket are accumulated in the local counter. The statistical results (bucket ID, RPS, data size) are reported to the "heat database" of the dynamic bucket scheduling layer every second (stored using Redis hash structure, with the key being the bucket ID and the value being {RPS:xxx, data size:xxx}).

[0125] (3) Setting the heat threshold:

[0126] Dynamic threshold algorithm: This algorithm uses the "mean of RPS + standard deviation" to set high-popularity and low-popularity thresholds, avoiding the problem of fixed thresholds failing to adapt to different business scenarios. The specific formula is as follows:

[0127] High heat threshold = mean RPS of all bins + 3 × standard deviation of bin RPS (when the RPS of a bin exceeds this value, it is judged as a high heat bin and needs to be split).

[0128] Low heat threshold = average RPS of all bins × 0.5 (when the RPS of a bin is lower than this value, it is judged as a low heat bin and needs to be merged).

[0129] Threshold update frequency: The mean and standard deviation are recalculated every 5 seconds to update the threshold, ensuring that the threshold can adapt to changes in the amount of data in real time.

[0130] For example: In a real-time order processing system for an e-commerce platform, there are currently 10 data buckets with RPS of 50,000, 60,000, 40,000, 50,000, 200,000, 50,000, 40,000, 60,000, 50,000, and 40,000 respectively. The calculation yields:

[0131] The average value is (5+6+4+5+20+5+4+6+5+4) / 10 = 64,000.

[0132] Standard deviation ≈ 48,000;

[0133] High heat threshold = 6.4 + 3 × 4.8 = 208,000 (the RPS of bin 4 is 200,000, which is close to the threshold and needs to be closely monitored; if it rises to 210,000 later, it will be judged as high heat).

[0134] Low heat threshold = 6.4 × 0.5 = 32,000 (currently all sub-buckets are above this value, no need to merge).

[0135] Furthermore, the implementation related to load parameters is achieved through a node load monitoring layer. The core function of this layer is to collect the load status of computing nodes in real time, generate load scores, and provide a basis for bucket allocation. The specific implementation is as follows:

[0136] (1) Load indicators:

[0137] Key metrics: CPU utilization (%) and memory utilization (%), reflecting the core resource usage of the node;

[0138] Auxiliary metrics: Network I / O (MB / s) and Disk I / O (MB / s) to avoid the problem of "CPU / memory idle but slow processing" caused by I / O bottlenecks.

[0139] (2) Data acquisition mechanism:

[0140] Data collection tools: Implementations that extend the native metrics system based on a real-time computing framework, for example:

[0141] Flink scenario: By implementing the MetricReporter interface, collect the TaskManager's cpuUsage (CPU utilization), heapMemoryUsage (heap memory utilization), networkSendRate (network sending rate), and diskkWriteRate (disk write rate) metrics;

[0142] Spark scenario: Use SparkListener to monitor metrics such as executorCpuUtilization and executorMemoryUtilization of the Executor;

[0143] Collection frequency: Synchronized with data popularity collection, set to 1 second to ensure that the time granularity of load data and popularity data are consistent;

[0144] Data storage: Node load data (node ​​ID, CPU utilization, memory utilization, network IO, disk IO) is stored in the "load database" of the dynamic bucketing scheduling layer (using the Redis sorted set structure, where the key is the node cluster ID and the value is {node ID: load score}).

[0145] (3) Load scoring model:

[0146] The scoring calculation logic is as follows: A weighted summation and normalization method is used to convert multi-dimensional load metrics into a load score of 0-10 (10 points indicates a node is completely overloaded, and 0 points indicates a node is completely idle). The specific formula is as follows:

[0147] Raw score = (CPU utilization × 0.4) + (Memory utilization × 0.3) + (Network I / O / Maximum Network I / O × 0.15) + (Disk I / O / Maximum Disk I / O × 0.15);

[0148] Load score = raw score × 10 (rounded to 1 decimal place);

[0149] Note: Maximum network I / O and maximum disk I / O are hardware specifications of the compute node (e.g., 1000MB / s), which are pre-entered into the system through configuration.

[0150] Load level classification:

[0151] Low load: 0-3 points (node ​​resources are sufficient to receive more buckets);

[0152] Medium load: 3-7 points (node ​​resources are normal, maintain the existing bucketing).

[0153] High load: 7-10 points (node ​​resources are tight, some buckets need to be transferred).

[0154] For example, the real-time load data for a Flink compute node (maximum network IO = 1000 MB / s, maximum disk IO = 500 MB / s) is as follows: CPU utilization = 60%, memory utilization = 50%, network IO = 300 MB / s, disk IO = 100 MB / s. The calculation yields:

[0155] The original score = (60 × 0.4) + (50 × 0.3) + (300 / 1000 × 0.15) + (100 / 500 × 0.15) = 24 + 15 + 0.045 + 0.03 = 39.075;

[0156] Load score = 39.075 / 10 = 3.9 points (medium load level, maintain the existing buckets).

[0157] This implementation uses the number of records per second per bucket as the core heat index. It quantifies sudden fluctuations in data distribution by accumulating the number of data entries and bytes per bucket in real time and calculating the periodic average. A dynamic threshold algorithm generates a first data threshold based on the mean and standard deviation of the number of records per second for all buckets. A second data threshold is generated by combining the mean with a preset coefficient. This threshold is periodically updated to adapt to changes in workload, solving the problem of lagging response to business fluctuations with fixed thresholds. Node load parameters integrate multi-dimensional resource indicators such as processor, memory, network, and disk through a weighted summation model to form a unified load score, providing a comprehensive and real-time decision-making basis for bucket allocation.

[0158] As an optional implementation, the bucket splitting operation includes:

[0159] The number of splits is determined based on the ratio of the data heat parameter to the first data threshold.

[0160] If the original bucket is a hash bucket type, the original hash range is split into multiple consecutive sub-ranges and assigned to each of the first target sub-buckets. If the original bucket is a time window bucket type, the original time window is split into multiple consecutive sub-windows and assigned to each of the first target sub-buckets.

[0161] The identifiers of each of the first target sub-buckets are numbered consecutively based on the original bucket identifiers.

[0162] In this implementation, the number of splits is dynamically determined based on the ratio of heat parameters to thresholds during the bucketing operation, ensuring that the granularity of sub-buckets matches the data pressure. For hash bucketing, the original hash range is split into consecutive sub-ranges and allocated to sub-buckets; for time window bucketing, the time window is split into consecutive sub-windows, preserving the locality of data in terms of time or business logic. The sub-bucket identifiers are designed to be consecutively numbered based on the original bucket identifiers, maintaining the logical order of the bucketing system and avoiding a surge in routing complexity caused by splitting, thus balancing processing efficiency and business interpretability during data redistribution.

[0163] As an optional implementation, the bucket merging operation includes:

[0164] Determine the continuity of the identifiers of each second target sub-bucket, and prioritize matching and merging second target sub-buckets with continuous identifiers to generate the target merged bucket and its corresponding identifier;

[0165] If there are no consecutive buckets, calculate the absolute value of the difference between the sum of the data heat parameters in each of the second target sub-buckets and the second data threshold, and select the bucket combination with the smallest absolute value of the difference for merging to generate the target merged bucket and its corresponding identifier.

[0166] In this implementation, during the bucket merging operation, buckets with consecutive identifiers are prioritized for merging to maintain the continuity of data range across business dimensions (such as adjacent time windows or hash value ranges), reducing the risk of uneven data distribution caused by cross-range merging. If no consecutive buckets are found, the absolute value of the difference between the sum of the heat parameters of the buckets to be merged and the threshold is calculated, and the bucket combination closest to the threshold is selected to ensure that the heat of the merged buckets is within a reasonable range, avoiding the generation of new hotspots due to excessive merging. The merged bucket identifier adopts the format of "minimum ID - maximum ID" to clearly identify the source of the merge, facilitating routing updates and data traceability.

[0167] As an optional implementation, the step of calculating the expected load parameters of each computing node and allocating each bucket to each computing node based on the expected load parameters according to the target bucketing scheme includes:

[0168] The initial bucketing method is determined based on the data heat parameters of each bucket and the current load parameters of each computing node;

[0169] The initial bucketing method is used to indicate the matching method of negative correlation between the data heat parameter and the current load parameter;

[0170] Based on the initial bucketing method, the expected load parameters of each computing node are calculated according to the current load parameters and data heat processing capability parameters of each computing node, as well as the data heat parameters corresponding to the buckets to be allocated to each computing node in the initial bucketing method.

[0171] Based on the initial bucketing method, a computing node with an expected load parameter lower than the second load threshold is selected as the first target node for each bucket. If there are multiple first target nodes that meet the conditions, a node with a network input / output rate lower than a preset latency threshold is selected as the first target node, until each bucket in the target bucketing scheme is matched to a first target node.

[0172] This implementation prioritizes matching high-heat buckets with low-load nodes through initial bucketing, establishing a negative correlation between load and heat for scheduling. Based on the node's current load parameters, data processing capacity, and the heat parameters of the bucket to be assigned, the expected load parameters after allocation are predicted, proactively mitigating the risk of node overload. Furthermore, nodes with expected loads below a threshold and minimal network latency are selected as target nodes for each bucket, ensuring load balancing while reducing data transmission latency. This dynamic matching loop between buckets and nodes solves the problem of resource allocation being disconnected from real-time load, improving overall resource utilization.

[0173] As an optional implementation, the upper limit of the number of bucket matching for each computing node is determined based on the number of processor cores of the corresponding computing node, and the method further includes:

[0174] Continuously monitor the current load parameters of each computing node;

[0175] If the current load parameter of any computing node exceeds the preset third load threshold, a bucket transfer operation is triggered, and the buckets on the corresponding computing node whose data heat parameter is lower than the third data threshold are selected and migrated to the second target node.

[0176] Wherein, the current load parameter of the second target node is lower than the third load threshold, and the number of bucket matching has not reached the upper limit.

[0177] This implementation limits the number of buckets that can be matched on a single node based on the number of processor cores, preventing performance degradation caused by thread scheduling overload. It continuously monitors node load parameters, automatically triggering a bucket migration operation when a node becomes overloaded, moving low-heat buckets to low-load nodes to achieve elastic resource scheduling. During the migration process, the target node must meet both load thresholds and the upper limit on the number of buckets to ensure system stability after the migration. This mechanism forms a closed loop of "monitoring-migration-rebalancing," effectively responding to sudden load fluctuations and ensuring the continuous and efficient operation of real-time computing.

[0178] In this application, the implementation method for re-bucketing based on bucket and node performance metrics is achieved through a dynamic bucket scheduling layer, which is responsible for "bucket adjustment decisions" and "bucket-node allocation," thus coordinating "data heat" and "node load." Please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram illustrating the process of splitting or merging buckets in a dynamic data bucketing method provided in one embodiment of this application. The following will describe the bucketing and re-bucketing scheduling methods in this application based on specific data from some application scenarios. The dynamic bucketing scheduling layer can read the bucket RPS and various thresholds every second from a preset "heat database" related to data heat parameters and load parameters to determine whether bucketing needs to be adjusted and to achieve coordination between the bucketing method and node load.

[0179] Bucket splitting rules:

[0180] Applicable scenarios: When the RPS of a bucket exceeds the high-heat threshold, or the load score of the computing node corresponding to the bucket is ≥7 (high load).

[0181] Splitting Algorithm: A hybrid "hash + range" splitting method is adopted to balance data uniformity and business interpretability. The specific steps are as follows:

[0182] Determine the split base: The number of splits is determined based on the ratio of the current RPS to the high popularity threshold. The formula is: Number of splits = ceil(current RPS / high popularity threshold) (ceil is rounded up). For example, if the current RPS is 300,000 and the high popularity threshold is 200,000, then the number of splits is 2.

[0183] Generate sub-bucket IDs: The original bucket ID is "B001", and the sub-bucket IDs after splitting are "B001-1" and "B001-2" (incrementing according to the number of splits);

[0184] Splitting logic:

[0185] If the original bucket is a hash bucket (such as user ID hash modulo): when splitting, the original hash range is split into multiple sub-ranges. For example, the original bucket B001 corresponds to "0-9" after the user ID hash modulo 100. After splitting into 2 sub-buckets, B001-1 corresponds to "0-4" and B001-2 corresponds to "5-9".

[0186] If the original bucket is a time window bucket (e.g., "10:00-10:05"): shorten the time window when splitting. For example, after splitting into 2 sub-buckets, B001-1 corresponds to "10:00-10:02" and B001-2 corresponds to "10:03-10:05".

[0187] Data routing update: The new bucketing rules (sub-bucket ID, splitting logic) are pushed to the "bucketing preprocessor" in the data access layer, and subsequent data will be allocated to sub-buckets according to the new rules.

[0188] For example, in a real-time comment processing system of a social media platform, the RPS of bucket B005 ("40-49" after hashing the user ID modulo 100) suddenly increases to 450,000. The high popularity threshold is 200,000, and the calculated number of splits is 3 (45 / 20=2.25, 3 after ceil). After splitting, sub-buckets B005-1 (40-42), B005-2 (43-46), and B005-3 (47-49) are generated. The data access layer then routes the data that falls within the corresponding range after hashing the user ID to the three sub-buckets.

[0189] Bucket merging rules:

[0190] Applicable scenario: The RPS of the bucket is lower than the low heat threshold, and the state is maintained for 3 consecutive statistical periods (3 seconds);

[0191] Merging Algorithm: A "merge adjacent buckets first" strategy is adopted to avoid uneven data distribution after merging. The specific steps are as follows:

[0192] Filtering and merging candidate sets: Traverse all buckets, filter out buckets with RPS below the low heat threshold, and form a candidate set;

[0193] Matching adjacent buckets: In the candidate set, prioritize merging buckets with consecutive bucket IDs (such as B002 and B003). If there are no consecutive buckets, select the bucket whose sum of RPS is closest to the low heat threshold (such as B002's RPS = 20,000, B005's RPS = 15,000, the sum = 35,000, which is close to the low heat threshold of 32,000).

[0194] Generate merged bucket IDs: The merged bucket IDs adopt the format of "minimum bucket ID - maximum bucket ID", for example, B002 and B003 are merged into "B002-003";

[0195] Merge logic:

[0196] Hash bucket merging: Merges multiple sub-ranges into a larger range. For example, after merging B002 (10-14) and B003 (15-19), the range becomes "10-19".

[0197] Time window binning and merging: Merge multiple short windows into one long window. For example, after merging B002 (10:00-10:02) and B003 (10:03-10:05), the window will be "10:00-10:05".

[0198] Data routing update: Push the merged bucket rules to the data access layer, and at the same time notify the computing nodes corresponding to the original buckets to stop receiving data.

[0199] For example, in a real-time logistics tracking system, the low-heat threshold is 30,000. Buckets B006 (RPS = 21,000) and B007 (RPS = 18,000) are below the threshold for three consecutive seconds and are adjacent buckets. After merging, bucket B006-007 is generated. The original data from B006 ("logistic tracking number hash modulo 100 20-24") and B007 ("25-29") are merged into "20-29". The data access layer will then route the corresponding range of logistics tracking data to B006-007.

[0200] (2) Bucket-node collaborative allocation strategy:

[0201] After completing the bucket adjustment, the dynamic bucketing scheduling layer needs to allocate the buckets (including the original buckets and newly split / merged buckets) to the optimal computing nodes to ensure balanced node load. The specific implementation is as follows:

[0202] Allocation principle:

[0203] High-heat buckets are prioritized for allocation to low-load nodes (load score 0-3).

[0204] Medium-heat bins are distributed to medium-load nodes (3-7 points).

[0205] Low-heat buckets can be allocated to medium-load nodes to avoid wasting resources on low-load nodes;

[0206] The number of buckets received by a single node shall not exceed the number of CPU cores of the node (e.g., an 8-core node can receive a maximum of 8 buckets) to avoid thread scheduling overload.

[0207] Allocation process:

[0208] Node filtering: Filter nodes that meet the load level requirements from the "load database". For example, when allocating high-heat buckets, filter nodes with a load score of 0-3.

[0209] Load prediction: Based on the current RPS of the bucket and the historical processing capacity of the node (e.g., a node needs to consume 5% CPU to process 10,000 RPS), predict the change in node load after bucket allocation. The formula is: Predicted load score = Current load score + (Bucket RPS / Node RPS processing capacity) × 10.

[0210] Optimal node selection: Among the selected nodes, the node with the lowest predicted load score and not exceeding 7 (medium load limit) is selected as the target node; if multiple nodes meet the condition, the node with the lowest network IO is selected (to reduce data transmission latency).

[0211] Allocation execution: The mapping relationship between bucket IDs and target node IDs is pushed to the data access layer (for data routing) and the computation execution layer (for nodes to prepare to receive data).

[0212] Dynamic adjustment: If the load score of a node rises to 7 or above after allocation, "bucket transfer" is immediately triggered - some of the low-heat nodes on the node are bucketed and redistributed to other low-load nodes.

[0213] Example: In a real-time financial risk control system, a newly split high-frequency bucket B004-1 (RPS = 150,000) needs to be allocated nodes. Node A (2.5 rating, RPS processing capacity = 50,000 / min, network IO = 100MB / s) and Node B (3.0 rating, RPS processing capacity = 40,000 / min, network IO = 150MB / s) with load scores of 0-3 are selected. The predicted load score is calculated as follows:

[0214] Node A: 2.5 + (15 / 5) × 10 = 2.5 + 3 = 5.5 points (≤ 7 points, meets the requirements);

[0215] Node B: 3.0 + (15 / 4) × 10 = 3.0 + 3.75 = 6.75 points (≤ 7 points, meets the requirements);

[0216] Because node A has a lower predicted load score and lower network I / O, B004-1 is assigned to node A.

[0217] Furthermore, this application addresses the issue of duplicate data calculations during bucket adjustments by using an incremental snapshot layer. Seamless switching of calculations is achieved by saving "incremental snapshots," as specifically implemented below:

[0218] (1) Snapshot content:

[0219] Incremental snapshots consist of three parts: "bucket metadata", "calculated offset", and "intermediate result data".

[0220] Bucket metadata: Bucket ID, bucketing rules before adjustment (such as hash range), and adjustment timestamp;

[0221] Calculate the offset: the reading position of the bucketed data in the data source (such as Kafka Topic, Partition, Offset), to ensure that data is read from the correct position after adjustment;

[0222] Intermediate result data: Intermediate results that were not calculated before the bucketing adjustment (such as temporary sums and counts of aggregation calculations) are stored in the format of "key-value pairs + timestamp" (the key is the identifier of the intermediate result, such as "total order amount", the value is the result value, and the timestamp is the result generation time).

[0223] (2) Snapshot generation timing and method:

[0224] Generation timing: Snapshots are generated only 1 second before bucket splitting / merging to avoid frequent snapshot generation consuming resources;

[0225] Generation method: When the dynamic bucketing scheduling layer triggers a bucketing adjustment, it sends a "snapshot generation instruction" to the corresponding compute node. The instruction includes the bucket ID and the adjustment type (split / merge).

[0226] After receiving the instruction, the compute node pauses the computation of the current bucket (pause time ≤ 100ms) and reads the computation offset and intermediate result data of the current bucket.

[0227] After compressing the snapshot data (bucket metadata + offset + intermediate results) (using the Snappy compression algorithm), it is uploaded to distributed storage (such as HDFS), and the snapshot storage path is reported to the dynamic bucketing scheduling layer.

[0228] The dynamic bucketing scheduling layer associates the snapshot path with the bucket ID and stores it in the "snapshot index library" (which uses MySQL for storage, and the table structure is: bucket ID, snapshot path, generation time, and adjustment type).

[0229] (3) Snapshot-based computational recovery:

[0230] Scenario recovery after splitting: After receiving data, the sub-bucket first obtains the snapshot path of the original bucket from the "snapshot index library", downloads the intermediate result data, and continues to calculate based on the intermediate result (e.g., if the "total order amount" of the original bucket is 1 million yuan, the sub-bucket will add the amount of the new data to the 1 million yuan in subsequent calculations).

[0231] Recovery of merged scenarios: Before receiving data after merging, the snapshots of all buckets to be merged are obtained from the "snapshot index library". The intermediate results are merged (e.g., the "order quantity" of bucket B002 is 500, the quantity of bucket B003 is 300, and the initial value after merging is 800). Then, the calculation continues based on the merged intermediate results.

[0232] Snapshot cleanup: After the bucket adjustment is completed and the calculation returns to normal (no duplicate or missing data for 5 consecutive seconds), the corresponding snapshot data is automatically deleted to release storage resources.

[0233] For example, in a real-time inventory system of an e-commerce platform, bucket B003 (RPS=250,000, high-popularity threshold=200,000) triggers a split, generating sub-buckets B003-1 and B003-2. Before the split, node C generates an incremental snapshot of B003: bucket metadata (B003, hash range 30-39, timestamp 20241001120000), calculated offset (KafkaTopic=order_topic, Partition=3, Offset=100000), and intermediate result data ("remaining inventory"=500, "sold quantity"=1200). After sub-buckets B003-1 and B003-2 download this snapshot, B003-1 continues to process order data with a hash range of 30-34 based on "remaining inventory 500", while B003-2 processes order data with a hash range of 35-39 based on the same initial value, avoiding duplicate statistics caused by recalculating from Offset=0.

[0234] In addition, the system architecture provided in this application may also include some auxiliary modules, specifically including a data access layer, a computation execution layer, and a result output layer.

[0235] The data access layer is based on Flink's SourceFunction and has a custom "bucketing preprocessor". After receiving upstream Kafka data, it determines the bucket to which the data belongs by means of "key matching + rule judgment" according to the bucketing rules pushed by the dynamic bucketing scheduling layer (stored in local cache and synchronized with the scheduling layer every 1 second). Then, it encapsulates the bucket ID and the data into a "bucket data object" and sends it to the target computing node.

[0236] After receiving the "bucketed data object" from the computing node (such as FlinkTaskManager), the computing execution layer calls the business-defined computing logic (such as order amount aggregation and risk control rule matching). If the computing needs to be resumed, the intermediate results of the incremental snapshot are loaded first. After the computing is completed, the results are marked with the bucket ID and timestamp and sent to the result output layer.

[0237] The output layer supports two modes: "output by bucket" and "merged output". Real-time recommendations, risk control and other scenarios require output by bucket (to ensure low latency), and the results are written to Redis (the key is the bucket ID + business identifier, and the value is the result value). Statistical reports and other scenarios require merged output, and the results of multiple buckets are aggregated and written to HBase or ClickHouse. The aggregation logic is configured by the business (such as summation and averaging).

[0238] Please see Figure 4 , Figure 4 This is a flowchart illustrating a data dynamic bucketing method provided in one embodiment of this application in a practical application scenario. Combined with the foregoing embodiments, it provides an application scenario related to real-time order processing in e-commerce, thereby demonstrating the technical effects of this application.

[0239] During major e-commerce promotions, the upstream business system (order system) sends order data for user ID=123 to the data access layer;

[0240] The data access layer determines the data bucket B003 based on the user ID hash modulo (123%100=23) and reports the RPS (220,000 / second) of B003 to the data heat monitoring layer.

[0241] B003 is currently assigned to compute node A. Node A reports load data (CPU=85%, calculated load score 7.5, high load) to the node load monitoring layer.

[0242] The dynamic bucket scheduling layer receives the RPS of B003 (220,000 / second, exceeding the high heat threshold of 200,000 / second) and the load score of node A (7.5 points, high load), determines that B003 needs to be split, and transfers some buckets on node A.

[0243] The dynamic bucketing scheduling layer instructs node A to generate an incremental snapshot of B003 (containing KafkaOffset=150000 and the intermediate result "sold amount=5000 yuan") and upload it to HDFS;

[0244] The dynamic bucketing scheduling layer generates the splitting rules for B003 (splitting it into 2 sub-buckets, B003-1 corresponding to the hash range 20-22, and B003-2 corresponding to 23-25), and pushes them to the data access layer; at the same time, it selects low-load nodes B (score 2.5) and C (score 3.0), assigns B003-1 to node B, and assigns B003-2 to node C;

[0245] The data access layer will then route data whose user ID hash modulo falls between 20 and 22 to B003-1 (node ​​B), data whose hash modulo falls between 23 and 25 to B003-2 (node ​​C), and data (23) with user ID=123 to B003-1 (node ​​B).

[0246] Node B downloads the incremental snapshot of B003, and based on the intermediate result "sold amount = 5000 yuan", adds the order amount of 100 yuan for user ID=123 to get the new result 5100 yuan;

[0247] Node B sends the result to the result output layer, which pushes the "Sold Amount Update" instruction to the downstream business system (inventory system) to complete the real-time inventory adjustment. The entire process takes ≤500ms and there is no duplicate data calculation.

[0248] In summary, the main points of this application are:

[0249] Firstly, a dynamic bucketing mechanism based on a two-dimensional framework of "data popularity - node load":

[0250] It pioneered a dynamic threshold algorithm based on "mean RPS of buckets + standard deviation", which eliminates the need for manually setting fixed thresholds and can adapt to changes in data volume in different business scenarios, solving the problem that fixed thresholds in existing technologies cannot cope with business fluctuations.

[0251] Achieve linked judgment between "data popularity" and "node load" - not only based on bucket RPS (popularity) to trigger adjustments, but also combined with the load score of the corresponding node, to avoid erroneous adjustments such as "bucket popularity is not exceeded but the node is overloaded" or "bucket popularity exceeds the limit but the node is still idle", and ensure the accuracy of bucket adjustment.

[0252] We propose a hybrid splitting method of "hash + range" and a strategy of "prioritizing the merging of adjacent buckets" to balance data uniformity and business interpretability, and solve the problem of secondary skew caused by unreasonable rule design in existing manual splitting.

[0253] Secondly, the collaborative scheduling mechanism between bucketing and computing nodes is a key technical aspect of this application for improving resource utilization, specifically including:

[0254] Establish a "load prediction model" to predict the node load changes after bucket allocation in advance based on bucket RPS and node processing capacity, avoid node overload caused by blind allocation, and solve the resource waste problem of "bucket allocation and node load disconnection" in the existing technology.

[0255] Achieve closed-loop scheduling of "dynamic bucketing-node allocation-load feedback" - continuously monitor node load after bucketing allocation, and immediately trigger bucket transfer if the load exceeds the limit to ensure that the node load is always maintained at a low to medium level (≤7 points) and ensure computing stability;

[0256] Set a constraint rule that "the maximum number of buckets per node = the number of CPU cores" to avoid overloading thread scheduling due to nodes receiving too many buckets, thereby improving computing efficiency.

[0257] Thirdly, the incremental snapshot switching mechanism for bucket adjustments is a key technical aspect of this application for solving data duplication, specifically including:

[0258] Define an incremental snapshot structure of "bucket metadata + calculated offset + intermediate results" to save only the core data required for adjustment, and avoid full snapshots from consuming too much storage and computing resources;

[0259] The system automates the process of "snapshot generation - snapshot download - intermediate result recovery". During bucket adjustment, the computing node does not need to be paused or restarted. The snapshot loading is completed within 100ms, enabling seamless business switching and solving the business interruption problem caused by the need to pause tasks for manual adjustment in existing technologies.

[0260] The design incorporates an "automatic snapshot cleanup" mechanism, which deletes snapshots after the computation has returned to normal, thus avoiding waste of storage resources while ensuring data security (snapshots are stored in distributed storage and support access control).

[0261] Fourthly, the modular architecture design compatible with multiple frameworks is a key technical point for improving the scalability and usability of this application, specifically including:

[0262] It adopts a modular design, with each layer communicating with each other through standardized interfaces (such as RESTAPI and Kafka message queues), supporting independent deployment and upgrades. For example, the data heat monitoring layer can be adapted to the Spark Streaming framework independently without modifying other modules.

[0263] It supports mainstream real-time computing frameworks (Flink, Spark Streaming) and data sources (Kafka, RabbitMQ, FlinkCDC). The adapter can be switched through the configuration file without modifying the core code, which solves the problem of poor scalability caused by the binding of the solution to a specific framework in the existing technology.

[0264] It provides visual configuration interfaces such as "bucketing rule configuration", "load threshold configuration" and "snapshot parameter configuration", which lowers the operation and maintenance threshold and allows ordinary operation and maintenance personnel to complete system configuration without being familiar with the underlying technology.

[0265] Therefore, compared with some feasible implementations of the prior art, the main advantages of this application are:

[0266] (1) Advantages compared to the "static bucketing + fixed node allocation scheme":

[0267] Data skew handling efficiency improved by over 90%: Existing solutions cannot automatically handle data skew, extending processing latency from the normal 500ms to over 5 minutes; this application, through "dual-dimensional dynamic bucketing," can identify skew within 1 second and complete bucket splitting and node allocation within 5 seconds, keeping processing latency within 1 second. For example, during e-commerce promotions, data skew for popular products reduces processing latency from 5 minutes in the existing solution to 500ms, avoiding business failures such as "overselling" and "inventory update delays."

[0268] The utilization rate of computing nodes is increased by more than 60%: the existing solution has a resource utilization rate of less than 50% due to static binding; this application improves the resource utilization rate to more than 80% through "bucket-node collaborative scheduling". For example, in a real-time risk control system, the average CPU utilization rate of the nodes in the existing solution is 40%, which is increased to 85% after adopting this application. The business data volume can be doubled without the need for additional node expansion.

[0269] Significantly improved business continuity: Existing solutions require pausing tasks when adjusting buckets, resulting in 5-10 minutes of business interruption; this application, through "incremental snapshot seamless switching," ensures uninterrupted business during bucket adjustments, increasing availability from 99.9% to 99.999% (reducing annual downtime from 8.76 hours to 5.256 minutes), meeting the needs of industries with extremely high continuity requirements such as finance and healthcare.

[0270] (2) Advantages compared to the "manual intervention dynamic binning scheme":

[0271] Operational efficiency improved by over 95%: Existing solutions require manual monitoring, analysis, and adjustment, taking 10-20 minutes to handle a single data skew. This application achieves fully automated processing, requiring no manual intervention and reducing processing time to less than 5 seconds. Operations personnel no longer need to spend time on bin adjustments and can focus on core business optimization. For example, during live streaming, the existing solution takes 15 minutes to handle data skew, while this application only takes 3 seconds, avoiding user complaints caused by real-time panel update delays.

[0272] The data consistency guarantee capability is significantly enhanced: Existing solutions are prone to data duplication during manual adjustments, with an error rate of about 5%; This application saves intermediate results through "incremental snapshots", and continues the calculation based on the snapshot after adjustment, with a data duplication / omission rate of ≤0.1%. For example, in a real-time reconciliation system, the existing solution requires 1 hour of manual verification and correction of data after manual adjustment, while with this application, no verification is required, directly ensuring the accuracy of reconciliation data.

[0273] Lowering technical barriers and operational costs: Existing solutions rely on professional operations and maintenance personnel familiar with the underlying framework logic, resulting in high labor costs. This application provides a visual configuration interface, allowing ordinary operations and maintenance personnel to master the configuration methods within one hour, while reducing manual intervention and lowering operational costs by more than 60%. For example, in a certain enterprise, the existing solution requires professional engineers to handle bucket adjustments, while after adopting this application, only ordinary operations and maintenance personnel are needed to complete all configurations.

[0274] (3) Advantages in industry adaptability and scalability:

[0275] Cross-industry adaptability: This application can be adapted to real-time data processing scenarios in multiple industries such as e-commerce, finance, logistics, social media, and healthcare.

[0276] In the financial sector: High-frequency trading data is handled with a bias towards certain data types, ensuring a real-time risk control response time of ≤100ms;

[0277] In the medical field: Processing real-time patient monitoring data (such as electrocardiogram data) to avoid monitoring delays caused by data skew;

[0278] Framework and data source compatibility advantages: It supports mainstream real-time computing frameworks such as Flink and Spark Streaming, as well as data sources such as Kafka and FlinkCDC. It can be connected without refactoring the existing system. For example, if an enterprise has existing Spark Streaming tasks, it only needs to add a "bucketing preprocessor" to the data access layer to use the dynamic bucketing function of this application. The transformation cycle is ≤1 day.

[0279] Business scalability: When the business data volume grows from millions to hundreds of millions, this application can provide seamless support through "automatic splitting and bucketing + dynamic expansion of nodes" without modifying the core rules. For example, if the number of users on a social platform grows from 10 million to 100 million, this application will automatically increase the number of buckets from 100 to 1000 and the number of nodes from 10 to 50, without any manual intervention.

[0280] This application also provides a data dynamic bucketing device, including:

[0281] The acquisition module is used to acquire business data and determine the data heat parameters of each bucket and the current load parameters of each computing node.

[0282] The processing module is used to determine the re-splitting operation mode based on the data heat parameters and the current load parameters;

[0283] The processing module is further configured to re-bucket the business data according to the re-bucketing operation method, obtain the target bucketing scheme, and generate an incremental snapshot;

[0284] The target bucketing scheme is used to indicate the ownership relationship between the business data and each bucket after re-bucketing, and the incremental snapshot includes bucket metadata, calculated offset and intermediate result data;

[0285] The processing module is also used to calculate the expected load parameters of each computing node, and allocate each bucket to each computing node based on the expected load parameters according to the target bucketing scheme;

[0286] The processing module is further configured to obtain the business data corresponding to the buckets on each computing node based on the incremental snapshot and through each computing node.

[0287] This implementation dynamically determines the re-bucketing operation mode by acquiring real-time bucketing popularity parameters and compute node load parameters of business data, achieving two-dimensional coordinated control of data distribution and resource load. Based on the joint determination of popularity and load, bucket splitting or merging operations are triggered, generating a target bucketing scheme and saving incremental snapshots to ensure that the ownership relationship of business data is clear and traceable during bucketing adjustments. Incremental snapshots record bucket metadata, calculation offsets, and intermediate result data, providing a core basis for lossless recovery of subsequent calculation states. Furthermore, by predicting the expected load parameters of compute nodes and combining them with the target bucketing scheme, buckets are dynamically allocated to load-matching nodes, solving the rigidity of resource allocation in static bucketing. Finally, compute nodes restore the business data processing flow based on incremental snapshots, avoiding data duplication or loss caused by bucketing adjustments, thereby significantly improving the efficiency of real-time data processing, resource utilization, and business continuity.

[0288] As an optional implementation, the processing module determines the specific method of re-splitting the data based on the data heat parameter and the current load parameter, including:

[0289] When the data heat parameter of any bucket exceeds the first data threshold, or the current load parameter of the corresponding computing node exceeds the first load threshold, a bucket splitting operation is triggered to determine the first target sub-bucket and the data splitting method, and to update the data routing according to the first target sub-bucket and the data splitting method.

[0290] If the data heat parameter of any bucket is continuously lower than the second data threshold within a preset number of statistical periods, a bucket merging operation is triggered. Multiple second target sub-buckets to be merged and the data merging method are determined, and the target merged bucket after merging is determined. Data routing is updated according to each second target sub-bucket, the target merged bucket, and the data merging method.

[0291] This implementation establishes a precise response mechanism for dynamic bucket adjustment by setting rules to trigger splitting operations when bucket popularity exceeds the limit or nodes are overloaded, and to trigger merging operations when low popularity persists. During splitting operations, the number of sub-buckets is determined based on a popularity threshold ratio, and a continuous splitting strategy based on hash ranges or time windows is used to ensure uniform data distribution and consistency with business logic. During merging operations, buckets with consecutive identifiers are merged first, or bucket combinations with a sum of popularity closest to the threshold are selected to avoid secondary data skew after merging. Seamless switching of bucket affiliation is achieved by updating data routing, ensuring data integrity while reducing node resource waste caused by unreasonable bucket granularity.

[0292] As an optional implementation, the data popularity parameter includes the number of records per second per bucket, and the specific method by which the processing module collects data for the data popularity parameter includes:

[0293] The business data and its corresponding bucket are acquired in real time. The number of data entries and bytes in the corresponding bucket are accumulated by a counter to determine the number of records per second of the bucket within a preset statistical period.

[0294] Furthermore, the first data threshold is determined based on the average and standard deviation of the number of records per second for all buckets, and the second data threshold is determined based on the product of the average number of records per second for the corresponding bucket and a preset coefficient. The first data threshold and the second data threshold are updated according to a preset update cycle.

[0295] Furthermore, the specific method by which the processing module calculates the current load parameters includes:

[0296] Obtain the original parameters of each computing node;

[0297] The raw parameters include processor utilization, memory utilization, network input / output rate, and disk input / output rate.

[0298] The current load parameters are calculated using a load scoring model.

[0299] The load scoring model is used to perform a weighted summation of the original parameters.

[0300] This implementation uses the number of records per second per bucket as the core heat index. It quantifies sudden fluctuations in data distribution by accumulating the number of data entries and bytes per bucket in real time and calculating the periodic average. A dynamic threshold algorithm generates a first data threshold based on the mean and standard deviation of the number of records per second for all buckets. A second data threshold is generated by combining the mean with a preset coefficient. This threshold is periodically updated to adapt to changes in workload, solving the problem of lagging response to business fluctuations with fixed thresholds. Node load parameters integrate multi-dimensional resource indicators such as processor, memory, network, and disk through a weighted summation model to form a unified load score, providing a comprehensive and real-time decision-making basis for bucket allocation.

[0301] As an optional implementation, the specific method by which the processing module performs the bucket splitting operation includes:

[0302] The number of splits is determined based on the ratio of the data heat parameter to the first data threshold.

[0303] If the original bucket is a hash bucket type, the original hash range is split into multiple consecutive sub-ranges and assigned to each of the first target sub-buckets. If the original bucket is a time window bucket type, the original time window is split into multiple consecutive sub-windows and assigned to each of the first target sub-buckets.

[0304] The identifiers of each of the first target sub-buckets are numbered consecutively based on the original bucket identifiers.

[0305] In this implementation, the number of splits is dynamically determined based on the ratio of heat parameters to thresholds during the bucketing operation, ensuring that the granularity of sub-buckets matches the data pressure. For hash bucketing, the original hash range is split into consecutive sub-ranges and allocated to sub-buckets; for time window bucketing, the time window is split into consecutive sub-windows, preserving the locality of data in terms of time or business logic. The sub-bucket identifiers are designed to be consecutively numbered based on the original bucket identifiers, maintaining the logical order of the bucketing system and avoiding a surge in routing complexity caused by splitting, thus balancing processing efficiency and business interpretability during data redistribution.

[0306] As an optional implementation, the specific method by which the processing module performs the bucket merging operation includes:

[0307] Determine the continuity of the identifiers of each second target sub-bucket, and prioritize matching and merging second target sub-buckets with continuous identifiers to generate the target merged bucket and its corresponding identifier;

[0308] If there are no consecutive buckets, calculate the absolute value of the difference between the sum of the data heat parameters in each of the second target sub-buckets and the second data threshold, and select the bucket combination with the smallest absolute value of the difference for merging to generate the target merged bucket and its corresponding identifier.

[0309] In this implementation, during the bucket merging operation, buckets with consecutive identifiers are prioritized for merging to maintain the continuity of data range across business dimensions (such as adjacent time windows or hash value ranges), reducing the risk of uneven data distribution caused by cross-range merging. If no consecutive buckets are found, the absolute value of the difference between the sum of the heat parameters of the buckets to be merged and the threshold is calculated, and the bucket combination closest to the threshold is selected to ensure that the heat of the merged buckets is within a reasonable range, avoiding the generation of new hotspots due to excessive merging. The merged bucket identifier adopts the format of "minimum ID - maximum ID" to clearly identify the source of the merge, facilitating routing updates and data traceability.

[0310] As an optional implementation, the processing module calculates the expected load parameters of each computing node, and allocates each bucket to each computing node based on the expected load parameters according to the target bucketing scheme, including:

[0311] The initial bucketing method is determined based on the data heat parameters of each bucket and the current load parameters of each computing node;

[0312] The initial bucketing method is used to indicate the matching method of negative correlation between the data heat parameter and the current load parameter;

[0313] Based on the initial bucketing method, the expected load parameters of each computing node are calculated according to the current load parameters and data heat processing capability parameters of each computing node, as well as the data heat parameters corresponding to the buckets to be allocated to each computing node in the initial bucketing method.

[0314] Based on the initial bucketing method, a computing node with an expected load parameter lower than the second load threshold is selected as the first target node for each bucket. If there are multiple first target nodes that meet the conditions, a node with a network input / output rate lower than a preset latency threshold is selected as the first target node, until each bucket in the target bucketing scheme is matched to a first target node.

[0315] This implementation prioritizes matching high-heat buckets with low-load nodes through initial bucketing, establishing a negative correlation between load and heat for scheduling. Based on the node's current load parameters, data processing capacity, and the heat parameters of the bucket to be assigned, the expected load parameters after allocation are predicted, proactively mitigating the risk of node overload. Furthermore, nodes with expected loads below a threshold and minimal network latency are selected as target nodes for each bucket, ensuring load balancing while reducing data transmission latency. This dynamic matching loop between buckets and nodes solves the problem of resource allocation being disconnected from real-time load, improving overall resource utilization.

[0316] As an optional implementation, the upper limit of the number of bucket matching for each computing node is determined based on the number of processor cores of the corresponding computing node, and the processing module is further configured to:

[0317] Continuously monitor the current load parameters of each computing node;

[0318] If the current load parameter of any computing node exceeds the preset third load threshold, a bucket transfer operation is triggered, and the buckets on the corresponding computing node whose data heat parameter is lower than the third data threshold are selected and migrated to the second target node.

[0319] Wherein, the current load parameter of the second target node is lower than the third load threshold, and the number of bucket matching has not reached the upper limit.

[0320] This implementation limits the number of buckets that can be matched on a single node based on the number of processor cores, preventing performance degradation caused by thread scheduling overload. It continuously monitors node load parameters, automatically triggering a bucket migration operation when a node becomes overloaded, moving low-heat buckets to low-load nodes to achieve elastic resource scheduling. During the migration process, the target node must meet both load thresholds and the upper limit on the number of buckets to ensure system stability after the migration. This mechanism forms a closed loop of "monitoring-migration-rebalancing," effectively responding to sudden load fluctuations and ensuring the continuous and efficient operation of real-time computing.

[0321] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, a processing module can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as program code in the device's memory, and its functions can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0322] Indicatively, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 5 The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as application programs, that can be executed by the processing component 302. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the methods of any of the embodiments described above.

[0323] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0324] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0325] This application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the method provided in any embodiment.

[0326] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0327] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0328] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for dynamic data bucketing, characterized in that, include: Obtain business data to determine the data heat parameters of each bucket and the current load parameters of each computing node; The re-splitting operation method is determined based on the data heat parameters and the current load parameters; According to the re-bucketing operation method, the business data is re-bucketed to obtain the target bucketing scheme and generate an incremental snapshot; The target bucketing scheme is used to indicate the ownership relationship between the business data and each bucket after re-bucketing, and the incremental snapshot includes bucket metadata, calculated offset and intermediate result data; Calculate the expected load parameters for each computing node, and allocate each bucket to each computing node based on the expected load parameters according to the target bucketing scheme; Based on the incremental snapshot, the business data corresponding to the buckets on each computing node is obtained through each computing node.

2. The method according to claim 1, characterized in that, The step of determining the re-splitting operation method based on the data heat parameters and the current load parameters includes: When the data heat parameter of any bucket exceeds the first data threshold, or the current load parameter of the corresponding computing node exceeds the first load threshold, a bucket splitting operation is triggered to determine the first target sub-bucket and the data splitting method, and to update the data routing according to the first target sub-bucket and the data splitting method. If the data heat parameter of any bucket is continuously lower than the second data threshold within a preset number of statistical periods, a bucket merging operation is triggered. Multiple second target sub-buckets to be merged and the data merging method are determined, and the target merged bucket after merging is determined. Data routing is updated according to each second target sub-bucket, the target merged bucket, and the data merging method.

3. The method according to claim 2, characterized in that, The data popularity parameter includes the number of records per second per bucket, and the data acquisition method for the data popularity parameter includes: The business data and its corresponding bucket are acquired in real time. The number of data entries and bytes in the corresponding bucket are accumulated by a counter to determine the number of records per second of the bucket within a preset statistical period. Furthermore, the first data threshold is determined based on the average and standard deviation of the number of records per second for all buckets, and the second data threshold is determined based on the product of the average number of records per second for the corresponding bucket and a preset coefficient. The first data threshold and the second data threshold are updated according to a preset update cycle. Furthermore, the calculation method for the current load parameters includes: Obtain the original parameters of each computing node; The raw parameters include processor utilization, memory utilization, network input / output rate, and disk input / output rate. The current load parameters are calculated using a load scoring model. The load scoring model is used to perform a weighted summation of the original parameters.

4. The method according to claim 2, characterized in that, The bucket splitting operation includes: The number of splits is determined based on the ratio of the data heat parameter to the first data threshold. If the original bucket is a hash bucket type, the original hash range is split into multiple consecutive sub-ranges and assigned to each of the first target sub-buckets. If the original bucket is a time window bucket type, the original time window is split into multiple consecutive sub-windows and assigned to each of the first target sub-buckets. The identifiers of each of the first target sub-buckets are numbered consecutively based on the original bucket identifiers.

5. The method according to claim 2, characterized in that, The bucket merging operation includes: Determine the continuity of the identifiers of each second target sub-bucket, and prioritize matching and merging second target sub-buckets with continuous identifiers to generate the target merged bucket and its corresponding identifier; If there are no consecutive buckets, calculate the absolute value of the difference between the sum of the data heat parameters in each of the second target sub-buckets and the second data threshold, and select the bucket combination with the smallest absolute value of the difference for merging to generate the target merged bucket and its corresponding identifier.

6. The method according to any one of claims 1-5, characterized in that, The step of calculating the expected load parameters of each computing node and allocating each bucket to each computing node based on the expected load parameters according to the target bucketing scheme includes: The initial bucketing method is determined based on the data heat parameters of each bucket and the current load parameters of each computing node; The initial bucketing method is used to indicate the matching method of negative correlation between the data heat parameter and the current load parameter; Based on the initial bucketing method, the expected load parameters of each computing node are calculated according to the current load parameters and data heat processing capability parameters of each computing node, as well as the data heat parameters corresponding to the buckets to be allocated to each computing node in the initial bucketing method. Based on the initial bucketing method, a computing node with an expected load parameter lower than the second load threshold is selected as the first target node for each bucket. If there are multiple first target nodes that meet the conditions, a node with a network input / output rate lower than a preset latency threshold is selected as the first target node, until each bucket in the target bucketing scheme is matched to a first target node.

7. The method according to claim 6, characterized in that, The upper limit of the number of bucket matching for each computing node is determined based on the number of processor cores of the corresponding computing node, and the method further includes: Continuously monitor the current load parameters of each computing node; If the current load parameter of any computing node exceeds the preset third load threshold, a bucket transfer operation is triggered, and the buckets on the corresponding computing node whose data heat parameter is lower than the third data threshold are selected and migrated to the second target node. Wherein, the current load parameter of the second target node is lower than the third load threshold, and the number of bucket matching has not reached the upper limit.

8. A data dynamic bucketing device, characterized in that, include: The acquisition module is used to acquire business data and determine the data heat parameters of each bucket and the current load parameters of each computing node. The processing module is used to determine the re-splitting operation mode based on the data heat parameters and the current load parameters; The processing module is further configured to re-bucket the business data according to the re-bucketing operation method, obtain the target bucketing scheme, and generate an incremental snapshot; The target bucketing scheme is used to indicate the ownership relationship between the business data and each bucket after re-bucketing, and the incremental snapshot includes bucket metadata, calculated offset and intermediate result data; The processing module is also used to calculate the expected load parameters of each computing node, and allocate each bucket to each computing node based on the expected load parameters according to the target bucketing scheme; The processing module is further configured to obtain the business data corresponding to the buckets on each computing node based on the incremental snapshot and through each computing node.

9. A computer device, characterized in that, The method includes one or more processors and a memory storing computer-readable instructions that, when executed by the one or more processors, perform the steps of the method as described in any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method as described in any one of claims 1-7.

Images