Database htap mixed load transaction analysis resource isolation scheduling method and device

Abstract

The application provides a database HTAP mixed load transaction analysis resource isolation scheduling method and device, designs an HTAP mixed load database and a database resource scheduling technical field, and aims to solve technical problems such as resource preemption, transaction jitter, resource allocation rigidity, and service quality that cannot be classified and guaranteed in the existing HTAP database mixed load running architecture, transaction load and analysis load sharing system resources. The application distinguishes transaction and analysis tasks through multi-dimensional feature recognition, constructs a double-layer isolated resource pool to realize physical isolation of core resources, adjusts resource weight and concurrent quota based on real-time load water level, and simultaneously has load peak shaving, peak shifting and self-healing capabilities, adapts to the original dual-mode storage architecture of the database, improves the overall resource utilization rate and running reliability of the mixed load while ensuring the stability of the core transaction.

Description

Technical Field

[0001] This invention relates to the field of HTAP hybrid load database and database resource scheduling technology, specifically to a method and apparatus for resource isolation scheduling of HTAP hybrid load transaction analysis database. Background Technology

[0002] Huateng HTAP distributed database adopts a dual-mode storage architecture of row-based and column-based storage, uniformly supporting mixed online transaction processing and online analytical processing, which is the mainstream architecture for the current core business system's digital intelligence integration. Currently, the industry's mixed load resource scheduling solutions are mainly divided into three categories. Among them, the fully shared resource scheduling solution allows transaction and analytical tasks to share global system resources without any isolation mechanism. Complex large queries, full table scans, and multi-dimensional aggregation calculations will continuously consume a large amount of resources, directly crowding out short transaction resources, causing core transaction latency to soar and business jitter to occur, failing to meet the service level requirements of core businesses.

[0003] The static resource grouping and isolation scheme can achieve basic isolation by statically dividing resource groups and allocating resource quotas and concurrency limits for two types of loads. However, the rigid resource allocation cannot adapt to business fluctuations. During off-peak periods, transaction resources are idle and wasted, while analysis tasks are congested due to insufficient resources. During peak periods, the fixed quota cannot be dynamically expanded, and transaction contention and delay issues still exist.

[0004] On the other hand, the simple read-write separation scheduling scheme relies on physical splitting of nodes to achieve isolation, but it has problems such as data synchronization delay and inconsistent snapshots. Moreover, there is still task preemption between engines within a single node, which cannot achieve kernel-level fine-grained resource isolation, and the problem of mixed load interference has not been completely solved. Summary of the Invention

[0005] The purpose of this invention is to provide a database HTAP hybrid load transaction analysis resource isolation scheduling method and apparatus that can solve the problem of mutual interference between transaction and analysis loads.

[0006] The specific technical solution of the present invention is as follows:

[0007] The resource isolation and scheduling method for database HTAP hybrid load transaction analysis includes the following steps:

[0008] S1. Multidimensional load intelligent pre-identification: The multidimensional load intelligent pre-identification is to receive HTAP mixed load SQL requests, extract multidimensional load features such as statement type, estimated number of returned rows, whether a full table scan is performed, aggregation calculation complexity, storage engine access type, estimated execution time, and transaction read / write type during the parsing and optimization stage, accurately distinguish four types of load: core OLTP transactions, ordinary OLTP transactions, real-time OLAP analysis, and offline batch OLAP analysis, and mark the corresponding QoS priority level.

[0009] S2. Dual-layer resource pool isolation initialization: The dual-layer resource pool isolation initialization is to build an independent TP transaction-specific resource pool and an AP analysis-specific resource pool, and to perform kernel-level hard isolation on CPU time slices, working memory, IOPS, disk bandwidth, worker thread pool, and task queue. All OLTP tasks are bound to the TP resource pool for execution, and all OLAP tasks are bound to the AP resource pool for execution. Cross-resource pool resource preemption is prohibited.

[0010] S3. Dynamic load level monitoring: The dynamic load level monitoring involves real-time collection of load indicators such as TP transaction QPS, transaction latency, task queue backlog, AP query concurrency, resource utilization, and cache hit rate of each node in the cluster, and generating a global real-time load level model.

[0011] S4. Adaptive weighted dynamic scheduling: The adaptive weighted dynamic scheduling is based on a global load level model to dynamically calculate the TP / AP resource weight ratio and maximum concurrent quota. During peak trading periods, the TP resource weight is automatically increased and the AP resource quota is tightened to ensure that core transactions occupy resources first. During off-peak trading periods, idle TP resources are automatically released and the AP resource quota is expanded to improve the resource utilization rate of analysis tasks.

[0012] S5. Hierarchical QoS queue management: The hierarchical QoS queue management constructs independent scheduling queues according to four load priorities. High-priority core transaction tasks are scheduled and executed first, while low-priority offline analysis tasks are subject to concurrency limiting, queue length limits, and timeout recycling strategies to achieve refined service quality assurance.

[0013] S6. Dual-engine isolated collaborative execution: The dual-engine isolated collaborative execution involves independently maintaining a row-store transaction execution queue and a column-store vectorized analysis execution queue. The thread resources, cache resources, and IO resources of the two types of queues do not compete with each other. The background tasks of database Delta index merging and hierarchical tile index sorting are staggered with the peak transaction periods to achieve kernel mechanism collaborative adaptation.

[0014] S7. Load peak shaving and self-healing: The load peak shaving and self-healing is to automatically perform task segmentation, staggered queuing, flexible current limiting and overload protection for sudden ultra-large analysis loads. After the load peak drops, it automatically resumes normal scheduling to achieve stable self-healing operation of mixed loads.

[0015] Preferably, the dual-layer resource pool hard isolation is a dual-layer architecture of global quota isolation plus single-node kernel resource isolation, which eliminates OLTP and OLAP load resource contention from the two dimensions of cluster resource allocation and single-node kernel scheduling, and eliminates the problem of mutual interference between long and short tasks within a single node.

[0016] Preferably, the adaptive weighted dynamic scheduling is based on real-time collected transaction load rate, analysis load rate, CPU, memory, IO level, and task queue backlog length, dynamically calculating the TP and AP resource weight ratio and maximum concurrency quota. During peak transaction periods, the TP resource weight is automatically increased and the AP resource quota and concurrency are tightened. During off-peak transaction periods, redundant TP resources are automatically released and the AP resource quota is expanded.

[0017] Preferably, the tiered QoS queue management configures differentiated CPU scheduling priorities, memory limits, IOPS quotas, maximum concurrency, queuing timeouts, and task circuit breaker thresholds for core transactions, ordinary transactions, real-time analysis, and offline batch analysis to ensure that core business SLAs are met with priority.

[0018] Preferably, during the dual-engine isolated collaborative execution process, the background tasks of partial merging of the Huateng database Delta incremental index, hierarchical tile index consolidation, and data flushing are automatically identified, and the background operation and maintenance tasks are scheduled to be executed during off-peak periods to avoid performance jitter caused by the superposition of background tasks and front-end load.

[0019] Preferably, the load shaving and peak-shifting self-healing mechanism dynamically segments and splits ultra-large batch analysis tasks, breaking down long-time large tasks into multiple short-term controllable sub-tasks, which are then executed in the background through staggered queuing to avoid system task avalanche caused by instantaneous resource exhaustion.

[0020] A database HTAP hybrid load transaction analysis resource isolation scheduling method and apparatus are disclosed, which are used to implement the database HTAP hybrid load transaction analysis resource isolation scheduling method, including: a multi-dimensional load intelligent identification and classification module, a two-layer resource pool isolation management module, a load level monitoring module, a dynamic weight adaptive scheduling module, a hierarchical QoS control module, a dual-engine queue isolation and collaboration module, and a load peak shaping and offloading self-healing module.

[0021] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a database HTAP hybrid load transaction analysis resource isolation scheduling method.

[0022] The beneficial effects of this invention are as follows:

[0023] This invention completely solves the problem of mutual interference between transaction and analysis tasks under HTAP mixed load by using a kernel-level two-layer resource hard isolation architecture. It effectively ensures the stable latency of core transactions and meets the service level requirements of high-reliability services. At the same time, relying on the dynamic weight adaptive scheduling mechanism driven by load level, it gets rid of the rigidity of static resource quotas, and significantly improves the overall resource utilization while ensuring isolation effect and avoiding resource idleness and waste.

[0024] Moreover, the hierarchical QoS control mechanism of this invention can be applied to the concurrent operation requirements of multi-level mixed services, achieving refined service quality assurance. In addition, combined with the independent queue scheduling of row storage and column storage dual engines, it is deeply adapted to the native kernel mechanism of the database, eliminating resource contention and task conflicts between engines, effectively resisting the impact of sudden analysis traffic, and greatly improving the high-concurrency stability of the HTAP cluster.

[0025] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the method steps of the present invention;

[0027] Figure 2 This is a schematic diagram of the device frame of the present invention;

[0028] Figure 3 This is a schematic diagram of the overall process of the HTAP transaction-analysis two-layer resource isolation intelligent scheduling of the present invention. Detailed Implementation

[0029] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] like Figures 1-2 As shown, the database HTAP hybrid load transaction analysis resource isolation scheduling method and device proposed in this invention rely on the native dual-mode storage architecture of Huateng distributed database row storage transaction engine and column storage analysis engine. It is suitable for high-concurrency, low-latency, and strong-isolation HTAP business scenarios such as financial real-time transactions, military measurement and control real-time analysis, and government digitalization hybrid business.

[0031] First, a multi-dimensional load intelligent pre-identification step is performed. When the system receives a mixed HTAP load SQL request, it extracts seven multi-dimensional load features during the SQL parsing and optimization stage: statement type, estimated number of returned rows, whether a full table scan is performed, aggregation calculation complexity, storage engine access type, estimated execution time, and transaction read / write type. Through a preset load classification model, the request is accurately divided into four types of load: core OLTP transaction, ordinary OLTP transaction, real-time OLAP analysis, and offline batch OLAP analysis. Each type of load is marked with a corresponding QoS priority level. Pre-classification and isolation are completed before the task is enqueued to avoid resource mismatch.

[0032] Based on this, the system completes the dual-layer resource pool isolation configuration during the initialization phase, constructing an independent TP transaction-specific resource pool and an AP analysis-specific resource pool. It adopts a dual-layer architecture of global quota isolation plus single-node kernel resource isolation, and performs kernel-level hard isolation for CPU time slices, working memory, IOPS, disk bandwidth, worker thread pool, and task queue. All OLTP tasks are uniformly bound to the TP resource pool for execution, and all OLAP tasks are uniformly bound to the AP resource pool for execution. Cross-resource pool resource contention is prohibited throughout the process. At the same time, the basic resource weights and concurrency quotas of the two types of resource pools are configured according to the initial business baseline, eliminating OLTP and OLAP load resource contention from the two dimensions of cluster resource allocation and single-node kernel scheduling.

[0033] During continuous system operation, the scheduler continuously executes the dynamic load level monitoring step, collecting load indicators such as TP transaction QPS, average transaction latency, task queue backlog, AP query concurrency, resource utilization, and cache hit rate from each node in the cluster in real time. Based on the collected multi-dimensional operational data, a global real-time load level model is generated to provide data support for subsequent dynamic resource scheduling.

[0034] Based on a global real-time load level model, the system executes an adaptive weighted dynamic scheduling step. Based on real-time collected transaction load rate, analysis load rate, CPU, memory, IO level, and task queue backlog length, the system dynamically calculates the resource weight ratio and maximum concurrency quota of the TP resource pool and AP resource pool. During peak transaction periods, the system automatically increases the TP resource weight and tightens the AP resource quota and concurrency to ensure that core transactions occupy system resources first. During off-peak transaction periods, the system automatically releases redundant idle resources on the TP side and expands the AP resource quota to improve the resource utilization of analysis tasks.

[0035] Then, the system synchronously executes the hierarchical QoS queue management steps, constructs independent scheduling queues according to the priority of four types of load, and configures differentiated CPU scheduling priorities, memory limits, IOPS quotas, maximum concurrency, queuing timeout time, and task circuit breaker thresholds for core transactions, ordinary transactions, real-time analysis, and offline batch analysis. High-priority core transaction tasks are scheduled and executed first, while low-priority offline analysis tasks are subject to concurrency limiting, queue length limits, and timeout recycling policies. Through differentiated scheduling rules, refined service quality assurance is achieved, preventing low-priority tasks from crowding out core business resources and ensuring that core business SLAs are met first.

[0036] To deeply adapt to the native kernel mechanism of Huateng Database, the system implements a dual-engine isolated collaborative execution step, independently maintaining the row-store transaction execution queue and the column-store vectorized analysis execution queue, so that the thread resources, cache resources, and IO resources of the two types of queues do not compete with each other; at the same time, the scheduler automatically identifies background tasks such as partial merging of Huateng Database Delta incremental indexes, hierarchical tile index consolidation, and data flushing, and schedules these background operation and maintenance tasks to be executed during off-peak transaction periods, avoiding performance jitter caused by the superposition of background tasks and foreground load.

[0037] Furthermore, in extreme scenarios involving sudden, massive analytical loads, the system possesses load shaving and self-healing capabilities. It automatically executes task fragmentation, staggered queuing, flexible rate limiting, and overload protection mechanisms. It dynamically fragments and splits massive batch analytical tasks, breaking down long-duration, large tasks into multiple short-term, controllable sub-tasks, which are then executed in the background through staggered queuing to avoid system task avalanches caused by instantaneous resource exhaustion. When the load peak subsides and system resource levels return to normal, the system automatically resumes its normal scheduling strategy, completing the load self-healing closed loop and achieving long-term stable operation under mixed loads.

[0038] Example 1

[0039] In this implementation of mixed load isolation scheduling during peak daytime trading hours, this example is based on a Huateng 6-node HTAP cluster. Daytime business primarily consists of high-concurrency OLTP short transactions such as core financial transactions, account updates, and hotspot queries, accompanied by a small amount of real-time report analysis. In this scenario, the scheduler first performs precise load identification on all incoming requests, marking high-frequency, small-amount transactions and account update requests as core OLTP transactions and configuring them with the highest QoS level, while marking real-time statistical report requests as lightweight real-time AP tasks and configuring them with medium to low priority.

[0040] After load categorization, core transactions are all bound to the dedicated TP resource pool, exclusively occupying 80% of CPU cores, hot memory, and priority IO scheduling rights; real-time analysis tasks are bound to the AP resource pool, allocated only 20% of the elastic resource quota. As daytime business continues, if the system detects that the TP transaction QPS has reached its peak and the task queue backlog has increased, it automatically raises the TP resource weight to 0.9, while tightening the maximum AP concurrency and prohibiting heavy offline analysis tasks from being submitted for execution during that period. At the queue scheduling level, core transaction tasks are prioritized and scheduled without any queuing delay; real-time analysis tasks are placed in a low-priority queue and executed at a limited rate; offline batch analysis tasks are directly intercepted and executed at night.

[0041] Ultimately, the row-based transaction queue handles short transactions at high speed, while the column-based analysis queue executes lightweight reports smoothly at low speed. There is no resource contention between the two engines, and the core transaction latency is stably maintained within 5ms with no significant jitter.

[0042] Example 2

[0043] During nighttime analysis of off-peak resource elastic scaling, when transaction traffic drops significantly during nighttime business hours, TP load levels fall below 20%, and a large number of transaction resources are idle. At this time, the scheduler detects the low TP load in real time and dynamically reduces the TP resource weight to 0.4, releasing a large amount of idle CPU, memory, and IO resources; at the same time, it automatically expands AP resource quotas and maximum concurrency to support the parallel execution of large-scale offline statistics, multi-dimensional aggregation, and data archiving analysis tasks.

[0044] Meanwhile, the scheduler fully utilizes the trading off-peak period to uniformly schedule kernel background tasks such as Delta index merging, hierarchical tile index consolidation, and data flushing, maximizing the value of idle resources. Throughout the nighttime scheduling process, the system consistently ensures the stable execution of sporadic nighttime transactions while significantly improving the throughput of batch analysis tasks, resulting in an overall resource utilization improvement of over 60% for mixed loads.

[0045] Example 3

[0046] During the implementation of peak shaving and self-healing for sudden surges in analytical load, during daytime business hours, if there is a sudden surge in large analytical tasks such as batch full table statistics and multidimensional report aggregation, the instantaneous AP load will spike significantly.

[0047] At this point, the system quickly identifies the heavy offline AP load and automatically triggers the peak shaving protection mechanism. On the one hand, it automatically splits the super-large analysis task into multiple sharded sub-tasks, limits the maximum concurrency of AP at a single moment, and avoids the instantaneous exhaustion of resources. On the other hand, it transfers low-priority offline tasks to the background for off-peak queuing to ensure that they do not preempt core transaction resources in the foreground.

[0048] When the load peak drops and the AP load level decreases, the system automatically resumes the normal scheduling strategy, gradually runs the backlog of analysis tasks, and achieves load self-healing. Throughout the process, the core front-end transaction business is unaffected and unaffected, and the system stability is not impacted by sudden load.

[0049] In summary

[0050] This invention addresses core issues in HTAP hybrid load operations, such as disordered preemption of transaction and analytical resources, rigid scheduling modes, and difficulty in guaranteeing graded service quality. It constructs a complete resource isolation and intelligent scheduling system. Through precise load classification and a two-layer resource isolation mechanism, the solution prevents mutual interference between different loads from the source, ensuring the stable operation quality of core services.

[0051] This invention, through dynamic adaptive scheduling and hierarchical management logic, effectively improves overall resource utilization while maintaining strong isolation. It is suitable for business fluctuations and has load shaving and self-healing capabilities, which can effectively resist sudden traffic surges and greatly improve the reliability and adaptability of mixed load operation, meeting the business needs of various high-concurrency and high-reliability scenarios.

[0052] The embodiments described above are merely illustrative of implementation methods of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A resource isolation and scheduling method for database HTAP hybrid load tracing transaction analysis, characterized in that, Includes the following steps: S1. Multidimensional load intelligent pre-identification: The multidimensional load intelligent pre-identification is to receive HTAP mixed load SQL requests, extract multidimensional load features such as statement type, estimated number of returned rows, whether a full table scan is performed, aggregation calculation complexity, storage engine access type, estimated execution time, and transaction read / write type during the parsing and optimization stage, accurately distinguish four types of load: core OLTP transactions, ordinary OLTP transactions, real-time OLAP analysis, and offline batch OLAP analysis, and mark the corresponding QoS priority level. S2. Dual-layer resource pool isolation initialization: The dual-layer resource pool isolation initialization is to build an independent TP transaction-specific resource pool and an AP analysis-specific resource pool, and to perform kernel-level hard isolation on CPU time slices, working memory, IOPS, disk bandwidth, worker thread pool, and task queue. All OLTP tasks are bound to the TP resource pool for execution, and all OLAP tasks are bound to the AP resource pool for execution. Cross-resource pool resource preemption is prohibited. S3. Dynamic load level monitoring: The dynamic load level monitoring involves real-time collection of load indicators such as TP transaction QPS, transaction latency, task queue backlog, AP query concurrency, resource utilization, and cache hit rate of each node in the cluster, and generating a global real-time load level model. S4. Adaptive weighted dynamic scheduling: The adaptive weighted dynamic scheduling is based on a global load level model to dynamically calculate the TP / AP resource weight ratio and maximum concurrent quota. During peak trading periods, the TP resource weight is automatically increased and the AP resource quota is tightened to ensure that core transactions occupy resources first. During off-peak trading periods, idle TP resources are automatically released and the AP resource quota is expanded to improve the resource utilization rate of analysis tasks. S5. Hierarchical QoS queue management: The hierarchical QoS queue management constructs independent scheduling queues according to four load priorities. High-priority core transaction tasks are scheduled and executed first, while low-priority offline analysis tasks are subject to concurrency limiting, queue length limits, and timeout recycling strategies to achieve refined service quality assurance. S6. Dual-engine isolated collaborative execution: The dual-engine isolated collaborative execution involves independently maintaining a row-store transaction execution queue and a column-store vectorized analysis execution queue. The thread resources, cache resources, and IO resources of the two types of queues do not compete with each other. The background tasks of database Delta index merging and hierarchical tile index sorting are staggered with the peak transaction periods to achieve kernel mechanism collaborative adaptation. S7. Load peak shaving and self-healing: The load peak shaving and self-healing is to automatically perform task segmentation, staggered queuing, flexible current limiting and overload protection for sudden ultra-large analysis loads. After the load peak drops, it automatically resumes normal scheduling to achieve stable self-healing operation of mixed loads.

2. The database HTAP hybrid load transaction analysis resource isolation scheduling method according to claim 1, characterized in that, The dual-layer resource pool hard isolation is a two-layer architecture consisting of global quota isolation and single-node kernel resource isolation. It eliminates OLTP and OLAP load resource contention from two dimensions: cluster resource allocation and single-node kernel scheduling, and eliminates the problem of mutual interference between long and short tasks within a single node.

3. The database HTAP hybrid load transaction analysis resource isolation scheduling method according to claim 1, characterized in that, The adaptive weighted dynamic scheduling is based on real-time collected transaction load rate, analysis load rate, CPU, memory, IO level, and task queue backlog length. It dynamically calculates the weight ratio of TP and AP resources and the maximum concurrency quota. During peak transaction periods, it automatically increases the weight of TP resources and tightens the quota and concurrency of AP resources. During off-peak transaction periods, it automatically releases redundant TP resources and expands the quota of AP resources.

4. The database HTAP hybrid load transaction analysis resource isolation scheduling method according to claim 1, characterized in that, The hierarchical QoS queue management system configures differentiated CPU scheduling priorities, memory limits, IOPS quotas, maximum concurrency, queuing timeouts, and task circuit breaker thresholds for core transactions, ordinary transactions, real-time analysis, and offline batch analysis, ensuring that core business SLAs are met with priority.

5. The database HTAP hybrid load transaction analysis resource isolation scheduling method according to claim 1, characterized in that, During the dual-engine isolated collaborative execution process, the system automatically identifies background tasks such as partial merging of the Huateng database Delta incremental index, hierarchical tile index consolidation, and data flushing. It schedules background operation and maintenance tasks to be executed during off-peak transaction periods to avoid performance fluctuations caused by the combined impact of background tasks and front-end load.

6. The database HTAP hybrid load transaction analysis resource isolation scheduling method according to claim 1, characterized in that, The load shaving and self-healing mechanism dynamically splits and divides large batch analysis tasks into multiple short-term controllable sub-tasks, which are then executed in the background to avoid system task avalanche caused by instantaneous resource exhaustion.

7. A database HTAP hybrid load transaction analysis resource isolation scheduling method, used to implement the method according to any one of claims 1 to 6, characterized in that, include: The system includes a multi-dimensional load intelligent identification and classification module, a dual-layer resource pool isolation management module, a load level monitoring module, a dynamic weight adaptive scheduling module, a hierarchical QoS control module, a dual-engine queue isolation and collaboration module, and a load peak shaving and off-peak self-healing module.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the database HTAP hybrid load transaction analysis resource isolation scheduling method according to any one of claims 1 to 6.

Images