Hybrid transaction processing dynamic resource scheduling method for NewSQL (Structured Query Language) database

Abstract

The invention provides a dynamic resource scheduling method for hybrid transaction processing of a NewSQL (Structured Query Language) database. According to the application, task operation, resource use and data hotspot indexes are acquired in multiple dimensions through the load detection module, and the system state is sensed comprehensively; the task classification and priority evaluation module quantitatively distinguishes OLTP and OLAP tasks and calculates priorities; the resource demand prediction module predicts a resource demand in advance based on a sliding window and a time sequence model; the dynamic scheduling decision module generates a resource allocation strategy through three-level decision, and gives consideration to elastic scheduling of a special pool and a shared pool; a closed-loop optimization strategy of the resource execution and feedback module; and the storage layer collaboration module reduces the calculation load through caching, IO merging and data fragmentation. And finally, the balance between OLTP low delay and OLAP high throughput is realized, and the resource utilization efficiency and transaction processing effectiveness of the NewSQL database in an HTAP scene are improved.

Description

Technical Field

[0001] This application relates to the field of databases, and more particularly to a dynamic resource scheduling method for hybrid transaction processing in NewSQL databases. Background Technology

[0002] With the development of the digital economy, enterprises are increasingly demanding real-time and efficient data processing, making Hybrid Transaction / Analytical Processing (HTAP) a core requirement in the database field. HTAP needs to support both Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP): OLTP focuses on high frequency, short processing time, and low latency (such as e-commerce order placement and payment transactions), while OLAP focuses on large data volumes, complex queries, and high throughput (such as real-time sales analysis and user behavior profiling).

[0003] NewSQL databases, combining the ACID properties of traditional relational databases with the scalability of NoSQL, have become the mainstream choice for HTAP scenarios. However, they face a core challenge: the resource requirements of OLTP and OLAP inherently conflict. OLTP needs to compete for limited CPU and memory resources to ensure low latency, while OLAP requires significant computational and I / O resources to complete complex analyses. Inappropriate resource allocation can lead to soaring OLTP latency or OLAP query timeouts, making it impossible to balance the performance of both types of tasks. Therefore, a dynamic resource scheduling method for hybrid transaction processing in NewSQL databases is needed to balance the resource scheduling requirements of both types of tasks and improve the effectiveness of database transaction execution. Summary of the Invention

[0004] The purpose of this application is to at least solve one of the aforementioned technical defects, particularly the technical defect in the prior art where the conflict between OLTP and OLAP resource scheduling requirements leads to insufficient effectiveness of database transaction operation.

[0005] Firstly, this application provides a method for dynamic resource scheduling using hybrid transaction processing in NewSQL databases, the method comprising:

[0006] The target task is accessed, and the metadata corresponding to the target task is forwarded to the load detection module;

[0007] The load detection module is used to obtain load data of the NewSQL cluster.

[0008] The load data includes task operation metrics, resource utilization metrics, and data hotspot metrics.

[0009] The task classification and priority evaluation module classifies the target task into online transaction processing or online analysis processing based on the metadata and the load data, and calculates the task priority parameter of the target task.

[0010] The resource demand prediction module determines the computing resource demand for the target period in real time based on the sliding window statistics and time series prediction model.

[0011] The dynamic scheduling decision module determines a resource allocation strategy in a dedicated resource pool and / or a shared resource pool based on the load data, the task priority parameters, and the resource requirements.

[0012] The resource allocation strategy is executed through the resource execution and feedback module, and the execution effect of the target task is monitored. The resource allocation strategy is then dynamically adjusted based on the execution effect.

[0013] As an optional implementation, calculating the task priority parameter of the target task includes:

[0014] Determine the weights of the dimensions of task type, response time requirements, business importance, and resource utilization.

[0015] Determine the dimensional scores for the task type, response time requirement, business importance, and resource utilization rate;

[0016] Specifically, based on the task type, fixed dimension scores are assigned to the online transaction processing and the online analysis processing; based on the response time requirement, score gradients are divided in descending order according to a preset duration interval; based on the business importance, the degree of importance is determined according to a preset business type and a dimension score positively correlated with the degree of importance is assigned; and based on the resource utilization rate, score gradients are divided in descending order according to a preset utilization rate interval.

[0017] As an optional implementation, the step of determining the resource requirements for the target time period in real time based on a sliding window statistical and time-series prediction model includes:

[0018] Set a fixed-duration sliding window and statistically analyze the multidimensional statistical indicators within the window.

[0019] The multidimensional statistical indicators are used to indicate the task operation status and computing resource usage over a period of time.

[0020] By using a time-series prediction model, the multidimensional statistical indicators and time characteristics are input, and the amount of computing resources required for the online transaction processing and the online analysis processing are output for the first, second and third time periods in the future.

[0021] The time-series prediction model is trained based on historical data corresponding to the multidimensional statistical indicators and the time features within a preset training period.

[0022] As an optional implementation, the process of determining the resource allocation strategy includes three levels of decision-making;

[0023] The first-level decision-making process uses a bipartite graph matching algorithm to allocate basic computing resources to each of the target tasks.

[0024] The second-level decision triggers a preemption mechanism based on the task priority parameters of each target task. For high-priority tasks whose task priority parameters exceed a preset threshold, if computing resources are insufficient, the shared resource pool resources of low-priority tasks will be occupied according to the reverse order of the task priority parameters, without exceeding the preset minimum resource guarantee ratio.

[0025] The third-level decision is used to perform a degradation operation on a first preset number of online analysis processes when there is still a shortage of computing resources after the high-priority tasks have completed their resource occupation or when the computing resources of the low-priority tasks are lower than the minimum resource guarantee ratio. The degradation operation includes data sharding, data sampling, or delayed execution.

[0026] As an optional implementation method, the dynamic adjustment method of the resource allocation strategy includes:

[0027] Obtain the response time of the online transaction processing and the execution time of the online analysis processing;

[0028] When the response time exceeds the first time threshold, the dynamic scheduling decision module is triggered to recalculate the resource allocation strategy and increase the computing resources for the online transaction processing.

[0029] When the execution time exceeds the second time threshold and there are idle computing resources in the shared resource pool, the computing resources for the online analysis and processing are increased; if there are no idle computing resources in the shared resource pool, the degradation operation is performed on the second preset number of online analysis and processing operations.

[0030] Furthermore, when the resource utilization rate parameter is continuously lower than the preset utilization rate ratio, the ratio between the dedicated resource pool and the shared resource pool is adjusted.

[0031] As an optional implementation, the method further includes storage layer coordination optimization through a storage layer coordination module, specifically including:

[0032] Cache optimization involves loading frequently accessed transaction processing data into the local cache of the compute node and dynamically retaining it for the first cache time; large table data is preloaded into the shared cache, and a least recently used eviction policy is adopted to update the data with the second cache time.

[0033] Data interaction and collaboration are carried out by allocating high priority to the online transaction processing and low priority to the online analysis processing, and merging the data read requests of the online analysis processing into batch data interactions, and executing the batch data interactions according to the load data.

[0034] Data sharding collaboration involves binding the data shards corresponding to the online analysis and processing to the target computing node based on the resource status of each computing node, and generating shard replicas in multiple computing nodes based on the load data.

[0035] Secondly, this application provides a NewSQL database hybrid transaction processing dynamic resource scheduling device, comprising:

[0036] The processing module is used to access the target task and forward the metadata corresponding to the target task to the load detection module;

[0037] The processing module is also used to obtain load data of the NewSQL cluster through the load detection module;

[0038] The load data includes task operation metrics, resource utilization metrics, and data hotspot metrics.

[0039] The processing module is further configured to classify the target task into online transaction processing or online analysis processing based on the metadata and the load data through the task classification and priority evaluation module, and calculate the task priority parameter of the target task;

[0040] The processing module is also used to determine the computing resource requirements for the target time period in real time by using the resource demand prediction module based on the sliding window statistics and time series prediction model.

[0041] The processing module is further configured to determine a resource allocation strategy in a dedicated resource pool and / or a shared resource pool based on the load data, the task priority parameters, and the resource requirements through a dynamic scheduling decision module.

[0042] The processing module is further configured to execute the resource allocation strategy and monitor the execution effect of the target task through the resource execution and feedback module, and dynamically adjust the resource allocation strategy based on the execution effect.

[0043] As an optional implementation, the specific method by which the processing module calculates the task priority parameter of the target task includes:

[0044] Determine the weights of the dimensions of task type, response time requirements, business importance, and resource utilization.

[0045] Determine the dimensional scores for the task type, response time requirement, business importance, and resource utilization rate;

[0046] Specifically, based on the task type, fixed dimension scores are assigned to the online transaction processing and the online analysis processing; based on the response time requirement, score gradients are divided in descending order according to a preset duration interval; based on the business importance, the degree of importance is determined according to a preset business type and a dimension score positively correlated with the degree of importance is assigned; and based on the resource utilization rate, score gradients are divided in descending order according to a preset utilization rate interval.

[0047] Thirdly, this application provides a computer device including one or more processors and a memory storing computer-readable instructions that, when executed by the one or more processors, perform the steps of the method described in the first aspect.

[0048] Fourthly, this application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method described in the first aspect.

[0049] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0050] Based on any of the above embodiments, this application first uses a load detection module to collect high-frequency task operation indicators, resource utilization indicators, and data hotspot indicators, covering multiple dimensions such as OLTP transaction throughput, response time, OLAP query complexity, and data scanning volume, providing panoramic data support for scheduling; the task classification and priority evaluation module accurately distinguishes task types based on SQL parsing and business tags, and quantifies priorities through a weighted scoring model to ensure that high-time-sensitive core tasks such as payments are allocated resources first; the resource demand prediction module combines sliding window statistics of short-cycle load fluctuations with LSTM models to capture long-cycle patterns, outputting future minute-level resource demands in real time to achieve forward-looking scheduling; the dynamic scheduling decision module adopts a three-level decision mechanism, allocating basic resources through bipartite graph matching, prioritizing shared pool resources to ensure critical tasks, and triggering data sharding or sampling degradation to address resource gaps, elastically balancing OLTP and OLAP conflicts; the resource execution and feedback module dynamically adjusts resource quotas based on containers or physical interfaces, and corrects strategies through real-time monitoring of response time and execution duration; the storage layer collaboration module optimizes data access paths, accelerates high-frequency OLTP operations with local caching, improves OLAP throughput with shared caching and batch IO, and reduces network overhead with data sharding and binding. In summary, this solution, through a closed-loop scheduling system with multi-module collaboration, ensures low latency for OLTP while meeting the high throughput requirements of OLAP, significantly improving the resource utilization efficiency and transaction processing effectiveness of NewSQL databases. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A schematic diagram of the system architecture of a NewSQL database hybrid transaction processing dynamic resource scheduling method provided in one embodiment of this application;

[0053] Figure 2 A schematic diagram of the overall process architecture of a NewSQL database hybrid transaction processing dynamic resource scheduling method provided in one embodiment of this application;

[0054] Figure 3 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] With the development of the digital economy, enterprises are increasingly demanding real-time and efficient data processing, making Hybrid Transaction / Analytical Processing (HTAP) a core requirement in the database field. HTAP needs to support both Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP): OLTP focuses on high frequency, short processing time, and low latency (such as e-commerce order placement and payment transactions), while OLAP focuses on large data volumes, complex queries, and high throughput (such as real-time sales analysis and user behavior profiling).

[0057] NewSQL databases combine the ACID properties of traditional relational databases with the scalability of NoSQL, making them the mainstream choice for HTAP scenarios. However, they face a core challenge: the resource requirements of OLTP and OLAP are inherently conflicting. OLTP needs to compete for limited CPU and memory resources to ensure low latency, while OLAP requires significant computing and I / O resources to complete complex analyses. Improper resource allocation can lead to soaring OLTP latency or timeouts in OLAP queries, making it impossible to balance the performance of both types of tasks. Therefore, dynamic resource scheduling for NewSQL has become a key technology for the successful implementation of HTAP scenarios.

[0058] The following describes some feasible implementations of HTAP resource scheduling for NewSQL:

[0059] Static resource allocation scheme:

[0060] Core logic: During the NewSQL cluster deployment phase, resource quotas for OLTP and OLAP are fixedly allocated through configuration files. For example, 50% of the cluster's CPU and memory are allocated to OLTP tasks, and 50% to OLAP tasks.

[0061] Typical implementations include: static resource partitioning in GoogleSpanner and fixed resource pool configuration in Alibaba Cloud PolarDB-X.

[0062] Operation mode: Once resources are allocated, they are not dynamically adjusted throughout the process. OLTP tasks use only the dedicated resource pool, while OLAP tasks use only another part of the dedicated resource pool.

[0063] Rule-based threshold-triggered scheduling scheme:

[0064] Core logic: Preset resource utilization thresholds (such as CPU utilization of 85% and memory utilization of 90%) and task type rules. When the resource utilization of a certain type of task exceeds the threshold, resource adjustment is triggered.

[0065] Typical implementations: Amazon Aurora's load threshold scheduling and PingCAPTiDB's simple load balancing strategy.

[0066] Operation mode: For example, if the CPU utilization of an OLTP task exceeds 85% for 3 consecutive seconds, 20% of the OLAP resource pool will be temporarily allocated to OLTP; when the length of the OLAP query queue exceeds 10, the resources will be reclaimed.

[0067] Simple load-aware scheduling scheme:

[0068] Core logic: By monitoring the utilization of basic resources such as CPU and memory, and combining this with the number of tasks, a greedy algorithm is used to allocate resources.

[0069] Typical implementation: The dynamic resource scheduling module of the open-source NewSQL database CockroachDB.

[0070] Operation mode: Real-time statistics of the current concurrent OLTP and OLAP tasks, and resource allocation according to the "task ratio". For example, if there are 100 OLTP tasks and 50 OLAP tasks, CPU resources will be allocated in a 2:1 ratio.

[0071] However, such feasible implementation methods have their limitations. For example, static resource allocation schemes have extremely poor flexibility and cannot adapt to dynamic load changes. For instance, during e-commerce promotions, OLTP tasks surge, and insufficient dedicated resource pools lead to payment delays; during the early morning when OLTP load is low, dedicated resource pools are idle and wasted, while OLAP tasks cannot quickly complete data analysis due to insufficient resources.

[0072] For example, rule-based threshold-triggered scheduling schemes suffer from rigid threshold settings and a lack of awareness of task characteristics. Relying solely on resource utilization thresholds without considering task priority (e.g., payment transactions are more important than ordinary queries) or query complexity (e.g., OLAP tasks involving multi-table joins consume more resources than single-table queries) leads to unreasonable scheduling decisions, potentially resulting in low-priority tasks occupying the resources of high-priority tasks.

[0073] For example, a simple load-aware scheduling scheme has a single monitoring dimension and a coarse decision-making logic. It only focuses on CPU and memory usage and the number of tasks, ignoring key factors such as IO, network resource load, data hotspot distribution, and business importance. Furthermore, the greedy algorithm is prone to resource allocation imbalance. For instance, allocating resources to complex OLAP tasks and simple OLTP tasks according to the proportion of the number of tasks can lead to OLAP tasks timeouts due to insufficient resources or OLTP tasks experiencing delays due to resource congestion.

[0074] Furthermore, these methods lack storage-compute coordination, are not adapted to the storage-compute separation architecture of NewSQL, and their scheduling only focuses on compute layer resources without considering the coordinated optimization of storage layer caching and IO resources. This results in low data access efficiency and indirectly exacerbates resource waste. Moreover, the scheduling feedback loop is missing; some solutions do not monitor the execution effect in real time after resource adjustments, making it impossible to correct unreasonable scheduling decisions in a timely manner, which can easily lead to performance fluctuations.

[0075] This application is intended to achieve the following objectives:

[0076] 1. Enables dynamic and precise allocation of resources, adapts to the dynamic load changes of NewSQL HTAP, and balances the low latency requirements of OLTP and the high throughput requirements of OLAP.

[0077] 2. Introduce multi-dimensional load monitoring and task priority assessment to ensure that high-importance and high-time-efficiency tasks receive priority access to resources.

[0078] 3. Combine resource demand forecasting with advance resource allocation to avoid performance degradation caused by resource shortages.

[0079] 4. Build a storage and computing collaborative scheduling mechanism to optimize data access efficiency and reduce resource consumption.

[0080] 5. Establish a real-time feedback closed loop to dynamically correct scheduling decisions and ensure system performance stability.

[0081] In summary, the technical concept of this application lies in the following: First, the load detection module frequently collects task operation indicators, resource utilization indicators, and data hotspot indicators, covering multiple dimensions such as OLTP transaction throughput, response time, OLAP query complexity, and data scan volume, providing comprehensive data support for scheduling; the task classification and priority evaluation module accurately distinguishes task types based on SQL parsing and business tags, and quantifies priorities through a weighted scoring model to ensure that high-time-sensitive core tasks such as payments are allocated resources preferentially; the resource demand prediction module combines sliding window statistics of short-cycle load fluctuations with LSTM models to capture long-cycle patterns, and outputs the results in real time. To address minute-level resource demands, proactive scheduling is implemented. The dynamic scheduling decision module employs a three-level decision-making mechanism: it allocates basic resources through bipartite graph matching, prioritizes shared pool resources to ensure critical tasks, and triggers data sharding or sampling degradation to address resource gaps, elastically balancing OLTP and OLAP conflicts. The resource execution and feedback module dynamically adjusts resource quotas based on containers or physical interfaces and uses real-time monitoring of response time and execution duration to implement closed-loop correction strategies. The storage layer collaboration module optimizes data access paths, accelerates high-frequency OLTP operations with local caching, improves OLAP throughput with shared caching and batch I / O, and reduces network overhead with data sharding and binding. In summary, this solution, through a multi-module collaborative closed-loop scheduling system, ensures low latency for OLTP while meeting high throughput requirements for OLAP, significantly improving the resource utilization efficiency and transaction processing effectiveness of NewSQL databases.

[0082] The methods provided in this application will be described in detail below based on the corresponding implementation methods in some practical application scenarios.

[0083] Figure 1 This is a schematic diagram of the system architecture of a NewSQL database hybrid transaction processing dynamic resource scheduling method provided in one embodiment of this application, as shown below. Figure 1 As shown, the following uses a system architecture and data flow corresponding to this application as an example to illustrate the execution of each specific implementation of the method provided in this application.

[0084] The hybrid transaction / analytic processing dynamic resource scheduling engine for NewSQL proposed in this application is based on a storage-compute separation architecture design for NewSQL. Its core consists of 6 functional modules, and it achieves dynamic resource scheduling through a closed-loop process of "monitoring, evaluation, prediction, decision-making, execution, and feedback".

[0085] The scheduling engine is embedded between the access layer and execution layer of the NewSQL database. It collects load data from the computing and storage layers in real time, combines task characteristics to perform priority evaluation and resource demand prediction, generates scheduling decisions, and sends them to computing and storage nodes. At the same time, the feedback module corrects the decisions in real time, forming a closed loop.

[0086] Specifically, this application provides a dynamic resource scheduling method for hybrid transaction processing in NewSQL databases, the method comprising:

[0087] The target task is accessed, and the metadata corresponding to the target task is forwarded to the load detection module;

[0088] The load detection module is used to obtain load data of the NewSQL cluster.

[0089] The load data includes task operation metrics, resource utilization metrics, and data hotspot metrics.

[0090] The task classification and priority evaluation module classifies the target task into online transaction processing or online analysis processing based on the metadata and the load data, and calculates the task priority parameter of the target task.

[0091] As an optional implementation, calculating the task priority parameter of the target task includes:

[0092] Determine the weights of the dimensions of task type, response time requirements, business importance, and resource utilization.

[0093] Determine the dimensional scores for the task type, response time requirement, business importance, and resource utilization rate;

[0094] Specifically, based on the task type, fixed dimension scores are assigned to the online transaction processing and the online analysis processing; based on the response time requirement, score gradients are divided in descending order according to a preset duration interval; based on the business importance, the degree of importance is determined according to a preset business type and a dimension score positively correlated with the degree of importance is assigned; and based on the resource utilization rate, score gradients are divided in descending order according to a preset utilization rate interval.

[0095] This implementation method determines the weights and scores of task type, response time requirements, business importance, and resource utilization. It constructs a multi-dimensional weighted scoring model by assigning fixed scores to task types, descending scores to response times within a range, positively correlated scores to business importance by type, and descending scores to resource utilization within a range. This model quantitatively assesses task priority. It prioritizes resource allocation for high-time-sensitive and high-importance tasks (such as payment transactions), preventing low-priority tasks from crowding out critical resources. This ensures precise matching of resource allocation with business needs, improving the rationality of scheduling decisions and system response efficiency.

[0096] The resource demand prediction module determines the computing resource demand for the target period in real time based on the sliding window statistics and time series prediction model.

[0097] As an optional implementation, the step of determining the resource requirements for the target time period in real time based on a sliding window statistical and time-series prediction model includes:

[0098] Set a fixed-duration sliding window and statistically analyze the multidimensional statistical indicators within the window.

[0099] The multidimensional statistical indicators are used to indicate the task operation status and computing resource usage over a period of time.

[0100] By using a time-series prediction model, the multidimensional statistical indicators and time characteristics are input, and the amount of computing resources required for the online transaction processing and the online analysis processing are output for the first, second and third time periods in the future.

[0101] The time-series prediction model is trained based on historical data corresponding to the multidimensional statistical indicators and the time features within a preset training period.

[0102] This implementation uses a fixed-duration sliding window to statistically analyze historical task performance and resource utilization, capturing short-term load fluctuation patterns. Combined with a time-series prediction model that inputs multi-dimensional statistical indicators and time characteristics, it outputs the resource requirements for online transaction processing and online analytical processing over multiple future time periods, achieving minute-level accurate prediction. This design, through the synergy of the sliding window and the LSTM model, considers both real-time load changes and long-term trends, proactively allocating resources to fill gaps and avoiding OLTP latency spikes or OLAP query timeouts due to resource shortages, thus ensuring stable system performance.

[0103] The dynamic scheduling decision module determines a resource allocation strategy in a dedicated resource pool and / or a shared resource pool based on the load data, the task priority parameters, and the resource requirements.

[0104] As an optional implementation, the process of determining the resource allocation strategy includes three levels of decision-making;

[0105] The first-level decision-making process uses a bipartite graph matching algorithm to allocate basic computing resources to each of the target tasks.

[0106] The second-level decision triggers a preemption mechanism based on the task priority parameters of each target task. For high-priority tasks whose task priority parameters exceed a preset threshold, if computing resources are insufficient, the shared resource pool resources of low-priority tasks will be occupied according to the reverse order of the task priority parameters, without exceeding the preset minimum resource guarantee ratio.

[0107] The third-level decision is used to perform a degradation operation on a first preset number of online analysis processes when there is still a shortage of computing resources after the high-priority tasks have completed their resource occupation or when the computing resources of the low-priority tasks are lower than the minimum resource guarantee ratio. The degradation operation includes data sharding, data sampling, or delayed execution.

[0108] This implementation employs a three-tiered decision-making mechanism to determine resource allocation strategies: The first tier allocates basic resources to tasks using a bipartite graph matching algorithm, maximizing resource utilization and minimizing latency; the second tier triggers a preemption mechanism in reverse order of task priority, ensuring resource needs for high-priority tasks while reserving a minimum resource guarantee ratio for low-priority tasks; the third tier performs degradation operations such as data sharding, data sampling, or delayed execution on online analytical processing to address resource shortages. This tiered strategy, through the synergy of basic allocation, elastic preemption, and graceful degradation, prioritizes critical tasks during resource-scarce periods while preventing low-priority tasks from being completely starved, achieving a dynamic balance of resource conflicts.

[0109] The resource allocation strategy is executed through the resource execution and feedback module, and the execution effect of the target task is monitored. The resource allocation strategy is then dynamically adjusted based on the execution effect.

[0110] As an optional implementation method, the dynamic adjustment method of the resource allocation strategy includes:

[0111] Obtain the response time of the online transaction processing and the execution time of the online analysis processing;

[0112] When the response time exceeds the first time threshold, the dynamic scheduling decision module is triggered to recalculate the resource allocation strategy and increase the computing resources for the online transaction processing.

[0113] When the execution time exceeds the second time threshold and there are idle computing resources in the shared resource pool, the computing resources for the online analysis and processing are increased; if there are no idle computing resources in the shared resource pool, the degradation operation is performed on the second preset number of online analysis and processing operations.

[0114] Furthermore, when the resource utilization rate parameter is continuously lower than the preset utilization rate ratio, the ratio between the dedicated resource pool and the shared resource pool is adjusted.

[0115] This implementation dynamically adjusts resource allocation strategies by acquiring real-time response times for online transaction processing and execution times for online analysis processing: when the response time exceeds a threshold, the strategy is recalculated and online transaction processing resources are increased; when the execution time exceeds a threshold and there are idle resources in the shared pool, online analysis processing resources are increased; otherwise, a degradation operation is triggered; when resource utilization remains consistently low, the ratio of the dedicated pool to the shared pool is adjusted. This feedback mechanism optimizes scheduling strategies in real-time based on task execution performance, avoiding performance fluctuations, ensuring that resource allocation always matches actual load requirements, and improving system stability and resource utilization efficiency.

[0116] As an optional implementation, the method further includes storage layer coordination optimization through a storage layer coordination module, specifically including:

[0117] Cache optimization involves loading frequently accessed transaction processing data into the local cache of the compute node and dynamically retaining it for the first cache time; large table data is preloaded into the shared cache, and a least recently used eviction policy is adopted to update the data with the second cache time.

[0118] Data interaction and collaboration are carried out by allocating high priority to the online transaction processing and low priority to the online analysis processing, and merging the data read requests of the online analysis processing into batch data interactions, and executing the batch data interactions according to the load data.

[0119] Data sharding collaboration involves binding the data shards corresponding to the online analysis and processing to the target computing node based on the resource status of each computing node, and generating shard replicas in multiple computing nodes based on the load data.

[0120] This implementation optimizes data access through a storage layer collaboration module: high-frequency transaction processing data is loaded into the local cache of compute nodes and dynamically retained for expiration, accelerating OLTP access; large table data is preloaded into a shared cache and updated using an LRU strategy, improving OLAP query efficiency; high-priority data interactions are allocated to online transaction processing, and read requests for online analytical processing are merged into batch interactions, reducing IO overhead; data shards are bound to compute node status and replicas are generated to distribute the pressure of hotspot access. This design, through the collaborative optimization of caching, IO, and data sharding, reduces cross-node data transmission, lowers compute layer resource consumption, and improves the overall execution efficiency of HTAP tasks.

[0121] This application achieves multi-dimensional collection of NewSQL cluster load data by accessing the target task and forwarding metadata to the load detection module, including task operation indicators, resource utilization indicators, and data hotspot indicators, thereby comprehensively perceiving the system status. Furthermore, through the task classification and priority evaluation module, the target task is accurately divided into online transaction processing or online analysis processing, and task priority parameters are calculated to provide a basis for resource scheduling decisions. The resource demand prediction module determines the resource demand for the target period in real time based on sliding window statistics and time series prediction models, achieving forward-looking resource planning. The dynamic scheduling decision module integrates load data, task priorities, and resource demands to generate resource allocation strategies in dedicated and shared resource pools, taking into account the differentiated needs of the two types of tasks. Finally, the resource execution and feedback module executes the strategies and monitors the effects, dynamically correcting resource allocation to form a closed loop of "monitoring-evaluation-prediction-decision-execution-feedback," effectively resolving resource conflicts between low latency in OLTP and high throughput in OLAP, and improving the effectiveness and stability of database transaction operation.

[0122] Figure 2 This is a schematic diagram of the overall process architecture of a NewSQL database hybrid transaction processing dynamic resource scheduling method provided in one embodiment of this application.

[0123] The following combination Figure 1 The system architecture and overall process shown are described in detail based on actual application scenarios, and the aforementioned implementation methods are explained in detail.

[0124] Load detection module:

[0125] Core functionality: Collects load data from NewSQL clusters in a multi-dimensional and high-frequency manner to provide data support for subsequent evaluation and prediction.

[0126] Monitoring dimensions and implementation methods:

[0127] 1. Task performance metrics:

[0128] OLTP task: Collect transaction throughput (TPS), average response time, concurrency, lock contention rate, and transaction type (read / write). Capture transaction lifecycle events (commit, rollback, blocking) through the Hook function of the NewSQL execution engine, with a sampling interval of 10ms.

[0129] OLAP task: Collect query execution time, scanned data volume, number of JOIN / aggregation operations, and query status (waiting / executing / completed). Extract query syntax features through the SQL parser and combine them with the log output statistics of the execution engine. The sampling interval is 50ms.

[0130] 2. Resource utilization rate indicators:

[0131] Computation layer: CPU utilization (statistics by core), memory utilization (distinguishing between heap memory and non-heap memory), and network bandwidth usage (transfer rate) are collected through the operating system's proc file system or the MetricsAPI of a container orchestration platform (such as Kubernetes), with a sampling interval of 20ms.

[0132] Storage layer: IOPS (read / write separation), storage latency (block storage response time), and cache hit rate are collected through the monitoring interface of the storage node (such as S3 compatible interface, local storage monitoring tool), with a sampling interval of 50ms.

[0133] 3. Data hotspot metrics: Collect data table / row access frequency and lock contention status, and use NewSQL storage engine metadata (such as index access count and lock wait queue) for statistics, with a sampling interval of 100ms.

[0134] Data processing method: The raw data collected is cleaned (outliers are filtered) and aggregated (average and maximum values ​​within 1 second) using a streaming processing framework (such as FlinkLite), and stored in a local time series database (such as InfluxDB). Data from the most recent 7 days is retained for training the prediction model.

[0135] Task classification and priority evaluation module:

[0136] Core functionality: Accurately differentiate between OLTP and OLAP tasks, evaluate task priority based on multi-dimensional indicators, and ensure that important tasks receive resources first.

[0137] Task categorization implementation:

[0138] 1. Classification rules:

[0139] OLTP task criteria: Transaction execution time ≤ 500ms, no aggregate functions (SUM / AVG / COUNT, etc.), no multi-table JOIN, no data modification operations (INSERT / UPDATE / DELETE) or simple queries (single table, conditional filtering fields include primary keys / indexes).

[0140] OLAP task criteria: transaction execution time > 500ms, includes aggregate functions / multi-table JOIN, scanned data volume > 100MB, query results used for statistical analysis (labeled with business tags).

[0141] 2. Implementation method: The SQL syntax parser (developed based on ANTLR4) extracts keywords, operation types, and estimated data volume information from the query statement, and automatically completes the classification by combining the task execution time threshold; at the same time, it supports users to add business tags (such as "payment transaction" and "real-time report") to tasks via API to assist in classification.

[0142] Priority evaluation implementation:

[0143] 1. Evaluation Model: The weighted scoring method is adopted. The total score = task type weight × type score + response time requirement weight × timeliness score + business importance weight × business score + resource utilization weight × resource score (the total weight is 1).

[0144] 2. Scoring Details:

[0145] The following is a feasible way to assign values. For task type, a weight of 0.4 can be assigned, with OLTP tasks receiving 1.0 point and OLAP tasks receiving 0.2 points. For response time requirements, a weight of 0.3 can be assigned, with ≤100ms receiving 1.0 point, 100ms~500ms receiving 0.7 points, 500ms~1s receiving 0.3 points, and >1s receiving 0.1 points. Business importance is determined based on business logic and can be assigned a weight of 0.2, with core business (payment, order placement) receiving 1.0 point, important business (order query, inventory update) receiving 0.7 points, and ordinary business (historical data query) receiving 0.3 points. Resource utilization can be assigned a weight of 0.1, with ≤20% receiving 1.0 point, 20%~50% receiving 0.7 points, 50%~80% receiving 0.3 points, and >80% receiving 0.1 points.

[0146] 3. Priority level classification: Total score ≥0.8 is P0 level (highest priority), 0.6~0.8 is P1 level, 0.4~0.6 is P2 level, and ≤0.4 is P3 level (lowest priority).

[0147] For example: In an e-commerce scenario, a user-submitted "Payment Order" task (OLTP type, response time requirement ≤100ms, core business, resource utilization rate 15%) has a total score of 0.4×1.0+0.3×1.0+0.2×1.0+0.1×1.0=1.0 points, with a priority of P0; a "Sales Analysis for the Past 3 Months" task (OLAP type, response time requirement ≤5s, ordinary business, resource utilization rate 60%) has a total score of 0.4×0.2+0.3×0.1+0.2×0.3+0.1×0.3=0.19 points, with a priority of P3.

[0148] Resource demand forecasting module:

[0149] Core functionality: Based on historical load data and real-time monitoring data, predict the resource requirements (CPU cores, memory size, IOPS) of OLTP / OLAP tasks in the near future and schedule resources in advance to avoid resource shortages.

[0150] Prediction model design: A hybrid model of "LSTM + sliding window" is adopted to take into account both long-term trends and short-term fluctuations.

[0151] 1. Sliding window: Set a 5-minute sliding window to statistically analyze the task type distribution, average resource utilization, and peak resource demand within the window, capturing short-term load fluctuations.

[0152] 2. LSTM model:

[0153] Input features: 10 time-series features from sliding window statistics (number of OLTP / OAP tasks, average CPU / memory / IO utilization, peak utilization, number of data hotspots) and time features (hour, weekday / weekend, whether it is a holiday).

[0154] Model training: Using historical data from the last 3 months (divided into training set / validation set / test set at 1-hour granularity, ratio 7:2:1), the Adam optimizer was used for training, with the objective function being mean squared error (MSE), and the iteration stopped when MSE ≤ 0.01.

[0155] Predicted output: The number of CPU cores, memory size (GB), and IOPS (read / write) required for OLTP / OLAP tasks in the next 1 minute, 5 minutes, and 10 minutes, respectively.

[0156] Real-time update mechanism: The system receives aggregated data from the load monitoring module every 100ms, updates the sliding window statistics, and triggers the model to re-predict, ensuring the real-time nature of the prediction results.

[0157] For example: During a major sales event, at 10:00, the load monitoring module recorded 500 concurrent OLTP tasks and 75% CPU utilization. Based on historical sales event data, the LSTM model predicted that the concurrent OLTP tasks would increase to 800 between 10:01 and 10:05, requiring an additional 3 CPU cores and 6GB of memory. There are currently 2 OLAP tasks, and it is predicted that 3 more complex queries will be added after 10:05, requiring an additional 4 CPU cores, 10GB of memory, and 500 IOPS of read operations.

[0158] Dynamic scheduling decision module:

[0159] Core functionality: Based on task priority, resource demand forecast results, and current resource status, formulate resource allocation strategies to resolve resource conflicts between OLTP and OLAP.

[0160] Resource pool architecture design:

[0161] 1. Allocation method: The computing resources of the NewSQL cluster are divided into "OLTP dedicated resource pool" and "OLAP shared resource pool". The dedicated resource pool accounts for 60% of the total computing resources (default configuration, supports dynamic adjustment) and the shared resource pool accounts for 40%.

[0162] 2. Resource Pool Rules:

[0163] OLTP tasks prioritize using the dedicated resource pool. When the dedicated resource pool is insufficient, resources from the shared resource pool can be preempted (the preemption limit is 50% of the shared resource pool).

[0164] OLAP tasks use the shared resource pool by default. When the shared resource pool is idle, it can use the idle resources of the dedicated resource pool (idle is defined as resource utilization rate ≤ 30% for 30 consecutive seconds). However, the resources must be released immediately when the OLTP task triggers preemption.

[0165] Scheduling algorithm implementation: It adopts a three-level decision logic of "Hungarian algorithm + priority preemption + graceful degradation".

[0166] 1. First level: Hungarian algorithm for allocating basic resources. Using a bipartite graph of "task resource demand and resource pool supply", the objective function is "maximizing total resource utilization + minimizing task latency", basic resources are allocated to tasks of each priority.

[0167] 2. Second level: Priority preemption mechanism. When a high-priority task (such as P0) has insufficient resources, it can preempt the shared resources occupied by a low-priority task (such as P3). The preemption order is P3→P2→P1, and after preemption, the resources of the low-priority task must not be lower than the minimum guarantee (such as OLAP task minimum guarantee of 1 CPU core and 2GB memory).

[0168] 3. Third Level: Graceful Degradation Mechanism. When preemption still cannot meet the needs of high-priority tasks, or when the resources for low-priority tasks are below the minimum guarantee, a degradation strategy is implemented for OLAP tasks:

[0169] Data sharding: Split large-scale queries (such as scanning 10 million rows) into 10 shards and execute them in batches.

[0170] Accuracy degradation: Non-core OLAP tasks can use sampled data for calculation (e.g., sampling 10% of the data for statistical analysis, with an error ≤5%).

[0171] Delayed execution: Add low-priority OLAP tasks to the waiting queue and trigger execution when resources are available.

[0172] Decision output: Generate resource allocation instructions, including the mapping relationship of "task computing nodes", CPU / memory quota for each task, IO priority, whether to enable the degradation strategy, etc.

[0173] For example: The cluster's total computing resources are 10 CPU cores and 32GB of memory. The dedicated resource pool has 6 CPU cores and 19.2GB of memory, and the shared resource pool has 4 CPU cores and 12.8GB of memory. Currently, a P0-level OLTP task requires 5 CPU cores and 15GB of memory (met by the dedicated resource pool), and a P3-level OLAP task requires 3 CPU cores and 8GB of memory (met by the shared resource pool). It is predicted that 3 new P0-level OLTP tasks will be added in one minute, requiring an additional 4 CPU cores and 8GB of memory.

[0174] 1. Basic allocation: The dedicated resource pool has 1 CPU core and 4.2GB of memory remaining, which cannot meet the requirement of 4 CPU cores.

[0175] 2. Preempting the shared resource pool: An OLAP task occupies 3 CPU cores. According to the rules, it can preempt 2 CPU cores of the shared resource pool (50% limit). At this time, the OLAP task has 1 CPU core and 8GB of memory remaining (meeting the minimum guarantee).

[0176] 3. Still short 1 CPU core: Check if there are any idle resources in the dedicated resource pool (no), trigger graceful degradation of the OLAP task, split one of the OLAP tasks into 2 shards, release 0.5 CPU cores, and at the same time temporarily adjust the idle threshold of the dedicated resource pool to 20%, release 0.5 CPU cores, and finally meet the resource requirements of the new OLTP task.

[0177] Resource execution and feedback module:

[0178] Core functions: Execute scheduling decision instructions, monitor the execution effect in real time, form a closed-loop feedback, and dynamically correct scheduling strategies.

[0179] Resource execution implementation:

[0180] 1. Command Issuance: Resource allocation commands are issued in two ways:

[0181] Containerized deployment scenario: Call the Kubernetes API to adjust the CPU / memory requests and limits of compute node Pods, and bind tasks to specified Pods.

[0182] Physical machine deployment scenario: Call the resource control interface of the NewSQL execution engine to set the resource quotas for tasks (such as CPU affinity and memory limit).

[0183] 2. Smooth Adjustment Mechanism: Resource adjustments employ a "gradient adjustment" strategy to avoid performance fluctuations caused by sudden increases or decreases. For example, when increasing CPU resources, one core is adjusted at a time with an interval of 50ms; when decreasing CPU resources, 0.5 cores are adjusted at a time with an interval of 100ms, ensuring smooth task operation.

[0184] Feedback mechanism implementation:

[0185] 1. Performance monitoring: Collect task execution metrics (OLTP response time, OLAP execution time) and resource utilization every 20ms and compare them with preset thresholds (e.g., OLTP response time ≤ 100ms, OLAP execution time ≤ 5s).

[0186] 2. Decision Correction:

[0187] If the OLTP response time exceeds the threshold of 120%, the dynamic scheduling decision module is immediately triggered to recalculate and prioritize allocating more resources to the OLTP task.

[0188] If the OLAP execution time exceeds the threshold of 150% and there is available space in the shared resource pool, the resource quota for the OLAP task will be increased; if there is no available space, a second degradation will be triggered.

[0189] If the resource utilization rate remains below 30% for 30 consecutive seconds, adjust the resource pool allocation ratio (e.g., reduce the dedicated resource pool to 50% and increase the shared resource pool to 50%).

[0190] For example: The scheduling decision allocates 4 CPU cores and 8GB of memory to an OLTP task. After execution, the feedback module detects that the OLTP response time is 120ms (120% of the threshold of 100ms), and immediately triggers rescheduling, preempting 1 CPU core from the shared resource pool and allocating it to the OLTP task. After 50ms, the response time is detected to have dropped to 80ms, meeting the threshold requirement.

[0191] Storage layer collaboration module:

[0192] Core functionality: Adapts to the storage-compute separation architecture of NewSQL, reduces the consumption of compute layer resources, and improves overall performance through optimized storage layer resource scheduling.

[0193] Collaborative optimization strategy:

[0194] 1. Cache collaboration:

[0195] OLTP task: Load frequently accessed data (such as product inventory tables and user account tables identified by data hotspot indicators) into the local cache (L1 cache) of the compute node. The cache validity period is 1 minute. When updating, a "write-through + invalidation" mechanism is used to ensure data consistency.

[0196] OLAP task: Preload large table data involved in the query into the shared cache (L2 cache). The cache is valid for 30 minutes and adopts the LRU (Least Recently Used) eviction policy, giving priority to retaining cached data from high-priority OLAP tasks.

[0197] 2. IO Collaboration:

[0198] Priority sorting: OLTP tasks are assigned the highest priority for IO, while OLAP tasks are assigned medium to low priority. When IO resources are scarce, OLAP task IO requests are queued for execution.

[0199] Batch scheduling: OLAP task read requests are merged into batch IO operations (e.g., merging 10 small file read requests into 1 batch request) to reduce IO overhead; during OLTP off-peak hours (e.g., 2-4 am), batch data reading of OLAP tasks is triggered to avoid occupying IO resources during peak hours.

[0200] 3. Data Sharding Collaboration: Based on the resource status of compute nodes, data shards for OLAP queries are bound to compute nodes to avoid cross-node data transmission consuming network resources; for data hotspot shards, the number of replicas is automatically increased (up to 3) to distribute access pressure.

[0201] For example, in an OLTP "Inventory Query" task, when accessing inventory data for product ID=10086, the storage layer collaboration module detects that this product is hot data and loads it into the compute node's L1 cache, reducing the IO latency of subsequent queries from 15ms to 2ms. In an OLAP "Regional Sales Analysis" task, which needs to read sales data tables from 10 regions, the collaboration module preloads the data into the L2 cache and merges read requests into batch operations, reducing IOPS consumption from 800 to 300, saving 50% of IO resources.

[0202] In fact, the overall process of this application can be as follows: Figure 2 The diagram shows a division into four parts.

[0203] 1. Task Access Phase: Users submit two types of tasks through the application—payment orders (OLTP) and sales analysis (OLAP). After receiving the tasks, the NewSQL access layer forwards the tasks to the execution engine for execution and forwards the task metadata (SQL statement, submission time, business tags) to the load monitoring module.

[0204] 2. Monitoring and Evaluation Phase: The load monitoring module collects real-time metrics such as the current cluster's OLTP concurrency, CPU utilization, and IOPS; the task classification module parses SQL statements to determine whether payment orders are OLTP or sales analysis is OLAP; the priority evaluation module calculates scores, classifying payment orders as P0 and sales analysis as P3.

[0205] 3. Prediction and Decision-Making Phase: The resource demand prediction module combines historical data and real-time indicators to predict that the number of concurrent OLTP tasks will increase to 800 in 1 minute, requiring additional resources; the dynamic scheduling decision-making module allocates a dedicated resource pool of 5 CPU cores and 15GB memory to P0-level OLTP tasks based on priority and prediction results, and allocates a shared resource pool of 3 CPU cores and 8GB memory to P3-level OLAP tasks; the storage layer collaboration module optimizes data access, loads product data into the L1 cache, and merges OLAP read requests.

[0206] 4. Execution and Feedback Phase: The resource execution module adjusts the resource quota of compute node Pods through the K8s API and executes tasks; the feedback module monitors the task execution effect in real time. If the OLTP response time meets the threshold, the current scheduling is maintained; if the response time exceeds the threshold, rescheduling is triggered to ensure performance stability.

[0207] Therefore, this application firstly collects high-frequency task operation indicators, resource utilization indicators, and data hotspot indicators through a load detection module, covering multiple dimensions such as OLTP transaction throughput, response time, OLAP query complexity, and data scan volume, providing comprehensive data support for scheduling; the task classification and priority evaluation module accurately distinguishes task types based on SQL parsing and business tags, and quantifies priorities through a weighted scoring model to ensure that high-time-sensitive core tasks such as payments are allocated resources first; the resource demand prediction module combines sliding window statistics of short-cycle load fluctuations with LSTM models to capture long-cycle patterns, outputting future minute-level resource demands in real time to achieve forward-looking scheduling; the dynamic scheduling decision module adopts a three-level decision mechanism, allocating basic resources through bipartite graph matching, prioritizing shared pool resources to ensure critical tasks, and triggering data sharding or sampling degradation to address resource gaps, elastically balancing OLTP and OLAP conflicts; the resource execution and feedback module dynamically adjusts resource quotas based on containers or physical interfaces, and corrects strategies through real-time monitoring of response time and execution duration; the storage layer collaboration module optimizes data access paths, accelerates high-frequency OLTP operations with local caching, improves OLAP throughput with shared caching and batch IO, and reduces network overhead with data sharding and binding. In summary, this solution, through a closed-loop scheduling system with multi-module collaboration, ensures low latency for OLTP while meeting the high throughput requirements of OLAP, significantly improving the resource utilization efficiency and transaction processing effectiveness of NewSQL databases.

[0208] This application also provides a NewSQL database hybrid transaction processing dynamic resource scheduling device, comprising:

[0209] The processing module is used to access the target task and forward the metadata corresponding to the target task to the load detection module;

[0210] The processing module is also used to obtain load data of the NewSQL cluster through the load detection module;

[0211] The load data includes task operation metrics, resource utilization metrics, and data hotspot metrics.

[0212] The processing module is further configured to classify the target task into online transaction processing or online analysis processing based on the metadata and the load data through the task classification and priority evaluation module, and calculate the task priority parameter of the target task;

[0213] The processing module is also used to determine the computing resource requirements for the target time period in real time by using the resource demand prediction module based on the sliding window statistics and time series prediction model.

[0214] The processing module is further configured to determine a resource allocation strategy in a dedicated resource pool and / or a shared resource pool based on the load data, the task priority parameters, and the resource requirements through a dynamic scheduling decision module.

[0215] The processing module is further configured to execute the resource allocation strategy and monitor the execution effect of the target task through the resource execution and feedback module, and dynamically adjust the resource allocation strategy based on the execution effect.

[0216] This implementation method achieves multi-dimensional collection of NewSQL cluster load data by accessing the target task and forwarding metadata to the load detection module, including task operation indicators, resource utilization indicators, and data hotspot indicators, thereby comprehensively perceiving the system status. Furthermore, through the task classification and priority evaluation module, the target task is accurately divided into online transaction processing or online analysis processing, and task priority parameters are calculated to provide a basis for resource scheduling decisions. The resource demand prediction module determines the resource demand for the target time period in real time based on sliding window statistics and time series prediction models, achieving forward-looking resource planning. The dynamic scheduling decision module integrates load data, task priorities, and resource demands to generate resource allocation strategies in dedicated and shared resource pools, taking into account the differentiated needs of the two types of tasks. Finally, the resource execution and feedback module executes the strategies and monitors the effects, dynamically correcting resource allocation to form a closed loop of "monitoring-evaluation-prediction-decision-execution-feedback," effectively resolving resource conflicts between low latency in OLTP and high throughput in OLAP, and improving the effectiveness and stability of database transaction operation.

[0217] As an optional implementation, the specific method by which the processing module calculates the task priority parameter of the target task includes:

[0218] Determine the weights of the dimensions of task type, response time requirements, business importance, and resource utilization.

[0219] Determine the dimensional scores for the task type, response time requirement, business importance, and resource utilization rate;

[0220] Specifically, based on the task type, fixed dimension scores are assigned to the online transaction processing and the online analysis processing; based on the response time requirement, score gradients are divided in descending order according to a preset duration interval; based on the business importance, the degree of importance is determined according to a preset business type and a dimension score positively correlated with the degree of importance is assigned; and based on the resource utilization rate, score gradients are divided in descending order according to a preset utilization rate interval.

[0221] This implementation method determines the weights and scores of task type, response time requirements, business importance, and resource utilization. It constructs a multi-dimensional weighted scoring model by assigning fixed scores to task types, descending scores to response times within a range, positively correlated scores to business importance by type, and descending scores to resource utilization within a range. This model quantitatively assesses task priority. It prioritizes resource allocation for high-time-sensitive and high-importance tasks (such as payment transactions), preventing low-priority tasks from crowding out critical resources. This ensures precise matching of resource allocation with business needs, improving the rationality of scheduling decisions and system response efficiency.

[0222] As an optional implementation, the processing module determines the specific method of resource demand for the target time period in real time based on the sliding window statistics and time series prediction model, including:

[0223] Set a fixed-duration sliding window and statistically analyze the multidimensional statistical indicators within the window.

[0224] The multidimensional statistical indicators are used to indicate the task operation status and computing resource usage over a period of time.

[0225] By using a time-series prediction model, the multidimensional statistical indicators and time characteristics are input, and the amount of computing resources required for the online transaction processing and the online analysis processing are output for the first, second and third time periods in the future.

[0226] The time-series prediction model is trained based on historical data corresponding to the multidimensional statistical indicators and the time features within a preset training period.

[0227] This implementation uses a fixed-duration sliding window to statistically analyze historical task performance and resource utilization, capturing short-term load fluctuation patterns. Combined with a time-series prediction model that inputs multi-dimensional statistical indicators and time characteristics, it outputs the resource requirements for online transaction processing and online analytical processing over multiple future time periods, achieving minute-level accurate prediction. This design, through the synergy of the sliding window and the LSTM model, considers both real-time load changes and long-term trends, proactively allocating resources to fill gaps and avoiding OLTP latency spikes or OLAP query timeouts due to resource shortages, thus ensuring stable system performance.

[0228] As an optional implementation, the process of determining the resource allocation strategy by the processing module includes three levels of decision-making;

[0229] The first-level decision-making process uses a bipartite graph matching algorithm to allocate basic computing resources to each of the target tasks.

[0230] The second-level decision triggers a preemption mechanism based on the task priority parameters of each target task. For high-priority tasks whose task priority parameters exceed a preset threshold, if computing resources are insufficient, the shared resource pool resources of low-priority tasks will be occupied according to the reverse order of the task priority parameters, without exceeding the preset minimum resource guarantee ratio.

[0231] The third-level decision is used to perform a degradation operation on a first preset number of online analysis processes when there is still a shortage of computing resources after the high-priority tasks have completed their resource occupation or when the computing resources of the low-priority tasks are lower than the minimum resource guarantee ratio. The degradation operation includes data sharding, data sampling, or delayed execution.

[0232] This implementation employs a three-tiered decision-making mechanism to determine resource allocation strategies: The first tier allocates basic resources to tasks using a bipartite graph matching algorithm, maximizing resource utilization and minimizing latency; the second tier triggers a preemption mechanism in reverse order of task priority, ensuring resource needs for high-priority tasks while reserving a minimum resource guarantee ratio for low-priority tasks; the third tier performs degradation operations such as data sharding, data sampling, or delayed execution on online analytical processing to address resource shortages. This tiered strategy, through the synergy of basic allocation, elastic preemption, and graceful degradation, prioritizes critical tasks during resource-scarce periods while preventing low-priority tasks from being completely starved, achieving a dynamic balance of resource conflicts.

[0233] As an optional implementation, the specific method by which the processing module dynamically corrects the resource allocation strategy includes:

[0234] Obtain the response time of the online transaction processing and the execution time of the online analysis processing;

[0235] When the response time exceeds the first time threshold, the dynamic scheduling decision module is triggered to recalculate the resource allocation strategy and increase the computing resources for the online transaction processing.

[0236] When the execution time exceeds the second time threshold and there are idle computing resources in the shared resource pool, the computing resources for the online analysis and processing are increased; if there are no idle computing resources in the shared resource pool, the degradation operation is performed on the second preset number of online analysis and processing operations.

[0237] Furthermore, when the resource utilization rate parameter is continuously lower than the preset utilization rate ratio, the ratio between the dedicated resource pool and the shared resource pool is adjusted.

[0238] This implementation dynamically adjusts resource allocation strategies by acquiring real-time response times for online transaction processing and execution times for online analysis processing: when the response time exceeds a threshold, the strategy is recalculated and online transaction processing resources are increased; when the execution time exceeds a threshold and there are idle resources in the shared pool, online analysis processing resources are increased; otherwise, a degradation operation is triggered; when resource utilization remains consistently low, the ratio of the dedicated pool to the shared pool is adjusted. This feedback mechanism optimizes scheduling strategies in real-time based on task execution performance, avoiding performance fluctuations, ensuring that resource allocation always matches actual load requirements, and improving system stability and resource utilization efficiency.

[0239] As an optional implementation, the processing module is further configured to perform storage layer collaborative optimization through the storage layer collaborative module, specifically including:

[0240] Cache optimization involves loading frequently accessed transaction processing data into the local cache of the compute node and dynamically retaining it for the first cache time; large table data is preloaded into the shared cache, and a least recently used eviction policy is adopted to update the data with the second cache time.

[0241] Data interaction and collaboration are carried out by allocating high priority to the online transaction processing and low priority to the online analysis processing, and merging the data read requests of the online analysis processing into batch data interactions, and executing the batch data interactions according to the load data.

[0242] Data sharding collaboration involves binding the data shards corresponding to the online analysis and processing to the target computing node based on the resource status of each computing node, and generating shard replicas in multiple computing nodes based on the load data.

[0243] This implementation optimizes data access through a storage layer collaboration module: high-frequency transaction processing data is loaded into the local cache of compute nodes and dynamically retained for expiration, accelerating OLTP access; large table data is preloaded into a shared cache and updated using an LRU strategy, improving OLAP query efficiency; high-priority data interactions are allocated to online transaction processing, and read requests for online analytical processing are merged into batch interactions, reducing IO overhead; data shards are bound to compute node status and replicas are generated to distribute the pressure of hotspot access. This design, through the collaborative optimization of caching, IO, and data sharding, reduces cross-node data transmission, lowers compute layer resource consumption, and improves the overall execution efficiency of HTAP tasks.

[0244] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, a processing module can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as program code in the device's memory, and its functions can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0245] Indicatively, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 3 The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as application programs, that can be executed by the processing component 302. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the methods of any of the embodiments described above.

[0246] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, MacOSX™, Unix™, Linux™, FreeBSD™, or similar.

[0247] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0248] This application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the method provided in any embodiment.

[0249] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0250] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0251] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dynamic resource scheduling method for hybrid transaction processing in NewSQL databases, characterized in that, include: The target task is accessed, and the metadata corresponding to the target task is forwarded to the load detection module; The load detection module is used to obtain load data of the NewSQL cluster. The load data includes task operation metrics, resource utilization metrics, and data hotspot metrics. The task classification and priority evaluation module classifies the target task into online transaction processing or online analysis processing based on the metadata and the load data, and calculates the task priority parameter of the target task. The resource demand prediction module determines the computing resource demand for the target period in real time based on the sliding window statistics and time series prediction model. The dynamic scheduling decision module determines a resource allocation strategy in a dedicated resource pool and / or a shared resource pool based on the load data, the task priority parameters, and the resource requirements. The resource allocation strategy is executed through the resource execution and feedback module, and the execution effect of the target task is monitored. The resource allocation strategy is then dynamically adjusted based on the execution effect.

2. The method according to claim 1, characterized in that, The calculation of the task priority parameter of the target task includes: Determine the weights of the dimensions of task type, response time requirements, business importance, and resource utilization. Determine the dimensional scores for the task type, response time requirement, business importance, and resource utilization rate; Specifically, based on the task type, fixed dimension scores are assigned to the online transaction processing and the online analysis processing; based on the response time requirement, score gradients are divided in descending order according to a preset duration interval; based on the business importance, the degree of importance is determined according to a preset business type and a dimension score positively correlated with the degree of importance is assigned; and based on the resource utilization rate, score gradients are divided in descending order according to a preset utilization rate interval.

3. The method according to claim 1, characterized in that, The process of determining resource requirements for a target time period in real time based on a sliding window statistical and time-series prediction model includes: Set a fixed-duration sliding window and statistically analyze the multidimensional statistical indicators within the window; The multidimensional statistical indicators are used to indicate the task operation status and computing resource usage over a period of time. By using a time-series prediction model, the multidimensional statistical indicators and time characteristics are input, and the amount of computing resources required for the online transaction processing and the online analysis processing are output for the first, second and third time periods in the future. The time-series prediction model is trained based on historical data corresponding to the multidimensional statistical indicators and the time features within a preset training period.

4. The method according to claim 1, characterized in that, The process of determining the resource allocation strategy includes three levels of decision-making; The first-level decision-making process uses a bipartite graph matching algorithm to allocate basic computing resources to each of the target tasks. The second-level decision triggers a preemption mechanism based on the task priority parameters of each target task. For high-priority tasks whose task priority parameters exceed a preset threshold, if computing resources are insufficient, the shared resource pool resources of low-priority tasks will be occupied according to the reverse order of the task priority parameters, without exceeding the preset minimum resource guarantee ratio. The third-level decision is used to perform a degradation operation on a first preset number of online analysis processes when there is still a shortage of computing resources after the high-priority tasks have completed their resource occupation or when the computing resources of the low-priority tasks are lower than the minimum resource guarantee ratio. The degradation operation includes data sharding, data sampling, or delayed execution.

5. The method according to claim 4, characterized in that, The dynamic adjustment methods for the resource allocation strategy include: Obtain the response time of the online transaction processing and the execution time of the online analysis processing; When the response time exceeds the first time threshold, the dynamic scheduling decision module is triggered to recalculate the resource allocation strategy and increase the computing resources for the online transaction processing. When the execution time exceeds the second time threshold and there are idle computing resources in the shared resource pool, the computing resources for the online analysis and processing are increased; if there are no idle computing resources in the shared resource pool, the degradation operation is performed on the second preset number of online analysis and processing operations. Furthermore, when the resource utilization rate parameter is continuously lower than the preset utilization rate ratio, the ratio between the dedicated resource pool and the shared resource pool is adjusted.

6. The method according to any one of claims 1-5, characterized in that, The method also includes storage layer collaborative optimization through a storage layer collaborative module, specifically including: Cache optimization involves loading frequently accessed transaction processing data into the local cache of the compute node and dynamically retaining it for the first cache time; large table data is preloaded into the shared cache, and a least recently used eviction policy is adopted to update the data with the second cache time. Data interaction and collaboration are carried out by allocating high priority to the online transaction processing and low priority to the online analysis processing, and merging the data read requests of the online analysis processing into batch data interactions, and executing the batch data interactions according to the load data. Data sharding collaboration involves binding the data shards corresponding to the online analysis and processing to the target computing node based on the resource status of each computing node, and generating shard replicas in multiple computing nodes based on the load data.

7. A dynamic resource scheduling device for hybrid transaction processing of NewSQL databases, characterized in that, include: The processing module is used to access the target task and forward the metadata corresponding to the target task to the load detection module; The processing module is also used to obtain load data of the NewSQL cluster through the load detection module; The load data includes task operation metrics, resource utilization metrics, and data hotspot metrics. The processing module is further configured to classify the target task into online transaction processing or online analysis processing based on the metadata and the load data through the task classification and priority evaluation module, and calculate the task priority parameter of the target task; The processing module is also used to determine the computing resource requirements for the target time period in real time by using the resource demand prediction module based on the sliding window statistics and time series prediction model. The processing module is further configured to determine a resource allocation strategy in a dedicated resource pool and / or a shared resource pool based on the load data, the task priority parameters, and the resource requirements through a dynamic scheduling decision module. The processing module is further configured to execute the resource allocation strategy and monitor the execution effect of the target task through the resource execution and feedback module, and dynamically adjust the resource allocation strategy based on the execution effect.

8. The apparatus according to claim 7, characterized in that, The specific method by which the processing module calculates the task priority parameter of the target task includes: Determine the weights of the dimensions of task type, response time requirements, business importance, and resource utilization. Determine the dimensional scores for the task type, response time requirement, business importance, and resource utilization rate; Specifically, based on the task type, fixed dimension scores are assigned to the online transaction processing and the online analysis processing; based on the response time requirement, score gradients are divided in descending order according to a preset duration interval; based on the business importance, the degree of importance is determined according to a preset business type and a dimension score positively correlated with the degree of importance is assigned; and based on the resource utilization rate, score gradients are divided in descending order according to a preset utilization rate interval.

9. A computer device, characterized in that, The method includes one or more processors and a memory storing computer-readable instructions that, when executed by the one or more processors, perform the steps of the method as described in any one of claims 1-6.

10. A storage medium, characterized in that, The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method as described in any one of claims 1-6.

Images