Big model-based cloud database multi-node resource scheduling optimization method and system

Abstract

This invention relates to the field of resource scheduling technology, specifically to a method and system for optimizing multi-node resource scheduling in cloud databases based on a large-scale model. The invention obtains a non-computational blocking index based on the hardware resource shortage level and processor effective execution status of computing nodes at the current sampling time. If preset abnormal conditions are met, a resource conflict score is obtained based on the matching degree between the resource consumption tendency characteristics of database services and hardware pressure indicators in the current period, as well as the load activity intensity. The target interfering service and its primary congested resource for the current period are selected. Based on the resource conflict score of the target interfering service, the cumulative congestion level of its associated database service node at the current sampling time is updated. Differential scheduling is performed on the database service node to which the target interfering service belongs based on the physical attributes of the primary congested resource. This solution eliminates implicit contention for underlying physical resources by perceiving hidden congestion, accurately attributing interference sources, and implementing differentiated governance.

Description

Technical Field

[0001] This invention relates to the field of resource scheduling technology, specifically to a method and system for optimizing multi-node resource scheduling in cloud databases based on a large model. Background Technology

[0002] In the field of cloud computing database services, to improve hardware resource utilization, multiple database service nodes are typically deployed in a hybrid manner on the same physical node. Current practices mainly rely on the operating system's control group to quota and limit the processor time and memory capacity of containers, and combine this with central processing unit (CPU) utilization monitoring to achieve load balancing.

[0003] However, in multi-tenant hybrid deployment scenarios of cloud data, especially in gray-box operation and maintenance scenarios where monitoring agents cannot know the underlying resource allocation logic of the database kernel, resource contention is not only reflected in CPU usage, but also in the competition for shared channels such as memory bandwidth and disk I / O. For example, when an instance performs non-CPU-intensive tasks such as full table scans or external sorting, although the CPU utilization is not high, it will frequently refresh the cache and occupy memory bandwidth or I / O channels, causing other instances on the same node to experience response delays due to waiting for underlying resources.

[0004] The existing patent document CN119201474A discloses a Spark offline task resource scheduling optimization method based on the input data volume. Specifically, it allocates corresponding computing resources to the task by statistically analyzing the number of rows in the task's input data table and matching them with preset resource rules. While this method optimizes resource allocation to some extent, its attribution logic remains at the single dimension of "more data volume means more resources are needed," which has significant drawbacks: First, estimating explicit resource requirements based on the static number of data rows cannot reflect dynamically generated memory bandwidth pressure or I / O queue depth during operation, making it difficult to perceive the actual hardware downtime. Second, the static scheduling based on preset rules cannot adapt to database gray-box operation and maintenance scenarios. When multiple instances share a channel and compete for resources, relying solely on the input data volume cannot distinguish between "high-load but harmless" business and "low-load but strong interference" hidden interference sources, leading to scheduling misjudgments or system instability, thereby restricting the overall performance of the cluster. Summary of the Invention

[0005] To address the technical challenge of accurately locating hidden resource interference sources causing contention for shared hardware channels in database gray-box operation and maintenance scenarios, this invention aims to provide a cloud database multi-node resource scheduling optimization method and system based on a large model. The specific technical solution adopted is as follows: In a first aspect, one embodiment of the present invention provides a cloud database multi-node resource scheduling optimization method based on a large model, the method comprising: Obtain hardware stress indicators for different resource types of computing nodes at the current sampling time; Based on the hardware resource scarcity and processor execution status of the computing node at the current sampling time, obtain the non-computational blocking index of the computing node at the current sampling time; The time interval between the current sampling time and the previous sampling time is recorded as the current period; if the non-computational blocking index meets the preset abnormal conditions, the resource conflict score of each database service in the current period is obtained based on the matching degree between the resource consumption tendency characteristics and hardware pressure indicators of each database service in the current period, as well as the load activity intensity of the corresponding service in the current period; and the target interfering service and its corresponding primary congested resource in the database services in the current period are selected. Based on the resource conflict score of the target interfering service, update the cumulative congestion level of its database service node at the current sampling time; when the cumulative congestion level meets the scheduling triggering condition, perform differentiated scheduling on the database service node to which the target interfering service belongs based on the physical attributes of the primary congested resource.

[0006] Further, obtaining the non-computational blocking index of the computing node at the current sampling time includes: The hardware pressure indicators of all resource types of the computing node at the current sampling time are normalized respectively, and the normalization results are arranged to obtain the hardware pressure vector. Obtain the processor time increment of the computing node in the current cycle, and use the ratio of the processor time increment to the total clock cycles of the current cycle as the effective execution ratio of the computing node in the current cycle. The non-computational blocking index of the computing node at the current sampling time is obtained based on the maximum component in the hardware pressure vector and the effective execution ratio of the processor; the effective execution ratio of the processor is negatively correlated with the non-computational blocking index.

[0007] Furthermore, obtaining the resource conflict score for each database service within the current period includes: Extract the SQL text of each database service in the current period and perform anonymization processing. Input the processed SQL text into a preset resource consumption feature model to obtain the initial consumption tendency ratio of each resource type. Arrange the L1 normalized results of the initial consumption tendency ratios of all resource types to obtain the resource consumption feature vector of each database service in the current period. Obtain the row scan rate and connection activity ratio of each database service in the current period; normalize the row scan rate to obtain the throughput pressure assessment value; calculate the product of the throughput pressure assessment value and the connection activity ratio, and select the minimum value between the product and the constant 1 as the load activity intensity coefficient of each database service in the current period. The hardware pressure vector is multiplied by the resource consumption feature vector of each database service in the current period. The result of the multiplication is weighted by the load activity intensity coefficient of the corresponding database service to obtain the resource conflict score of each database service in the current period.

[0008] Furthermore, the selection of target interfering services and their corresponding primary congested resources in the database services within the current period includes: The database service with the highest resource conflict score among all database services in the current period is selected and designated as the target interfering service. Calculate the product of the resource consumption feature vector of the target interference service and the component of the same resource type in the hardware pressure vector at the current sampling time, and select the resource type corresponding to the largest product as the primary congestion resource of the target interference service in the current period.

[0009] Further, updating the cumulative congestion level of the database service node to which the target interfering service belongs at the current sampling time based on the resource conflict score of the target interfering service includes: Set the initial value of the cumulative congestion level of each database service node on the compute node to zero; The database service node containing the target interference service in the current period is recorded as the target database node in the current period. The resource conflict score of the target interference service is added to the cumulative congestion of the target database node in the previous sampling time at the current sampling time to obtain the cumulative congestion of the target database node at the current sampling time. If the compute node does not meet the preset abnormal conditions at the current sampling time, the cumulative congestion of all database service nodes on the compute node at the previous sampling time at the current sampling time will be multiplied by the preset attenuation coefficient, and the product will be used as the cumulative congestion of the corresponding database service node at the current sampling time.

[0010] Furthermore, the differentiated scheduling of the database service node to which the target interference service belongs includes: The resource types include two categories: incompressible resources and compressible resources; If the primary congested resource is an incompressible resource, then a rate limiting operation is performed on the database service node to which the target interfering service belongs, and the node is marked as pending scheduling to perform an asynchronous migration operation; If the primary congested resource is a compressible resource, then a quota compression operation is performed on the database service node to which the target interfering service belongs: the quota compression coefficient is determined based on the resource conflict score of the target interfering service; the product of the preset baseline calculated quota of the database service node to which the target interfering service belongs and the quota compression coefficient is used as the resource quota limit value of the database service node, and written into the operating system kernel control group file of the computing node.

[0011] Furthermore, after performing differentiated scheduling, it also includes: If the database service node to which the target interfering service belongs performs a migration operation, then the post-verification operation on the target interfering service will be stopped until the migration is completed and then restored. If the database service node to which the target interference service belongs performs rate limiting or quota compression, then a post-verification operation is performed on the target interference service. The post-validation operation includes: obtaining the non-computational congestion index of the computing node at the next sampling time from the current sampling time; when the non-computational congestion index at the next sampling time meets the preset abnormal conditions, canceling the rate limiting or quota compression operation performed on the database service node to which the target interference service belongs; multiplying the component of the primary congested resource in the resource consumption feature vector of the target interference service by a preset attenuation correction factor and re-normalizing it to obtain the updated resource consumption feature vector; and excluding the target interference service of the current period from being selected as the new target interference service within the preset exemption period from the current sampling time.

[0012] Furthermore, the preset abnormal condition is that the non-computational blocking index of the computing node at the current sampling time is greater than a preset blocking threshold.

[0013] Furthermore, the hardware pressure vector and the resource consumption feature vector have the same resource type corresponding to the same dimension component.

[0014] Secondly, another embodiment of the present invention provides a cloud database multi-node resource scheduling and optimization system based on a large model, the system comprising: The data acquisition module is used to obtain hardware stress indicators of different resource types of computing nodes at the current sampling time; The non-computational blocking assessment module is used to obtain the non-computational blocking index of the computing node at the current sampling time based on the degree of hardware resource shortage and the effective execution state of the processor at the current sampling time. The resource conflict attribution module is used to record the time period between the current sampling time and the previous sampling time as the current period; if the non-computational blocking index meets the preset abnormal conditions, then based on the matching degree between the resource consumption tendency characteristics of each database service and the hardware pressure index in the current period, as well as the load activity intensity of the corresponding service in the current period, the resource conflict score of each database service in the current period is obtained; and the target interfering service and its corresponding primary congested resource in the database services in the current period are selected. The resource scheduling optimization module is used to update the cumulative congestion level of the database service node to which the target interfering service belongs at the current sampling time based on the resource conflict score of the target interfering service; when the cumulative congestion level meets the scheduling triggering condition, differential scheduling is performed on the database service node to which the target interfering service belongs based on the physical attributes of the primary congested resource.

[0015] The present invention has the following beneficial effects: In this embodiment of the invention, a non-computational blocking index is constructed by combining the hardware resource scarcity of computing nodes with the effective execution status of processors. This effectively captures implicit resource contention scenarios where processors are idle but underlying hardware channels are saturated, achieving high-sensitivity perception of bottlenecks in underlying shared hardware channels. It not only considers the intensity of service load activity but also introduces the matching degree between service resource consumption tendencies and current hardware pressure indicators to calculate resource conflict scores. This effectively filters high-load services unrelated to current hardware bottlenecks, thereby accurately locating the implicit interference source causing shared hardware channel contention—the target interfering service and its primary congested resource—in a gray-box environment. This enables refined positioning of shared hardware channel contention sources in gray-box operation and maintenance scenarios. By introducing cumulative congestion, misscheduling caused by instantaneous monitoring fluctuations is avoided. Differentiated scheduling is performed based on the physical attributes of the primary congested resource, achieving precise intervention based on bottleneck attributes. This quickly eliminates implicit contention of underlying physical resources at the root cause while minimizing intervention costs, enhancing resource determinism and service stability in multi-node environments. Attached Figure Description

[0016] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 The flowchart illustrates the steps of a cloud database multi-node resource scheduling optimization method based on a large model, as provided in one embodiment of the present invention. Figure 2A flowchart illustrating a method for obtaining resource conflict scores according to an embodiment of the present invention; Figure 3 This is a system architecture diagram of a cloud database multi-node resource scheduling optimization system based on a large model, provided as an embodiment of the present invention. Detailed Implementation

[0018] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a cloud database multi-node resource scheduling optimization method and system based on a large model proposed by the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0020] The following description, in conjunction with the accompanying drawings, details the specific scheme of the cloud database multi-node resource scheduling optimization method and system based on a large model provided by this invention.

[0021] Example 1: This invention proposes a multi-node resource scheduling optimization method for cloud databases based on a large model. Please refer to [link / reference]. Figure 1 The diagram illustrates a flowchart of a cloud database multi-node resource scheduling optimization method based on a large model, according to an embodiment of the present invention. The method includes: Step S1: Obtain the hardware stress indicators of different resource types for the computing node at the current sampling time.

[0022] A compute node refers to a physical server or virtualized host in a cloud database cluster that hosts multiple database tenant containers, providing shared processor, memory, and disk I / O hardware resources for database operations. The monitoring period is defined as the duration between the moment the compute node's operating system starts and completes the initialization of the monitoring agent and the current sampling moment. Within the monitoring period, the system triggers data collection every 5 seconds, defining a sampling moment, and the time interval between any two adjacent sampling moments is considered a sampling period. Implementers can set the duration of the sampling period according to the specific system load.

[0023] At each sampling moment, the monitoring agent accesses the stress pause information interface of the compute node's operating system kernel to read the cumulative pause time of each resource type at that sampling moment. The cumulative pause time specifically uses the "some" metric value from the stress pause information interface, representing the cumulative duration for which at least one task within the compute node is paused while waiting for a corresponding resource. The difference between the cumulative pause time of the same resource type at each sampling moment and the previous sampling moment is used as the hardware stress index for each resource type at each sampling moment, quantifying the actual blocking degree of each shared hardware channel between the two sampling moments. In this embodiment of the invention, resource types include processors, memory, and disk I / O.

[0024] Step S2: Based on the hardware resource shortage level and processor effective execution status of the computing node at the current sampling time, obtain the non-computational blocking index of the computing node at the current sampling time.

[0025] In multi-tenant hybrid deployment scenarios of cloud databases, system performance degradation is not always caused by the exhaustion of computing resources. It often manifests in a hidden way, with a low actual processor execution rate and a high degree of stagnation in the underlying hardware bus or I / O channels. By combining the degree of hardware resource scarcity with the effective execution status of the processor, and analyzing the non-computational blocking index, we can accurately separate the hidden resource contention scenarios where the processor is idle but the system is sluggish from regular high-computational loads. This solves the technical bottleneck of traditional single CPU indicators being unable to capture system sluggishness caused by non-computational intensive tasks in gray-box monitoring scenarios, and achieves high-sensitivity perception of bottlenecks in the underlying shared hardware channels.

[0026] Step S3: Record the time period between the current sampling time and the previous sampling time as the current period; if the non-computational blocking index meets the preset abnormal conditions, then obtain the resource conflict score of each database service in the current period based on the matching degree between the resource consumption tendency characteristics and hardware pressure indicators of each database service in the current period, as well as the load activity intensity of the corresponding service in the current period; and select the target interfering service and its corresponding primary congested resource in the database services in the current period.

[0027] In a multi-tenant gray-box operation and maintenance environment, simply selecting interference sources based on load activity intensity has limitations. For example, high-throughput, purely compute-intensive services may not be crowding out currently congested I / O or memory channels. By combining the resource consumption tendency characteristics of each database service within the current period with the degree of matching between hardware stress indicators, and analyzing resource conflict scores from two aspects—resource consumption direction and physical load intensity—it is possible to filter out services with high loads but unrelated to the current hardware bottleneck. This allows for precise identification of the real target interference service and its primary congested resources in the gray-box environment, significantly reducing the attribution misjudgment rate and achieving refined positioning of contributors to the underlying hardware bottleneck.

[0028] In this embodiment of the invention, database services within the current period refer to SQL requests initiated and executed by database service nodes on computing nodes within the current period, which are active or consume resources within the current period. A database service node is a logical functional entity that runs independently on a computing node and provides database computing capabilities or storage services. Each database service node may concurrently execute multiple SQL requests within the current period based on the requests of its tenants.

[0029] Step S4: Based on the resource conflict score of the target interfering service, update the cumulative congestion level of its database service node at the current sampling time; when the cumulative congestion level meets the scheduling triggering condition, perform differentiated scheduling on the database service node to which the target interfering service belongs based on the physical attributes of the primary congested resource.

[0030] Determining cumulative congestion by scoring resource conflicts of the target interfering service effectively filters out sporadic resource fluctuations, ensuring that governance actions are triggered only for persistent sources of congestion. When the cumulative congestion meets the scheduling triggering conditions, it indicates that the target interfering service's interference with underlying hardware resources has become highly persistent, and differentiated scheduling is performed based on the physical attributes of the primary congested resource. This strategy, based on integral feedback and divide-and-conquer governance, ensures both the system's tolerance to sudden fluctuations and the ability to quickly and accurately eliminate implicit contention for underlying physical resources with minimal scheduling costs, significantly improving the service stability of multi-tenant clusters.

[0031] Preferably, in some possible implementations of this invention, the method for obtaining the non-computational blocking index includes: normalizing the hardware pressure indicators of all resource types of the computing node at the current sampling time, and arranging the normalization results to obtain a hardware pressure vector; obtaining the processor time increment of the computing node in the current cycle, and using the ratio of the processor time increment to the total clock cycles of the current cycle as the effective execution ratio of the computing node in the current cycle; and obtaining the non-computational blocking index of the computing node at the current sampling time based on the maximum component in the hardware pressure vector and the effective execution ratio of the processor. It should be noted that if the system is in an idle state, all components of the hardware pressure vector are set to zero and the subsequent scheduling process is skipped.

[0032] It should be noted that in this embodiment of the invention, the minimax normalization method is used to normalize the hardware stress indicators for all resource types. The closer a component in the hardware stress vector is to 1, the higher the shortage of the corresponding resource type, and the more severe the pause caused by the task waiting for that resource. In this embodiment of the invention, the monitoring agent periodically reads the operating system kernel's statistics file (such as / proc / stat) to obtain the cumulative execution time of the processor at each sampling moment since system startup. The difference between the cumulative execution time at the current sampling moment and the previous sampling moment is used as the processor time increment for the current period, representing the duration of the processor's busy state in the current period. The product of the total number of logical processor cores of the computing node and the duration of the current period is obtained as the total clock cycle, representing the maximum total processing time that the computing node can theoretically provide. The effective execution ratio of the processor characterizes the proportion of time that the processor actually uses for business operations in the current period. In the multi-tenant mixed load scenario of cloud databases, the system performance degradation is often manifested as a low processor execution ratio and a high degree of hardware pause, usually caused by the saturation of non-computing resources such as memory bandwidth or disk I / O, resulting in the processor being in an ineffective waiting state. Therefore, the maximum component in the hardware pressure vector is positively correlated with the non-computational blocking index, while the processor effective execution ratio is negatively correlated with the non-computational blocking index. This can accurately separate the implicit resource contention scenario where the processor is idle but the system is sluggish from the regular high computational load.

[0033] In this embodiment of the invention, the ratio obtained by taking the maximum component of the hardware pressure vector of the computing node at the current sampling moment as the numerator and the sum of the processor effective execution ratio of the current cycle and a preset minimum positive number as the denominator is used as the non-computational blocking index of the computing node at the current sampling moment. The preset minimum positive number is used to prevent the denominator from being zero, which would render the fraction meaningless; in this embodiment, it is set to... Implementers can configure this according to their specific circumstances. The non-computational congestion index reflects the degree of hardware ineffective pauses in computing nodes caused by non-computational resource bottlenecks; the larger the value, the more severe the non-computational congestion caused by contention for shared channels such as memory bandwidth or disk I / O.

[0034] In this embodiment of the invention, the preset abnormal condition includes: the non-computational blocking index is greater than a preset blocking threshold. If the non-computational blocking index does not meet the preset abnormal condition, the computing node is in normal operation or under processor-intensive conventional computing load at the current sampling time, and the subsequent scheduling process is not triggered. The monitoring agent program continues to monitor in real time, collects data at the next sampling time, and obtains the non-computational blocking index. If the non-computational blocking index meets the preset abnormal condition, the system falls into a hidden blocking state caused by non-computational intensive tasks, and it is necessary to locate the interference source causing the system lag.

[0035] In this embodiment of the invention, the method for obtaining the preset blocking threshold is as follows: based on the monitoring logs of the computing node during the historical benchmark operation, a first sample sequence under normal computing-intensive load conditions and a second sample sequence under abnormal conditions confirmed to be non-computing resource-constrained (such as disk I / O queuing or memory bandwidth saturation) are extracted respectively; the statistical upper limit value (e.g., 95th percentile) of the first sample sequence and the statistical lower limit value (e.g., 5th percentile) of the second sample sequence are obtained respectively, and the midpoint of the numerical range between the statistical upper limit value and the statistical lower limit value is selected as the preset blocking threshold; preferably, it is set to 3.

[0036] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining the resource conflict score is described in [reference needed]. Figure 2 The diagram illustrates a flowchart of a method for obtaining resource conflict scores according to an embodiment of the present invention, the method comprising: Step S310: Extract the SQL text of each database business in the current period and perform de-identification processing, and input the processed SQL text into the preset resource consumption feature model to obtain the initial consumption tendency ratio of each resource type; arrange the L1 normalized results of the initial consumption tendency ratios of all resource types to obtain the resource consumption feature vector of each database business in the current period.

[0037] It should be noted that the monitoring agent periodically accesses the real-time performance dynamic view of the database management system, captures active SQL text within the current period, and uses a parameterized desensitization mechanism to replace constant values ​​and user-sensitive information with generic placeholders. While ensuring business data privacy, it extracts desensitized SQL templates that retain the core execution logic and operator structure. A unique signature of the desensitized SQL template is determined, and the feature cache table maintained locally on the compute node is retrieved. If a record matching the unique signature exists in the feature cache table, the corresponding initial consumption tendency ratio is directly extracted. If no matching record exists, the desensitized SQL template is asynchronously input into the resource consumption feature model to obtain the initial consumption tendency ratio for each resource type, and this is updated in the feature cache table. The feature cache table is a key-value mapping structure stored in the local memory of the compute node. The key is a unique identifier generated by hashing the SQL text, and the value is the resource consumption feature vector of that SQL text. The resource consumption feature model is a weighted prediction model built based on an expert knowledge operator library. During the offline phase, it is pre-trained by learning the mapping rules between massive database operators and underlying hardware resource consumption. This model combines logical operators in the anonymized SQL template (e.g., mapping full table scans to high I / O tendency, hash joins to high memory bandwidth tendency) to output the initial consumption tendency ratio, representing the proportion of each resource type consumed by the SQL. For SQL that has not yet been parsed or appears for the first time, the initial consumption tendency ratio of each resource type is temporarily set to... Where N is the total number of resource types, ensuring uninterrupted monitoring; after the model completes inference, the initial consumption tendency is updated using the inference results. The resource consumption feature vector represents the consumption weight of database services on different hardware channels.

[0038] Step S320: Obtain the row scan rate and connection activity ratio of each database service in the current period; normalize the row scan rate to obtain the throughput pressure assessment value; calculate the product of the throughput pressure assessment value and the connection activity ratio, and select the minimum value between the product and the constant 1 as the load activity intensity coefficient of each database service in the current period.

[0039] It should be noted that, in this embodiment of the invention, the monitoring agent obtains the total number of rows read by each database service at each sampling time by periodically polling the real-time execution statistics view of the database. The ratio of the difference between the total number of rows read by each database service at the current sampling time and the previous sampling time to the duration of the current period is used as the row scan rate of the corresponding service in the current period, reflecting the throughput intensity of the service on the system I / O and memory channel data throughput.

[0040] The monitoring agent utilizes the database session performance view to calculate the total increase in the duration of each database service in the running or resource-waiting state within the current period, recorded as the cumulative execution time. The ratio of the cumulative execution time to the total clock cycles of the current period is used as the connection activity ratio of that database service, quantifying the degree to which the service occupies concurrent execution slots on the computing nodes. The connection activity ratio is the ratio of the average number of active connections for the corresponding database service in the current period to the preset maximum connection pool limit, reflecting the instantaneous concurrent load intensity of the database service in the current period. The combined increase of both leads to a significant increase in the consumption of underlying physical resources by the database service, thus increasing the intensity of physical workload. Therefore, both the row scan rate and the connection activity ratio are positively correlated with the load activity intensity coefficient.

[0041] In one specific implementation of this invention, the load activity intensity coefficient is expressed by the formula:

[0042] In the formula, This represents the load activity intensity coefficient of the k-th database service within the current period. This represents the row scan rate for the k-th database transaction within the current period. This represents the percentage of connection activity for the k-th database service within the current period. is the preset saturation constant; ln is the logarithmic function with the natural constant as the base; min is the minimum value function; This represents the throughput pressure assessment value for the k-th database service within the current period. It should be noted that... The differences in scanning rates across different orders of magnitude are smoothed to ensure that low-rate fluctuations are not ignored and to limit the excessive dominance of high-rate scans. It can identify heavy-load tasks that involve large-scale data movement and occupy execution threads for extended periods. The minimum function prevents calculation coefficient overflow caused by jumps in underlying performance counters or extremely high-frequency scanning.

[0043] In this embodiment of the invention, a preset saturation constant is used. Calibration is performed based on hardware performance benchmark tests and historical traffic characteristics of compute nodes: High-concurrency full table scan tasks are simulated using load testing tools, and the peak average number of rows scanned per second for each core is recorded when the disk I / O or memory bandwidth reaches the hardware saturation threshold; this peak value is used as a preset saturation constant; preferably set to... .

[0044] Step S330: Perform a dot product operation on the hardware pressure vector and the resource consumption feature vector of each database service in the current period. Use the load activity intensity coefficient of the corresponding database service to weight the dot product operation result to obtain the resource conflict score of each database service in the current period.

[0045] In one specific implementation of this invention, the resource conflict scoring is expressed by the formula:

[0046] In the formula, Score the resource conflict for the k-th database service in the current period; This represents the load activity intensity coefficient of the k-th database service within the current period. For the computing node at the current sampling time The hardware pressure vector; This represents the resource consumption feature vector for the k-th database service within the current period. It should be noted that... This value quantifies the degree of match between the resource shortage status of computing nodes (i.e., hardware stress indicators) and the resource consumption tendency of database services. A higher value indicates a greater correlation between the resource consumption characteristics of the database services and the hardware bottleneck status of the computing nodes on the same dimension. It also utilizes a load activity intensity coefficient. Weighting to improve resource conflict scoring The higher the level of business, the more it overlaps with the system bottleneck in terms of consumption direction and actually applies extremely high physical load. As a result, the kth database business is more likely to cause implicit resource congestion on the computing node.

[0047] It is important to note that the hardware stress vector and the resource consumption feature vector have the same resource type corresponding to the same dimension component.

[0048] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining the target interference service and its primary congested resource includes: selecting the database service with the highest resource conflict score among all database services in the current period, and denoting it as the target interference service; calculating the product of the resource consumption feature vector of the target interference service and the component of the same resource type in the hardware pressure vector at the current sampling time, and selecting the resource type corresponding to the largest product as the primary congested resource of the target interference service in the current period.

[0049] It should be noted that the database service with the highest resource conflict score is the core source of interference causing implicit resource congestion on compute nodes, providing a clear target for subsequent differentiated scheduling. Selecting the resource type corresponding to the largest product as the primary congested resource can identify the physical resource that causes the strongest blocking interference to compute nodes among multi-dimensional conflicts, providing physical path guidance for subsequent local compression or cross-machine migration scheduling.

[0050] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining the cumulative congestion degree includes: setting the initial value of the cumulative congestion degree of each database service node on the computing node to zero; designating the database service node containing the target interference service in the current period as the target database node in the current period; accumulating the resource conflict score of the target interference service to the cumulative congestion degree of the target database node at the previous sampling time at the current sampling time to obtain the cumulative congestion degree of the target database node at the current sampling time; if the computing node does not meet the preset abnormal conditions at the current sampling time, multiplying the cumulative congestion degree of all database service nodes on the computing node at the previous sampling time at the current sampling time by a preset attenuation coefficient, and using this as the cumulative congestion degree of the corresponding database service node at the current sampling time. It should be noted that the cumulative congestion degree of the first sampling time within the monitoring period is set to zero.

[0051] It should be noted that the resource conflict score of the target interfering service is added to the cumulative congestion level of the target database node at the previous sampling time to obtain its latest cumulative congestion level at the current sampling time. The physical mechanism of the accumulation process is that if a database service continuously generates severe resource conflicts, the cumulative congestion level of its node will rapidly increase and accelerate towards the physical scheduling trigger threshold, thereby achieving accurate capture of persistent interference sources. In order to release historical non-persistent or sporadic interference traces and prevent minor conflicts from accumulating disorderly across cycles and causing misjudgments, a linear decay operation is performed on the database service nodes of computing nodes that do not meet the preset abnormal conditions at the current sampling time. A bottom-line protection is used to ensure that the cumulative congestion level is non-negative, maintaining the safety and stability of the scheduling control closed loop.

[0052] In this embodiment of the invention, a preset attenuation coefficient is used to make the cumulative congestion automatically decrease over time, ensuring that the system quickly eliminates the impact of historical extreme value interference on the current scheduling state. Based on the target half-life period, this embodiment sets it to 0.8, but implementers can set it according to specific circumstances.

[0053] In this embodiment of the invention, the scheduling triggering condition includes: the cumulative congestion level of the database service node to which the target interfering service belongs in the current period is greater than a preset scheduling triggering threshold at the current sampling time. It should be noted that if the scheduling triggering condition is not met, it indicates that the target interfering service in the current period may be in an intermittent fluctuation phase or has not yet reached the governance threshold. The system will not perform physical-level scheduling actions, but will maintain the current configuration and continue monitoring the loop, waiting for further observation at the next sampling time. If the scheduling triggering condition is met, it indicates that the hidden resource interference behavior has a high degree of persistence or stubbornness, and the system will then activate the physical intervention procedure to perform subsequent differentiated scheduling.

[0054] In this embodiment of the invention, in a preset non-production test environment, a typical non-computationally intensive blocking event (such as high-concurrency external sorting or full table scan) is simulated and triggered. When the non-computationally intensive blocking index of the event is continuously higher than the preset abnormal conditions, the average resource conflict score generated by the target interfering service in a single sampling period is recorded. The maximum number of consecutive sampling periods that the system can tolerate for this type of moderate to severe interference behavior is set (for example, set to 3 periods). The product of the average resource conflict score and the maximum number of consecutive sampling periods that can be tolerated is used as the preset scheduling trigger threshold for the computing node; preferably, it is set to 3.

[0055] Preferably, in some possible implementations of the embodiments of the present invention, the differentiated scheduling method includes: resource types including incompressible resources and compressible resources; if the primary congested resource is an incompressible resource, then a rate limiting operation is performed on the database service node to which the target interfering service belongs, and the node is marked as pending scheduling to perform an asynchronous migration operation; if the primary congested resource is a compressible resource, then a quota compression operation is performed on the database service node to which the target interfering service belongs: the quota compression coefficient is determined based on the resource conflict score of the target interfering service; the product of the preset baseline quota of the database service node to which the target interfering service belongs and the quota compression coefficient is used as the resource quota limit value of the database service node, and written into the operating system kernel control group file of the computing node.

[0056] It should be noted that incompressible resources include disk I / O bandwidth and memory capacity, while compressible resources include processor time slices. If the primary congested resource is an incompressible resource, since such resources are usually shared by the physical hardware bus and cannot be completely isolated through software logic, this embodiment adopts a "local real-time suppression and remote asynchronous migration" strategy: the disk I / O bandwidth limit of the database service node is limited by the operating system kernel control group of the compute node to achieve real-time pressure reduction; at the same time, the database service node is marked as pending scheduling, and after entering the load trough period, a cross-node migration task is triggered to physically relocate the database service node to another server with sufficient resources within the cluster. The long-term management of implicit resource contention is achieved through the redistribution of physical space, and the local rate limiting rules are released after the migration is completed. Specifically, if the hardware pressure index of all resource types is less than the preset idle threshold and the non-computational blocking index is less than the preset silent threshold at a certain moment, the current compute node is determined to have entered the load trough period. The preset idle threshold is the maximum value of the hardware pressure index of each resource type under the idle state of the compute node, preferably set to 0.1; under the standard load of a single business, the non-compute blocking index when the compute node is in the best service quality state is selected as 50% of the preset silent threshold, preferably set to 0.5.

[0057] If the primary congested resource is compressible, processor quota compression is applied to the interference source to reduce the pulse frequency of its underlying I / O calls and memory throughput, indirectly reducing pressure on non-computationally intensive blocking. This eliminates the implicit contention for underlying physical resources at its source while minimizing interference. The percentage reduction in CPU power required by the system for the database tenant container belonging to the target interfering service, under the current severity of resource conflict, is quantified and denoted as the quota compression coefficient. The quota compression coefficient for the target interfering service. ,in, This is the minimum quota ratio constant; Score the resource conflicts for the target disruptive business in the current cycle; To adjust the sensitivity coefficient. Multiplying the calculated quota by a preset benchmark effectively avoids the risk of exponential quota decay caused by continuous scheduling triggers, ensuring that services maintain a basic operational level. Finally, the system writes the calculated value into the operating system kernel control group file, achieving precise suppression of the interference source's computing power locally, and quickly restoring the responsiveness of critical services with minimal scheduling overhead.

[0058] In this embodiment of the invention, stress tests are conducted to record the response status of each database service when only extremely low computing power (e.g., 5% to 15% of normal load) is retained. The critical proportion that ensures the database service process does not experience connection timeouts or crashes due to computing power exhaustion is used as the minimum quota ratio constant; preferably, it is set to 0.1. Multiple sets of resource conflict scores with gradients are simulated in the test environment to observe different... The value represents the rate at which the non-computational blocking index of a computing node falls back; this will enable the system to exert significant computing power suppression without causing drastic business fluctuations when the resource conflict score reaches a moderate level (e.g., around 0.5). The value is used as an adjustment sensitivity coefficient; the preferred setting is 1. The monitoring agent directly reads the initial configuration file of the corresponding database tenant container in the operating system kernel control group, obtains the upper limit of CPU time slice allocation of the database service node to which the target interference service belongs without any dynamic resource scheduling intervention, and records it as the preset baseline calculation quota.

[0059] Preferably, in some possible implementations of the embodiments of the present invention, after performing differentiated scheduling, the method further includes: if the database service node to which the target interfering service belongs performs a migration operation, then stop performing the post-verification operation on the target interfering service until the migration is completed and resumed; if the database service node to which the target interfering service belongs performs a rate limiting operation or a quota compression operation, then perform the post-verification operation on the target interfering service; the post-verification operation includes: obtaining the non-computational blocking index of the computing node at the next sampling time at the current sampling time; when the non-computational blocking index at the next sampling time meets a preset abnormal condition, canceling the rate limiting operation or quota compression operation performed on the database service node to which the target interfering service belongs; multiplying the component of the primary congested resource in the resource consumption feature vector of the target interfering service by a preset attenuation correction factor and re-normalizing it to obtain the updated resource consumption feature vector; within a preset exemption period starting from the current sampling time, excluding the target interfering service of the current period from being selected as a new target interfering service.

[0060] It should be noted that if the database service node to which the target interfering service belongs performs a cross-node migration operation, involving spatial data replacement and significant physical latency, the system will temporarily suspend the post-verification operation for the target interfering service until it receives a migration completion signal from the container orchestration system before resuming status monitoring. If quota compression or rate limiting operations are performed, which are logical suppressions that take effect at the millisecond level, the system will obtain the non-computational blocking index of the compute node immediately before the next sampling time and can then perform post-verification operations.

[0061] The post-validation operation includes: if the non-computational congestion index at the next sampling time meets the preset anomaly condition, indicating that the system is still congested and the previous scheduling was ineffective, then the previous attribution conclusion may be invalid. The system immediately triggers an error correction procedure: on the one hand, it cancels the rate limiting operation command or quota compression command for the database service node to prevent affecting normal business; on the other hand, it multiplies the component of the primary congested resource in the resource consumption feature vector of the target interfering business by a preset attenuation correction factor and re-normalizes it with L1 to obtain an updated resource consumption feature vector, so as to correct the model's consumption prediction of the business on the primary congested resource. At the same time, within the preset exemption period from the current sampling time, the business is forcibly excluded in the subsequent scheduling screening process, thereby prompting the scheduling logic to analyze and select the database business with the second largest resource conflict score as the target interfering business. Finally, through dynamic trial and error and closed-loop feedback, the system achieves accurate convergence of implicit resource congestion.

[0062] In this embodiment of the invention, by simulating a typical hardware blocking scenario, the decrease in the resource conflict score of the target interfering service in the next cycle is observed when different attenuation ratios (e.g., 0.3, 0.5, 0.7) are applied to the resource consumption feature vector of the target interfering service after the posterior verification judgment fails. The attenuation ratio that ensures rapid elimination of misjudged services and prompts the system to switch to the true source of interference is defined as the preset attenuation correction factor; preferably set to 0.5. The preset exemption duration is set to 25 seconds, which can be set by the implementer according to specific circumstances.

[0063] This invention is now complete.

[0064] Example 2: This invention proposes a cloud database multi-node resource scheduling optimization system based on a large model. Please refer to [link / reference]. Figure 3 This diagram illustrates a system architecture of a cloud database multi-node resource scheduling optimization system based on a large model, according to an embodiment of the present invention. The system includes: The data acquisition module 510 is used to acquire hardware pressure indicators of different resource types of computing nodes at the current sampling time; The non-computational blocking assessment module 520 is used to obtain the non-computational blocking index of the computing node at the current sampling time based on the degree of hardware resource shortage and the effective execution state of the processor at the current sampling time. The resource conflict attribution module 530 is used to record the time period between the current sampling time and the previous sampling time as the current period; if the non-computational blocking index meets the preset abnormal conditions, the resource conflict score of each database service in the current period is obtained based on the matching degree between the resource consumption tendency characteristics of each database service and the hardware pressure index in the current period, as well as the load activity intensity of the corresponding service in the current period; and the target interfering service and its corresponding primary congested resource are selected in the database services in the current period. The resource scheduling optimization module 540 is used to update the cumulative congestion of the database service node to which the target interfering service belongs at the current sampling time based on the resource conflict score of the target interfering service; when the cumulative congestion meets the scheduling triggering condition, it performs differentiated scheduling on the database service node to which the target interfering service belongs based on the physical attributes of the primary congested resource.

[0065] It should be noted that the devices provided in the above embodiments are only illustrative examples of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the cloud database multi-node resource scheduling optimization system based on a large model and the cloud database multi-node resource scheduling optimization method based on a large model provided in the above embodiments belong to the same concept. For details of their specific implementation, please refer to the method embodiments, which will not be repeated here.

[0066] Example 3: This embodiment also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the above-described related method steps to implement the cloud database multi-node resource scheduling optimization method based on a large model provided in the above embodiment.

[0067] The computer-readable storage medium provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects it can achieve can be referred to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0068] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0069] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0070] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A cloud database multi-node resource scheduling optimization method based on a large model, characterized in that, The method includes: Obtain hardware stress indicators for different resource types of computing nodes at the current sampling time; Based on the hardware resource scarcity and processor execution status of the computing node at the current sampling time, obtain the non-computational blocking index of the computing node at the current sampling time; The time interval between the current sampling time and the previous sampling time is recorded as the current period; if the non-computational blocking index meets the preset abnormal conditions, the resource conflict score of each database service in the current period is obtained based on the matching degree between the resource consumption tendency characteristics and hardware pressure indicators of each database service in the current period, as well as the load activity intensity of the corresponding service in the current period; and the target interfering service and its corresponding primary congested resource in the database services in the current period are selected. Based on the resource conflict score of the target interfering service, update the cumulative congestion level of its database service node at the current sampling time; when the cumulative congestion level meets the scheduling triggering condition, perform differentiated scheduling on the database service node to which the target interfering service belongs based on the physical attributes of the primary congested resource.

2. The cloud database multi-node resource scheduling optimization method based on a large model according to claim 1, characterized in that, The process of obtaining the non-computational blocking index of the computing node at the current sampling time includes: The hardware pressure indicators of all resource types of the computing node at the current sampling time are normalized respectively, and the normalization results are arranged to obtain the hardware pressure vector. Obtain the processor time increment of the computing node in the current cycle, and use the ratio of the processor time increment to the total clock cycles of the current cycle as the effective execution ratio of the computing node in the current cycle. The non-computational blocking index of the computing node at the current sampling time is obtained based on the maximum component in the hardware pressure vector and the effective execution ratio of the processor; the effective execution ratio of the processor is negatively correlated with the non-computational blocking index.

3. The cloud database multi-node resource scheduling optimization method based on a large model according to claim 2, characterized in that, The process of obtaining resource conflict scores for each database service within the current period includes: Extract the SQL text of each database service in the current period and perform anonymization processing. Input the processed SQL text into a preset resource consumption feature model to obtain the initial consumption tendency ratio of each resource type. Arrange the L1 normalized results of the initial consumption tendency ratios of all resource types to obtain the resource consumption feature vector of each database service in the current period. Obtain the row scan rate and connection activity ratio of each database service in the current period; normalize the row scan rate to obtain the throughput pressure assessment value; calculate the product of the throughput pressure assessment value and the connection activity ratio, and select the minimum value between the product and the constant 1 as the load activity intensity coefficient of each database service in the current period. The hardware pressure vector is multiplied by the resource consumption feature vector of each database service in the current period. The result of the multiplication is weighted by the load activity intensity coefficient of the corresponding database service to obtain the resource conflict score of each database service in the current period.

4. The cloud database multi-node resource scheduling optimization method based on a large model according to claim 3, characterized in that, The selection of target interfering services and their corresponding primary congested resources in the database services within the current period includes: The database service with the highest resource conflict score among all database services in the current period is selected and designated as the target interfering service. Calculate the product of the resource consumption feature vector of the target interference service and the component of the same resource type in the hardware pressure vector at the current sampling time, and select the resource type corresponding to the largest product as the primary congestion resource of the target interference service in the current period.

5. The cloud database multi-node resource scheduling optimization method based on a large model according to claim 1, characterized in that, The step of updating the cumulative congestion level of the database service node to which the target interfering service belongs at the current sampling time based on the resource conflict score of the target interfering service includes: Set the initial value of the cumulative congestion level of each database service node on the compute node to zero; The database service node containing the target interference service in the current period is recorded as the target database node in the current period. The resource conflict score of the target interference service is added to the cumulative congestion of the target database node in the previous sampling time at the current sampling time to obtain the cumulative congestion of the target database node at the current sampling time. If the compute node does not meet the preset abnormal conditions at the current sampling time, the cumulative congestion of all database service nodes on the compute node at the previous sampling time at the current sampling time will be multiplied by the preset attenuation coefficient, and the product will be used as the cumulative congestion of the corresponding database service node at the current sampling time.

6. The cloud database multi-node resource scheduling optimization method based on a large model according to claim 1, characterized in that, The differential scheduling of the database service node to which the target interference service belongs includes: The resource types include two categories: incompressible resources and compressible resources; If the primary congested resource is an incompressible resource, then a rate limiting operation is performed on the database service node to which the target interfering service belongs, and the node is marked as pending scheduling to perform an asynchronous migration operation; If the primary congested resource is a compressible resource, then a quota compression operation is performed on the database service node to which the target interfering service belongs: the quota compression coefficient is determined based on the resource conflict score of the target interfering service; the product of the preset baseline calculated quota of the database service node to which the target interfering service belongs and the quota compression coefficient is used as the resource quota limit value of the database service node, and written into the operating system kernel control group file of the computing node.

7. The cloud database multi-node resource scheduling optimization method based on a large model according to claim 1, characterized in that, After performing differentiated scheduling, the following is also included: If the database service node to which the target interfering service belongs performs a migration operation, then the post-verification operation on the target interfering service will be stopped until the migration is completed and then restored. If the database service node to which the target interference service belongs performs rate limiting or quota compression, then a post-verification operation is performed on the target interference service. The post-validation operation includes: obtaining the non-computational congestion index of the computing node at the next sampling time from the current sampling time; when the non-computational congestion index at the next sampling time meets the preset abnormal conditions, canceling the rate limiting or quota compression operation performed on the database service node to which the target interference service belongs; multiplying the component of the primary congested resource in the resource consumption feature vector of the target interference service by a preset attenuation correction factor and re-normalizing it to obtain the updated resource consumption feature vector; and excluding the target interference service of the current period from being selected as the new target interference service within the preset exemption period from the current sampling time.

8. The cloud database multi-node resource scheduling optimization method based on a large model according to claim 1, characterized in that, The preset abnormal condition is that the non-computational blocking index of the computing node at the current sampling time is greater than the preset blocking threshold.

9. The cloud database multi-node resource scheduling optimization method based on a large model according to claim 3, characterized in that, The hardware pressure vector and the resource consumption feature vector have the same resource type corresponding to the same dimension component.

10. A cloud database multi-node resource scheduling and optimization system based on a large model, characterized in that, The system includes: The data acquisition module is used to obtain hardware stress indicators of different resource types of computing nodes at the current sampling time; The non-computational blocking assessment module is used to obtain the non-computational blocking index of the computing node at the current sampling time based on the degree of hardware resource shortage and the effective execution state of the processor at the current sampling time. The resource conflict attribution module is used to record the time period between the current sampling time and the previous sampling time as the current period; if the non-computational blocking index meets the preset abnormal conditions, then based on the matching degree between the resource consumption tendency characteristics of each database service and the hardware pressure index in the current period, as well as the load activity intensity of the corresponding service in the current period, the resource conflict score of each database service in the current period is obtained; and the target interfering service and its corresponding primary congested resource in the database services in the current period are selected. The resource scheduling optimization module is used to update the cumulative congestion level of the database service node to which the target interfering service belongs at the current sampling time based on the resource conflict score of the target interfering service; when the cumulative congestion level meets the scheduling triggering condition, differential scheduling is performed on the database service node to which the target interfering service belongs based on the physical attributes of the primary congested resource.

Images