Resource allocation method, apparatus, device, storage medium, and program

Abstract

This application provides a resource allocation method, apparatus, device, storage medium, and program. The method includes: obtaining memory usage parameters and the number of ordered table files for each of multiple merged storage instances; determining a health status index for each merged storage instance based on the memory usage parameters and the number of ordered table files; determining the task priority for each merged storage instance based on the business attribute weight and health status index; and executing the merging task corresponding to each merged storage instance based on the task priority. This method improves the stability of system read and write operations.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a resource allocation method, apparatus, device, storage medium, and program. Background Technology

[0002] In distributed storage systems, the Log-Structured Merge (LSM) tree structure is used to efficiently manage write and query operations on large-scale data.

[0003] In existing technologies, multiple merged storage instances (i.e., LSM instances) can be deployed to achieve data sharding, parallel processing, and differentiated management of different types of metadata. However, the lack of coordination in resource scheduling among multiple instances and the disordered execution of merge tasks restrict the overall efficiency of the system. Especially in shared resource scenarios, the merge tasks of non-critical instances may preempt the system resources of critical business instances, causing critical business operations to be blocked due to metadata processing delays, resulting in low system read and write stability. Summary of the Invention

[0004] This application provides a resource allocation method, apparatus, device, storage medium, and program to solve the technical problem of low system read / write stability.

[0005] Firstly, this application provides a resource allocation method, including:

[0006] Obtain the memory usage parameters and the number of ordered table files for each of the multiple merged storage instances;

[0007] Based on the memory usage parameters and the number of ordered table files of each merged storage instance, the health status index of each merged storage instance is determined;

[0008] The task priority of each merged storage instance is determined based on the business attribute weight and health status index corresponding to each merged storage instance.

[0009] The merge task corresponding to each merged storage instance is executed according to the task priority of each merged storage instance.

[0010] This application solves the problem of task priority allocation under shared resources for multiple merged storage instances, avoiding low-priority tasks from preempting high-priority resources and ensuring the stability of critical business operations.

[0011] Optionally, the health status index includes a write health index and a read health index; for any merged storage instance; the health status index of the merged storage instance is determined based on the memory usage parameters and the number of ordered table files of the merged storage instance, including:

[0012] Get the memory usage threshold and the ordered table threshold;

[0013] Based on the memory usage parameters and the memory usage threshold, determine the write health index corresponding to the merged storage instance;

[0014] The read health index corresponding to the merged storage instance is determined based on the number of ordered table files and the ordered table threshold.

[0015] In this application, read and write health indices can be calculated based on memory usage parameters and the number of ordered table files, respectively, to achieve multi-dimensional quantitative assessment of the health status of merged storage instances and improve the matching degree between the health status index and the actual running status of the instance.

[0016] Optionally, based on the number of ordered table files and the ordered table threshold, the read health index corresponding to the merged storage instance is determined, including:

[0017] Determine the cache ratio parameter corresponding to the merged storage instance, the cache ratio parameter being used to indicate...;

[0018] The initial health index corresponding to the merged storage instance is determined based on the number of ordered table files and the ordered table threshold.

[0019] The initial health index is weighted and adjusted according to the cache ratio parameter to obtain the read health index.

[0020] In this application, the introduction of a cache ratio parameter makes the health status assessment more closely match the actual performance requirements, avoids resource allocation errors caused by deviations in a single indicator, and improves the stability of critical business operations.

[0021] Optionally, the initial health index is weighted and adjusted according to the cache ratio parameter to obtain the read health index, including:

[0022] Identify the business model of the storage system through business assessment components;

[0023] Adjust the reading index weight of the health reading index according to the business model described above;

[0024] The initial health index is adjusted based on the cache ratio parameter and the read index weight to obtain the read health index.

[0025] In this application, the dynamic adaptation of business models improves the scenario adaptability of health status assessment, and the identification of business models makes the adjustment of health status weights more in line with actual business needs.

[0026] Optionally, obtain the memory usage threshold and the ordered table threshold, including:

[0027] By using machine learning models, historical operating data and system load trends are analyzed and processed to generate threshold adjustment strategies for health status.

[0028] The memory usage threshold and the ordered table threshold are determined according to the threshold adjustment strategy.

[0029] In this application, the health threshold can be dynamically adjusted to avoid the limitations of static thresholds in complex scenarios and improve the flexibility of resource scheduling.

[0030] Optionally, for any merged storage instance; based on the task priority corresponding to the merged storage instance, execute the merge task corresponding to the merged storage instance, including:

[0031] Based on the task priority, at least one subtask corresponding to at least one task type is determined in the merged task;

[0032] For any given subtask, determine the target priority of the subtask based on the task type corresponding to the subtask;

[0033] Each subtask is executed according to its target priority.

[0034] In this application, merging tasks can be executed sequentially according to task priority to achieve unified scheduling and reasonable allocation of shared resources among multiple instances, thereby improving the read and write response speed of critical business instances and the overall operational stability of the system.

[0035] Optionally, the target priority of the subtask is determined according to the task type corresponding to the subtask, including:

[0036] Obtain the health status index of the merged storage instance corresponding to the subtask, and the initial priority of the task type corresponding to the subtask;

[0037] For any given subtask, the initial priority is adjusted based on the health status index corresponding to the subtask to obtain the target priority.

[0038] In this application, the initial priority of task type and instance health status index can be dynamically adjusted to match the priority of subtasks with the performance shortcomings of instances and the core nature of tasks, thereby improving the granularity and rationality of resource allocation.

[0039] Secondly, embodiments of this application provide a resource scheduling device, including an acquisition module, a first determination module, a second determination module, and an execution module:

[0040] The acquisition module is used to acquire the memory usage parameters and the number of ordered table files of each of the multiple merged storage instances.

[0041] The first determining module is used to determine the health status index of each merged storage instance based on the memory usage parameters and the number of ordered table files of each merged storage instance;

[0042] The second determining module is used to determine the task priority corresponding to each merged storage instance based on the business attribute weight and health status index corresponding to each merged storage instance;

[0043] The execution module is used to execute the merging task corresponding to each merged storage instance according to the task priority corresponding to each merged storage instance.

[0044] Thirdly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0045] The memory stores computer-executed instructions;

[0046] The processor executes computer execution instructions stored in the memory to implement the method as described in any of the first aspects.

[0047] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect.

[0048] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects.

[0049] The resource allocation method, apparatus, device, storage medium, and program provided in this application can obtain the memory usage parameters and the number of ordered table files for each merged storage instance. Based on these parameters, a health status index for each merged storage instance is determined. The storage scheduling controller can then determine the task priority of each merged storage instance based on its corresponding business attribute weight and health status index. Finally, the storage scheduling controller executes the merging tasks corresponding to each merged storage instance according to its task priority. Collecting the memory usage parameters and the number of ordered table files for each merged storage instance provides a quantitative basis for task priority calculation, while the introduction of business attribute weights ensures that resource allocation prioritizes critical tasks. By combining instance health status with business attribute weights, the problem of task priority allocation among multiple merged storage instances under shared resources is solved, preventing low-priority tasks from preempting high-priority resources and improving system read / write stability. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0051] Figure 1 A schematic diagram illustrating the application scenarios provided in the embodiments of this application;

[0052] Figure 2 A flowchart illustrating a resource allocation method provided in an embodiment of this application;

[0053] Figure 3 A flowchart illustrating another resource allocation method provided in an embodiment of this application;

[0054] Figure 4 A schematic diagram of the architecture of a resource allocation method provided in an embodiment of this application;

[0055] Figure 5 This application provides a schematic diagram of the structure of a resource scheduling device according to an embodiment of the present application;

[0056] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0057] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0058] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0059] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application. Please refer to [link / reference]. Figure 1 In this scenario, storage system 100 may include a storage scheduling controller 101 and multiple merged storage instances 102, suitable for scenarios with high write throughput. For example, storage system 100 may be a NoSQL database, a distributed file system, etc. The merged storage instance 102 is an independent data storage unit implemented using a log structure merge tree structure, and has independent memory tables, hierarchical files, and a merge scheduling mechanism.

[0060] The storage scheduling controller 101 can obtain the memory usage parameters and the number of ordered table files for each merged storage instance 102. Based on these parameters, it determines the health status index of each merged storage instance 102. The storage scheduling controller 101 can then determine the task priority for each merged storage instance 102 based on its corresponding business attribute weight and health status index. Finally, the storage scheduling controller 101 executes the merge tasks for each merged storage instance 102 according to its assigned task priority.

[0061] In related technologies, multiple merged storage instances can be deployed to achieve data sharding, parallel processing, and differentiated management of different types of metadata. However, issues such as lack of coordination in resource scheduling among multiple instances and disordered execution of merge tasks restrict the overall efficiency of the system. Especially in shared resource scenarios, merge tasks of non-critical instances may preempt system resources of critical business instances, causing critical business operations (such as real-time read and write requests) to be blocked due to metadata processing delays, resulting in low system read and write stability.

[0062] The resource allocation method provided in this application offers multi-dimensional quantitative basis for task priority calculation by collecting memory usage parameters and the number of ordered table files of merged storage instances. The introduction of business attribute weights allows for the allocation of higher overall weights to critical business instances, ensuring that resource allocation prioritizes critical tasks. By combining instance health status with business attribute weights, the method solves the task priority allocation problem among multiple merged storage instances sharing resources, preventing low-priority tasks from preempting high-priority resources and improving system read / write stability.

[0063] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0064] Figure 2 This is a flowchart illustrating a resource allocation method provided in an embodiment of this application. Please refer to [link / reference]. Figure 2 The method may include:

[0065] S201. Obtain the memory usage parameters and the number of ordered table files for each of the multiple merged storage instances.

[0066] The execution entity in this application embodiment can be a storage scheduling controller or a resource scheduling device installed on the storage scheduling controller. The resource scheduling device can be implemented by software or by a combination of software and hardware.

[0067] The memory usage parameter is used to characterize the current memory size occupied by immutable memory tables in the merged storage instance. It can be the number of bytes of memory used by the immutable memory table, the proportion of total configured memory, etc.

[0068] The number of ordered table files is the number of ordered table files contained in the topmost layer of the merged storage instance, which reflects the read efficiency and merge pressure of the merged storage instance.

[0069] The health monitor component can be pre-configured to poll and collect data on each merged storage instance at a preset period, or it can be triggered by events to actively collect data when the memory usage and number of ordered table files of the merged storage instance change.

[0070] The health observer component communicates with each merged storage instance to collect the running status information reported by each merged storage instance in real time, and extracts the memory usage parameters and the number of ordered table files from it.

[0071] S202. Determine the health status index of each merged storage instance based on the memory usage parameters and the number of ordered table files of each merged storage instance.

[0072] The health status index includes the written health index and the read health index.

[0073] The write health index indicates the health of the write performance of the merged storage instance; the read health index indicates the health of the read performance of the merged storage instance.

[0074] In some embodiments, the memory usage threshold and the ordered table threshold are obtained; the write health index corresponding to the merged storage instance is determined based on the memory usage parameters and the memory usage threshold; and the read health index corresponding to the merged storage instance is determined based on the number of ordered table files and the ordered table threshold.

[0075] Memory usage thresholds can be used to filter whether the write pressure on merged storage instances is within a normal range. For example, a memory usage threshold can be the maximum memory value or maximum percentage that an immutable memory table is allowed to use.

[0076] The ordered table threshold can be used to filter whether the number of top-level files in a merged storage instance is within a reasonable range. For example, the ordered table threshold can be the maximum number of ordered table files allowed in the top-level hierarchy.

[0077] In this application, read and write health indices can be calculated based on memory usage parameters and the number of ordered table files, respectively, to achieve multi-dimensional quantitative assessment of the health status of merged storage instances and improve the matching degree between the health status index and the actual running status of the instance.

[0078] S203. Determine the task priority for each merged storage instance based on the business attribute weight and health status index corresponding to each merged storage instance.

[0079] Business attribute weights are quantitative parameters used to indicate the criticality, real-time requirements, and other business characteristics of the corresponding merged storage instance in the system.

[0080] For example, critical business instances have a higher weight than non-critical instances. The overall business weight of critical business instances can be configured to 0.8~1.0, while the overall business weight of non-critical business instances can be configured to 0.2~0.5. Critical business instances can be assigned a higher overall business weight to prioritize resource allocation.

[0081] Business attribute weights can be pre-configured in the storage scheduling controller and can be statically configured or dynamically updated based on information such as the business type, user level, latency sensitivity level, or whether it is a core business. They can also be adaptively adjusted according to the current system load and business operation status.

[0082] The write health index and read health index can be weighted and summed based on business attribute weights to generate task priorities.

[0083] For example, assume the business attribute weight is the same as the overall business weight. Then, weighting coefficients can be assigned to the write health index and the read health index respectively. ,in, , These are the weighting coefficients for write and read performance, respectively.

[0084] Task priority is a numerical value calculated based on health status indicators and business attribute weights, used to determine the execution order of data sinking and file merging tasks. For example, for instances with high write pressure, the priority of the immutable in-memory table sinking task can be higher than that of the normal level merging task.

[0085] For example, when the write health index weight is 0.6 and the read health index weight is 0.4, the task priority = 0.6 × write health index + 0.4 × read health index.

[0086] In this application, by combining business attribute weights with multidimensional health indices, the fairness of resource allocation and the balance of business priorities are achieved. In a storage system where multiple merged storage instances coexist, resource contention among multiple instances is suppressed, thereby achieving reasonable resource allocation and system performance optimization.

[0087] S204. Execute the merge task corresponding to each merged storage instance according to the task priority of each merged storage instance.

[0088] Merge tasks can be executed in descending order of task priority for each merged storage instance.

[0089] Merge tasks are merge operations that require shared resources (such as CPU and memory) in merging storage instances, including immutable memory table sinking and ordered table file sinking.

[0090] For example, sinking an immutable memory table (IMMTABLE) involves persisting the read-only memory table data in memory as a sorted string table file (SST) stored on disk; sinking the L0 layer involves merging and persisting the sorted string table file of the top layer to the next layer.

[0091] For any merged storage instance, at least one subtask corresponding to at least one task type can be determined in the merged task based on task priority; for any subtask, the target priority of the subtask can be determined based on the task type corresponding to the subtask; and each subtask can be executed based on its target priority.

[0092] Different task types correspond to different subtasks, and the same task type can correspond to one or more subtasks.

[0093] The task types include immutable memory table sinking and L0 level ordered table sinking; subtasks can include memory release subtasks and ordered table file merging subtasks.

[0094] Among them, the immutable memory table sinking type corresponds to the memory release subtask, and the L0 layer ordered table sinking type corresponds to the ordered table file merging subtask.

[0095] In this application, merging tasks can be executed sequentially according to task priority to achieve unified scheduling and reasonable allocation of shared resources among multiple instances, thereby improving the read and write response speed of critical business instances and the overall operational stability of the system.

[0096] The resource allocation method provided in this application collects memory usage parameters and the number of ordered table files of the merged storage instances, providing a quantitative basis for task priority calculation. The introduction of business attribute weights ensures that resource allocation prioritizes critical tasks. By combining instance health status with business attribute weights, the problem of task priority allocation among multiple merged storage instances under shared resources is solved, preventing low-priority tasks from preempting high-priority resources and improving system read / write stability.

[0097] Figure 3 This is a flowchart illustrating another resource allocation method provided in an embodiment of this application. Please refer to... Figure 3 The method may include:

[0098] S301. Obtain the memory usage parameters and the number of ordered table files for each of the multiple merged storage instances.

[0099] The execution process of S301 can be found in the execution process of S201, and will not be repeated here.

[0100] S302. Obtain the memory usage threshold and the ordered table threshold.

[0101] The memory usage threshold and the ordered table threshold can be fixed values. In some embodiments, adaptive threshold adjustments can be made to improve the flexibility of health status assessment.

[0102] Specifically, machine learning models can be used to analyze and process historical operating data and system load trends to generate threshold adjustment strategies for health status; based on the threshold adjustment strategies, memory usage thresholds and ordered table thresholds can be determined.

[0103] Machine learning models can be used for time series forecasting.

[0104] Historical operational data consists of resource preemption records, merge task delay records, and read / write performance fluctuation data generated by multiple merged storage instances during historical operation.

[0105] The system load trend is a prediction of the changes in CPU utilization, memory utilization, IO load, and concurrent tasks in the future period based on historical system load data.

[0106] Machine learning models can extract features from historical operating data, learn the correspondence between resource contention and health thresholds under different load scenarios, and predict future system load levels by combining system load trends. Based on the learned correspondence and the predicted load level, a threshold adjustment strategy matching the current system state can be generated.

[0107] The threshold adjustment strategy involves dynamically adjusting the memory usage threshold and the number of ordered table files threshold. For example, the threshold is relaxed under low load scenarios and tightened under high load scenarios.

[0108] For example, in low-load scenarios, the memory usage threshold is relaxed to 150MB to reduce resource scheduling frequency; in high-load scenarios, the threshold is tightened to 80MB to more strictly control the health status.

[0109] In this application, the health threshold can be dynamically adjusted to avoid the limitations of static thresholds in complex scenarios and improve the flexibility of resource scheduling.

[0110] S303. Determine the write health index corresponding to the merged storage instance based on the memory usage parameters and memory usage threshold.

[0111] The write health index can range from [0,1]. The smaller the memory usage parameter, the closer the write health index is to 1 (better write performance); the larger the memory usage parameter, the closer the write health index is to 0 (greater write pressure).

[0112] When the memory usage parameter is less than or equal to the memory usage threshold, the write health index is 1, indicating that the instance's write pressure is extremely low and the write performance is in optimal condition.

[0113] When the memory usage parameter exceeds the memory usage threshold, the greater the exceedance, the lower the write health index.

[0114] Specifically, when the memory usage parameter is greater than the memory usage threshold but less than or equal to the upper limit of memory usage, the write health index = 1 - (memory usage parameter - memory usage threshold) / (upper limit of memory usage threshold - memory usage threshold). That is, the write health index is linearly reduced according to the proportion of memory usage exceeding the memory usage threshold, so as to achieve smooth quantification of write pressure.

[0115] When the memory usage parameter is greater than the upper limit threshold, the write health index is 0.1 (or 0), indicating that the instance memory usage has exceeded the capacity limit, the write performance has dropped significantly, and resources need to be allocated first to execute the immutable memory table sinking task.

[0116] For example, if the memory usage threshold is 70MB and the memory usage limit threshold is 100MB, when the memory usage parameter is 60MB (≤70MB), the write health index = 1; when the memory usage parameter is 85MB (between 70 and 100MB), the write health index = 1 - (85 - 70) / (100 - 70) = 1 - 0.5 = 0.5; when the memory usage parameter is 110MB (>100MB), the write health index = 0.1.

[0117] In this application, a piecewise linear calculation method is used to ensure that the health index is sensitive to write pressure, while avoiding exponential mutations caused by a single threshold, so that the write health index is more in line with the actual write performance status of the instance.

[0118] S304. Determine the read health index corresponding to the merged storage instance based on the number of ordered table files and the ordered table threshold.

[0119] The execution process for determining the read health index can be found in S303, which describes the execution process for determining the write health index. The read health index ranges from [0,1]. The fewer the number of ordered table files, the closer the initial health index is to 1 (better read performance); the more ordered table files, the closer the initial health index is to 0 (greater read pressure).

[0120] In addition, when determining the health index, adjustments can be made based on the cache ratio parameter.

[0121] Specifically, determine the cache ratio parameter corresponding to the merged storage instance; determine the initial health index corresponding to the merged storage instance based on the number of ordered table files and the ordered table threshold; and adjust the initial health index by weighting it according to the cache ratio parameter to obtain the read health index.

[0122] The cache ratio parameter is the ratio of the amount of cached data to the number of ordered table files in the merged storage instance. For example, cache ratio parameter = cache size / number of ordered table files.

[0123] The initial health index is determined by referring to the execution process of S303. The ordered table threshold and the ordered table upper limit threshold need to be configured first, and the calculation is performed in a piecewise linear manner.

[0124] The weighted adjustment involves multiplying the initial health index by the cache ratio parameter to obtain the final read health index.

[0125] A higher cache ratio parameter means that more sample data can be read directly from the cache, providing stronger support for read performance, and therefore a higher read health index. A lower cache ratio parameter means that more read requests need to access disk files, resulting in weaker read performance, and therefore a lower read health index.

[0126] For example, if the ordered table threshold is 10 and the upper limit threshold is 20, when the number of ordered table files is 8, the initial health index = 1; the cache ratio parameter is 0.9, then the read health index = 1 × 0.9 = 0.9; when the number of ordered table files is 15, the initial health index = 1 - (15 - 10) / (20 - 10) = 0.5, the cache ratio parameter is 0.4, then the read health index = 0.5 × 0.4 = 0.2.

[0127] In this application, the introduction of a cache ratio parameter makes the health status assessment more closely match the actual performance requirements, avoids resource allocation errors caused by deviations in a single indicator, and improves the accuracy of merged task scheduling and the stability of critical business operations.

[0128] In some embodiments, the impact of the storage system's business model on the read health index can be considered when determining the read health index.

[0129] Specifically, the business evaluation component identifies the storage system's business model; based on the business model, the read index weight of the read health index is adjusted; and the initial health index is adjusted based on the cache ratio parameter and the read index weight to obtain the read health index.

[0130] The business evaluation component can use machine learning models to identify the current business pattern of the storage system based on data such as historical read / write request frequency and data access characteristics.

[0131] The business model refers to the operating mode of the storage system. For example, read-heavy, write-light, batch write, and configuration modification-intensive modes.

[0132] In read-heavy, write-light mode, the read health index weight can be 0.8; in batch write mode, the read health index weight can be 0.3.

[0133] For example, the number of ordered table files is 12 (ordered table threshold is 10, ordered table upper limit threshold is 20), the initial health index = 1 - (12 - 10) / (20 - 10) = 0.8; the cache ratio parameter is 0.7, the read index weight is 0.8 (read-heavy, write-light mode), then the read health index = 0.8 × 0.7 × 0.8 = 0.448.

[0134] In this application, dynamic adaptation of business models improves the scenario adaptability of health status assessment. The identification of business models makes the adjustment of health status weights more in line with actual business needs. In read-heavy, write-light scenarios, the system prioritizes resource allocation for L0 layer ordered table file sinking tasks to reduce the impact of read performance degradation on business. In batch write scenarios, the system prioritizes resource allocation for immutable memory table sinking tasks to avoid business blocking caused by memory overflow.

[0135] In some embodiments, only the read index weight corresponding to the business mode is considered when adjusting the initial health index, without considering the cache ratio parameter. For example, in the read-heavy, write-light mode, the read health index = initial health index × 0.8; in the batch write mode, the read health index = initial health index × 0.3.

[0136] S305. Determine the task priority for each merged storage instance based on the business attribute weight and health status index corresponding to each merged storage instance.

[0137] The execution process of S305 can be found in the execution process of S203, and will not be repeated here.

[0138] S306. For any merged storage instance, based on task priority, determine at least one subtask corresponding to at least one task type in the merged task.

[0139] Under the same task type, based on the shortcomings of the instance's current health status index, at least one targeted subtask is further broken down. The number and type of subtasks are determined by the execution objectives of the task type and the actual performance requirements of the instance.

[0140] When executing a merge task, it is split into corresponding subtasks (such as memory release subtasks and ordered table file merge subtasks) according to the task type (e.g., immutable memory table sinking task and L0 level ordered table file sinking task).

[0141] Each subtask independently determines its execution priority based on the health status index and task priority of its respective merged storage instance.

[0142] For example, the memory release subtask within the immutable memory table sinking task needs to be executed first due to excessive memory usage; while the write amplification suppression subtask dynamically adjusts its priority based on the degree of write amplification.

[0143] In this application, resources can be allocated more accurately by processing subtasks independently, thus avoiding resource waste and task blocking.

[0144] S307. For any subtask, determine the target priority of the subtask based on the task type corresponding to the subtask.

[0145] In some embodiments, the health status index of the merged storage instance corresponding to the subtask and the initial priority of the task type corresponding to the subtask can be obtained; for any subtask, the initial priority is adjusted according to the health status index corresponding to the subtask to obtain the target priority.

[0146] The initial priority is a baseline priority preset based on the core importance of the task type. For example, the initial priority of an immutable memory table sinking task is higher than that of an L0 layer ordered table file sinking task, and the initial priority of an emergency resource reclamation subtask is higher than that of a regular optimization subtask.

[0147] The subtask can be at least one of the following: executing the memory release subtask in the immutable memory table sinking task; executing the write amplification suppression subtask in the immutable memory table sinking task; executing the ordered table file merging subtask in the L0 level ordered table file sinking task.

[0148] In this application, the initial priority of task type and instance health status index can be dynamically adjusted to match the priority of subtasks with the performance shortcomings of instances and the core nature of tasks, thereby improving the granularity and rationality of resource allocation and further avoiding the blocking of critical subtasks.

[0149] S308. Execute each subtask according to the target priority corresponding to each subtask.

[0150] The system can schedule shared resources (CPU, memory, I / O bandwidth) to execute subtasks in descending order of their target priority.

[0151] If there are subtasks with the same target priority, the subtasks of the merged storage instance corresponding to the critical business will be executed first, or the order will be adjusted according to the resource consumption requirements of the subtasks (lightweight tasks take priority).

[0152] For example, a storage system has three subtasks to be executed: Subtask A (target priority 0.9, memory release subtask, critical business instance), Subtask B (target priority 0.6, ordered table file merging subtask, ordinary business instance), and Subtask C (target priority 0.3, write amplification suppression subtask, ordinary business instance). The storage scheduler prioritizes allocating 80% of CPU and memory resources to Subtask A to quickly release memory; after Subtask A is completed, it allocates 50% of resources to Subtask B to perform file merging; Subtask C is then triggered to execute when system resources are idle.

[0153] During execution, the progress of subtasks and the status of system resources are monitored in real time. If a high-priority subtask encounters resource shortages during execution, it can dynamically preempt the allocated resources of a low-priority subtask. If the health status index of the corresponding instance recovers to a safe range after the subtask is executed (e.g., write health index ≥ 0.7, read health index ≥ 0.7), subsequent subtasks of the same type can be terminated early or their priority can be reduced.

[0154] In this application, by scheduling resources in an orderly manner based on target priority, the execution of subtasks is matched with system resources and business requirements, ensuring the rapid implementation of key subtasks, avoiding execution blockage caused by disorderly resource preemption, and further improving the overall operational stability and read / write response efficiency of the storage system.

[0155] The resource allocation method provided in this application can quantitatively assess the health status of merged storage instances through multi-dimensional parameters such as memory usage, number of ordered table files, and cache ratio. Combined with an adaptive threshold adjustment mechanism, it calculates write and read health indices in a piecewise linear manner to identify read and write performance bottlenecks, thus overcoming the limitations of static threshold assessment in complex scenarios. By weighting business attribute weights and health status indices, it achieves differentiated calculation of task priorities, preventing low-priority instance tasks from unreasonably preempting critical business resources and ensuring the stability and real-time response capabilities of core businesses. Furthermore, through dynamic adaptation of business models and real-time monitoring and adjustment during subtask execution, it further enhances the scenario adaptability and scheduling flexibility of health status assessment, comprehensively optimizing the read and write performance, resource utilization, and operational stability of the storage system, effectively supporting storage needs with high write throughput and multiple business scenarios.

[0156] Figure 4 This is a schematic diagram illustrating the architecture of a resource allocation method provided in an embodiment of this application. Please refer to [link / reference]. Figure 4The architecture includes a health observer, a health public subscription, and a priority scheduler.

[0157] The health observer is responsible for collecting the health status of merged storage instances in the storage system. It observes the health status index of critical merged storage instances, collecting key indicators such as memory usage and the number of ordered table files; it observes the health status index of non-critical merged storage instances, collecting corresponding operational status data; and it publishes the collected health status information of each instance to the health public subscription module.

[0158] The Health Status Public Subscription serves as a central hub for health data transfer and sharing. It receives health status information published by health observers and provides data subscription services to subscribers, enabling priority schedulers to obtain real-time health data.

[0159] The priority scheduler obtains the health status index of each merged storage instance by subscribing to the health status public subscription module. Based on the health status index and business attribute weight of each instance, it calculates the priority of the merged task. According to the task priority from high to low, it arranges the resource usage order of each task to achieve orderly scheduling of shared resources.

[0160] This application achieves global awareness of the health status of multiple merged storage instances and precise allocation of resources, avoiding unreasonable preemption of high-priority resources by low-priority tasks and ensuring the stability of critical business operations.

[0161] Figure 5 This is a schematic diagram of the structure of a resource scheduling device provided in an embodiment of this application. Please refer to [link / reference]. Figure 5 The resource scheduling device 500 includes an acquisition module 501, a first determination module 502, a second determination module 503, and an execution module 504.

[0162] The acquisition module 501 is used to acquire the memory usage parameters and the number of ordered table files of each of the multiple merged storage instances.

[0163] The first determining module 502 is used to determine the health status index of each merged storage instance based on the memory usage parameters and the number of ordered table files of each merged storage instance.

[0164] The second determining module 503 is used to determine the task priority corresponding to each merged storage instance based on the business attribute weight and health status index corresponding to each merged storage instance.

[0165] The execution module 504 is used to execute the merge task corresponding to each merged storage instance according to the task priority of each merged storage instance.

[0166] Optionally, the health status index includes a write health index and a read health index; for any merged storage instance; the first determination module 502 is specifically used for:

[0167] Get the memory usage threshold and the ordered table threshold;

[0168] Determine the write health index corresponding to the merged storage instance based on memory usage parameters and memory usage thresholds;

[0169] The read health index corresponding to the merged storage instance is determined based on the number of ordered table files and the ordered table threshold.

[0170] Optionally, the first determining module 502 is specifically used for:

[0171] Determine the cache ratio parameters corresponding to the merged storage instances;

[0172] The initial health index corresponding to the merged storage instance is determined based on the number of ordered table files and the ordered table threshold.

[0173] The initial health index is weighted and adjusted based on the cache ratio parameter to obtain the read health index.

[0174] Optionally, the first determining module 502 is specifically used for:

[0175] Identify the business model of the storage system through business assessment components;

[0176] Adjust the weighting of the health index reading function according to the business model.

[0177] The initial health index is adjusted based on the cache ratio parameter and the read index weight to obtain the read health index.

[0178] Optionally, the first determining module 502 is specifically used for:

[0179] By using machine learning models, historical operating data and system load trends are analyzed and processed to generate threshold adjustment strategies for health status.

[0180] Based on the threshold adjustment strategy, determine the memory usage threshold and the ordered table threshold.

[0181] Optionally, for any merged storage instance; execution module 504 is specifically used for:

[0182] Based on task priority, in the merged tasks, at least one subtask corresponding to at least one task type is identified;

[0183] For any given subtask, determine the target priority of the subtask based on the task type it corresponds to.

[0184] Execute each subtask according to its target priority.

[0185] Optionally, execution module 504 is specifically used for:

[0186] Obtain the health status index of the merged storage instance corresponding to the subtask, as well as the initial priority of the task type corresponding to the subtask;

[0187] For any given subtask, the initial priority is adjusted based on the health status index corresponding to the subtask to obtain the target priority.

[0188] The resource allocation device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0189] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Please refer to... Figure 6 The electronic device 600 may include a processor 601 and a memory 602 communicatively connected to the processor 601. Exemplarily, the processor 601 and the memory 602 are interconnected via a bus 603.

[0190] Memory 602 stores computer-executed instructions;

[0191] The processor 601 executes computer execution instructions stored in the memory 602, causing the processor 601 to perform the resource allocation method as shown in the above method embodiment.

[0192] Accordingly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the resource allocation method of the above-described method embodiments.

[0193] Accordingly, embodiments of this application may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the resource allocation method shown in the above method embodiments.

[0194] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0195] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0196] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0197] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0198] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0199] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0200] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0201] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0202] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0203] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A resource allocation method, characterized in that, include: Obtain the memory usage parameters and the number of ordered table files for each merged storage instance in multiple merged storage instances; Based on the memory usage parameters and the number of ordered table files of each merged storage instance, the health status index of each merged storage instance is determined; The task priority of each merged storage instance is determined based on the business attribute weight and health status index corresponding to each merged storage instance. The merge task corresponding to each merged storage instance is executed according to the task priority of each merged storage instance.

2. The method according to claim 1, characterized in that, The health status index includes a write health index and a read health index; For any merged storage instance; Based on the memory usage parameters and the number of ordered table files of the merged storage instance, a health status index for the merged storage instance is determined, including: Get the memory usage threshold and the ordered table threshold; Based on the memory usage parameters and the memory usage threshold, determine the write health index corresponding to the merged storage instance; The read health index corresponding to the merged storage instance is determined based on the number of ordered table files and the ordered table threshold.

3. The method according to claim 2, characterized in that, Based on the number of ordered table files and the ordered table threshold, the read health index corresponding to the merged storage instance is determined, including: Determine the cache ratio parameter corresponding to the merged storage instance; The initial health index corresponding to the merged storage instance is determined based on the number of ordered table files and the ordered table threshold. The initial health index is weighted and adjusted according to the cache ratio parameter to obtain the read health index.

4. The method according to claim 3, characterized in that, The initial health index is weighted and adjusted according to the cache ratio parameter to obtain the read health index, including: Identify the business model of the storage system through business assessment components; Adjust the reading index weight of the health reading index according to the business model described above; The initial health index is adjusted based on the cache ratio parameter and the read index weight to obtain the read health index.

5. The method according to claim 2, characterized in that, Obtain the memory usage threshold and the ordered table threshold, including: By using machine learning models, historical operating data and system load trends are analyzed and processed to generate threshold adjustment strategies for health status. The memory usage threshold and the ordered table threshold are determined according to the threshold adjustment strategy.

6. The method according to claim 1, characterized in that, For any merged storage instance; based on the task priority corresponding to the merged storage instance, execute the merge task corresponding to the merged storage instance, including: Based on the task priority, at least one subtask corresponding to at least one task type is determined in the merged task; For any given subtask, determine the target priority of the subtask based on the task type corresponding to the subtask; Each subtask is executed according to its target priority.

7. The method according to claim 6, characterized in that, Based on the task type corresponding to the subtask, the target priority of the subtask is determined, including: Obtain the health status index of the merged storage instance corresponding to the subtask, and the initial priority of the task type corresponding to the subtask; For any given subtask, the initial priority is adjusted based on the health status index corresponding to the subtask to obtain the target priority.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, are used to implement the method described in any one of claims 1-7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images