A resource allocation method, system and electronic device for a data warehouse
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CISDI INFORMATION TECH CO LTD
- Filing Date
- 2026-04-10
- Publication Date
- 2026-06-09
Smart Images
Figure CN122173295A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a resource allocation method, system and electronic device for a data warehouse. Background Technology
[0002] As enterprises continue to deepen their digital transformation, the scale of data is growing exponentially, and business scenarios are becoming more diversified. Data warehouses need to support multiple types of data processing and application needs simultaneously, including full batch processing, incremental minute-level processing, and near real-time second-level processing on the data processing end, and high-concurrency point queries and high-throughput analysis queries on the data application end. Complex and mixed scenarios place higher demands on data warehouses.
[0003] However, the resource allocation and load control technologies of related data warehouses have significant shortcomings and are difficult to adapt to the above requirements. On the one hand, resource allocation is too simplistic, often dividing resource nodes only according to business type without considering data characteristics, business priorities, and processing time requirements. This can easily lead to resource contention when scenarios overlap, resulting in increased latency for core businesses, wasted resources for non-core businesses, and low resource utilization. On the other hand, load adjustment mechanisms are rigid, often based on simple rate limiting or expansion triggered by fixed thresholds. They only monitor single performance indicators such as CPU (Central Processing Unit) utilization and memory utilization, lacking a multi-dimensional load assessment system. They can only passively respond to overload and lack tiered adjustment strategies, which can easily lead to resource waste or insufficient stability. Meanwhile, resource allocation and load regulation in the relevant solutions are mostly independent modules, which are not deeply integrated with the back pressure mechanism of the streaming system, microservice architecture, and hybrid rate limiting algorithm. They cannot achieve full-link coordination of the three links of resource allocation, load monitoring and request regulation. They also do not design dedicated resource coordination mechanisms for scenarios such as the interweaving of full batch processing and near real-time processing, and parallel processing of high concurrency and high throughput queries. They mostly adopt a "one-size-fits-all" strategy, which makes it impossible to meet the needs of some scenarios and makes it difficult to guarantee system performance and stability.
[0004] In summary, data warehouses in high-concurrency, multi-service scenarios suffer from inaccurate resource allocation, unintelligent load balancing, and insufficient technical collaboration, severely hindering data processing efficiency, system stability, and resource utilization. Clearly, a new resource allocation method for data warehouses is urgently needed to address at least one of these problems.
[0005] It should be noted that the above content only provides background information related to this application and does not necessarily constitute prior art. Summary of the Invention
[0006] This application provides a resource allocation method, system, and electronic device for data warehouses to solve the technical problem of inaccurate resource allocation in data warehouses.
[0007] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.
[0008] This application provides a resource allocation method for a data warehouse, comprising: acquiring a data processing request, the data processing request carrying a target business scenario type; dynamically routing the data processing request to a target resource node group according to the matching result of the target business scenario type and an initial business scenario type, wherein the data warehouse cluster is pre-divided into multiple resource node groups corresponding to the initial business scenario type; determining the comprehensive load value of the data warehouse cluster at a target time; if the comprehensive load value is greater than a pre-emptive trigger threshold, and a preset business scenario type exists among the target business scenario types, determining a dedicated resource pool from the target resource node group for the target task corresponding to the preset business scenario type, and dynamically releasing the dedicated resource pool after the target task is completed.
[0009] In one embodiment of this application, based on the aforementioned scheme, determining the comprehensive load value of the data warehouse cluster at a target time includes: obtaining load indicators of multiple dimensions of each resource node in the data warehouse cluster at the target time; and calculating the comprehensive load value at the target time by weighted summation of the load indicators of the multiple dimensions according to preset weight coefficients.
[0010] In one embodiment of this application, determining the comprehensive load value of the data warehouse cluster at a target time based on the aforementioned scheme further includes: obtaining historical load indicators of multiple dimensions within a preset time window ending at the target time; calculating the average load indicator of each of the multiple dimensions within the preset time window based on the historical load indicators; and calculating the comprehensive load value at the target time by weighted summation of the average load indicators of each dimension based on the preset weight coefficient.
[0011] In one embodiment of this application, after determining the comprehensive load value of the data warehouse cluster at a target time based on the aforementioned scheme, the method further includes: determining a target key parameter for calculating the target write frequency according to a predefined interval in which the comprehensive load value is located; calculating the target write frequency based on the actual write frequency at the current time, the comprehensive load value, and the target key parameter; and adjusting the data write frequency according to the target write frequency.
[0012] In one embodiment of this application, based on the aforementioned scheme, the target key parameters for calculating the target write frequency are determined according to the predefined interval in which the comprehensive load value is located, including: presetting a warning range for the comprehensive load, determining different predefined intervals according to the warning range, and presetting the target key parameters corresponding to each predefined interval; comparing the comprehensive load value with the warning range, determining the predefined interval in which the comprehensive load value is located according to the comparison result, and then determining the target key parameters for calculating the target write frequency.
[0013] In one embodiment of this application, based on the foregoing scheme, the target write frequency is calculated using the following formula: ,in, It is the target write frequency. It is the actual write frequency at the current moment. This is the overall load value. It is the target load. It is the gain coefficient, and the target key parameters include the target load and the gain coefficient.
[0014] In one embodiment of this application, after dynamically releasing the dedicated resource pool based on the aforementioned scheme, the method further includes: triggering a resource release event; responding to the resource release event, automatically adjusting the resource quotas of other running tasks according to a preset adjustment strategy, so as to utilize the released resources for elastic expansion, wherein the preset adjustment strategy is determined based on task priority and / or the resources required by the task.
[0015] This application also provides a resource allocation system for a data warehouse, comprising: a unified access entry module for receiving data processing requests, the data processing requests carrying a target business scenario type; a resource node partitioning module for partitioning the data warehouse cluster into multiple resource node groups corresponding to initial business scenario types; a request routing and dynamic allocation module connected to the unified access entry module and the resource node partitioning module, for dynamically routing the data processing request to a target resource node group based on the matching result between the target business scenario type and the initial business scenario type; a load monitoring module for monitoring the comprehensive load value of the data warehouse cluster at a target time; and a resource pre-allocation and dynamic release module for determining a dedicated resource pool from the target resource node group for the target task corresponding to the preset business scenario type if the comprehensive load value is greater than the pre-allocation trigger threshold and a preset business scenario type exists among the target business scenario types, and dynamically releasing the dedicated resource pool after the target task is completed.
[0016] In one embodiment of this application, based on the foregoing scheme, the resource allocation system for the data warehouse further includes a hierarchical back pressure controller connected to the load monitoring module, used to determine the target key parameters for calculating the target write frequency according to the predefined range in which the comprehensive load value is located, calculate the target write frequency based on the actual write frequency at the current moment, the comprehensive load value, and the target key parameters, and adjust the data write frequency according to the target write frequency.
[0017] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a resource allocation method for a data warehouse as described in any of the above embodiments.
[0018] The beneficial effects of this application are as follows: It innovatively proposes a condition-triggered resource pre-allocation mechanism. When the comprehensive load value exceeds the pre-allocation trigger threshold and a preset business scenario type exists in the target business scenario type, resource pre-allocation is initiated. After the target task is completed, the resources are dynamically released, realizing the comprehensive optimization of service quality assurance for critical businesses and cluster resource utilization. It also realizes dynamic resource allocation, effectively solving the problems of resource competition and inaccurate resource allocation in mixed load scenarios, thereby improving data processing efficiency, system stability and resource utilization.
[0019] In addition, based on the predefined range of the comprehensive load value, the target key parameters used to calculate the target write frequency are dynamically adjusted, realizing fine-grained traffic regulation and intelligent overload protection, effectively solving the problems of unintelligent load regulation and overload in mixed load scenarios.
[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0022] In the attached diagram: Figure 1 This is a schematic flowchart illustrating a resource allocation method for a data warehouse, as shown in an exemplary embodiment of this application. Figure 2 This is a flowchart illustrating a resource allocation method for a data warehouse, as shown in another exemplary embodiment of this application. Figure 3This is a block diagram illustrating a resource allocation system for a data warehouse, as shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating the module composition and data flow of a resource allocation system for a data warehouse, as shown in an exemplary embodiment of this application. Figure 5 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation
[0023] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0024] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0025] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.
[0026] First, it should be noted that dynamic routing refers to the technology by which routers automatically build and adjust routing tables by exchanging routing information, and select the optimal path based on network conditions, as opposed to static routing.
[0027] A data warehouse (DW or DWH) is a strategic collection of all types of data that supports decision-making processes at all levels within an enterprise. It is a single data store created for analytical reporting and decision support purposes. It provides guidance for business process improvement, monitoring of time, cost, quality, and control for enterprises that require business intelligence.
[0028] ETL is an abbreviation for Extract, Transform, and Load, and is a core data processing technology used to integrate and process data from multiple source systems into a target data storage (such as a data warehouse or data lake).
[0029] I / O (Input / Output) typically refers to the input and output of data between internal memory and external memory or other peripheral devices.
[0030] Figure 1 This is a schematic flowchart illustrating a resource allocation method for a data warehouse, as shown in an exemplary embodiment of this application. (Refer to...) Figure 1 As shown, the resource allocation method for a data warehouse includes at least steps S110 to S140, which are described in detail below: In step S110, a data processing request is obtained.
[0031] The data processing request carries a unique identifier for the target business scenario type.
[0032] In this embodiment, data processing requests include, but are not limited to, query requests for obtaining or reading data, and write requests for creating, updating, or deleting data.
[0033] In step S120, the data processing request is dynamically routed to the target resource node group based on the matching result between the target business scenario type and the initial business scenario type.
[0034] The data warehouse cluster is pre-divided into multiple isolated resource node groups corresponding to the initial business scenario type.
[0035] In this embodiment, a data warehouse cluster refers to a distributed system composed of multiple computing nodes, used to provide data storage, query processing, and data analysis services. The initial business scenario type refers to a predefined business classification based on both data characteristics and business priority. Each initial business scenario type corresponds to one or more resource node groups, used to achieve static mapping between requests and resources. Based on both data characteristics and business priority, the nodes of the data warehouse cluster are divided into multiple resource node groups corresponding to the initial business scenario types. If the target business scenario type is the same as the initial business scenario type, the data processing request is dynamically routed to the resource node group corresponding to the initial business scenario type, and the resource node group corresponding to the initial business scenario type is determined as the target resource node group. If the target business scenario type is different from the initial business scenario type, it is routed to the default general resource node group.
[0036] It is understandable that dividing a data warehouse cluster into multiple resource node groups and obtaining data processing requests do not have a strict sequential execution order. Resource node groups can be divided first, data processing requests can be obtained first, or resource node groups can be divided and data processing requests can be obtained simultaneously. This application does not impose any restrictions on this.
[0037] In step S130, the overall load value of the data warehouse cluster at the target time is determined.
[0038] In one embodiment of this application, normalized load metrics of each resource node in the data warehouse cluster at a target time are obtained; the load metrics of the multiple dimensions are weighted and summed according to preset weight coefficients to calculate the comprehensive load value at the target time. The load metrics of the multiple dimensions include CPU utilization, memory utilization, disk I / O utilization, network I / O utilization, and the length of the queue of tasks to be processed.
[0039] In one embodiment of this application, historical load indicators of multiple dimensions are obtained within a preset time window with the target time as the endpoint; based on the historical load indicators, the average load indicator of each dimension within the preset time window is calculated to smooth instantaneous fluctuations and improve the stability of load assessment; the average load indicators of each dimension are weighted and summed according to preset weight coefficients to calculate the comprehensive load value at the target time.
[0040] In this embodiment, by sensing the comprehensive load status at the target time, the core control algorithm is dynamically adjusted, achieving more refined and stable traffic control than the traditional single threshold method. Specifically, a multi-dimensional load assessment model continuously monitors the load metrics of the data warehouse cluster and each resource node. The comprehensive load value of each resource node at the target time t is denoted as... This value can be calculated by weighting the various indicators: Equation (1) in, The total load value at the target time t. These represent the CPU utilization (central processing unit utilization), memory utilization, disk I / O utilization, network I / O utilization, and the length of the task queue after normalization using a preset saturation function at the target time t, respectively. These are the preset weighting coefficients for CPU utilization, memory utilization, disk I / O utilization, network I / O utilization, and the length of the pending task queue, respectively, and they all satisfy the normalization condition. The preset weighting coefficients can be configured according to the resource sensitivity of different business scenarios, making the load assessment more closely match the actual business characteristics. The preset weighting coefficients are stored in a preset location (e.g., the configuration center) and are dynamically retrieved based on the identifier of the business scenario type.
[0041] Furthermore, the average value of the metrics within a sliding time window (i.e., a preset time window) can be used to calculate the load, smoothing out instantaneous peaks and obtaining a more stable load assessment. Overall Load Value It can be calculated using the following formula: Equation (2) in, The total load value is the target time t, where T is the length of the sliding time window (or window size), which is the length of the preset time window and is a time length. , , , and They are time points CPU utilization, memory utilization, disk I / O utilization, network I / O utilization, and pending task queue length at any given time. The value of is from target time tT to target time t; These are the preset weighting coefficients for CPU utilization, memory utilization, disk I / O utilization, network I / O utilization, and the length of the pending task queue, respectively, and they all satisfy the normalization condition. .
[0042] In step S140, if the overall load value is greater than the pre-emptive trigger threshold and there is a preset business scenario type in the target business scenario type, a dedicated resource pool for the target task corresponding to the preset business scenario type is determined from the target resource node group. After the target task is completed, the dedicated resource pool is dynamically released.
[0043] In this embodiment, a dedicated resource pool for the target task corresponding to a preset business scenario type is determined from the target resource node group. Resource nodes in the dedicated resource pool are locked, and the resource allocation mapping table is updated. After the target task is completed, the dedicated resource pool is dynamically released. Specifically, several idle resource nodes are selected from the target resource node group, and the node status labels in the resource directory service are modified to mark them as "dedicated locked." The routing mapping table is then updated so that requests other than those for the preset business scenario type cannot be routed to the locked resource nodes.
[0044] In this embodiment, high-priority services are defined in advance according to requirements. For example, services with a priority higher than a preset priority are defined as high-priority services. Taking a preset service scenario type as a high-priority service as an example, the target task corresponding to the preset service scenario type is a high-priority task. That is, when the comprehensive load value is greater than the pre-occupancy trigger threshold and there is a high-priority service, a dedicated resource pool for the high-priority task is determined from the target resource node group. After the high-priority task is completed, the dedicated resource pool is dynamically released.
[0045] In one embodiment of this application, after the dedicated resource pool is dynamically released, a resource release event is triggered; in response to the resource release event, the resource quotas of other running tasks are automatically adjusted according to a preset adjustment strategy to utilize the released resources for elastic scaling, wherein the preset adjustment strategy is determined based on task priority and / or the resources required by the task.
[0046] In this embodiment, if a preset adjustment strategy is determined based on task priority, the preset adjustment strategy is to determine the resource quota ratio of other tasks based on the task priority, and then determine the resource quota of other tasks based on the resource quota ratio. If a preset adjustment strategy is determined based on the resources required by a task, the preset adjustment strategy is to determine the resource quota ratio of other tasks based on the resources required by the task, and then determine the resource quota of other tasks based on the resource quota ratio. If a preset adjustment strategy is determined based on both task priority and the resources required by the task, the preset adjustment strategy is to determine the resource quota ratio of other tasks based on the task priority, the first weight corresponding to the task priority, the resources required by the task, and the second weight corresponding to the resources required by the task, and then determine the resource quota of other tasks based on the resource quota ratio. The number of other tasks can be one or more, the task priority and the resource quota ratio have a first correspondence, and the resources required by the task and the resource quota ratio have a second correspondence.
[0047] In this embodiment, after the dedicated resource pool is released, a resource release event is triggered, enabling other running tasks (such as low-priority tasks) other than the target task corresponding to the preset business scenario type to automatically and elastically expand to utilize the released resources.
[0048] In one embodiment of this application, after determining the overall load value of the data warehouse cluster at the target time, the target key parameters for calculating the target write frequency are determined according to the predefined range in which the overall load value is located; the target write frequency is obtained by calculation based on the actual write frequency at the current time, the overall load value, and the target key parameters; and the data write frequency is adjusted according to the target write frequency.
[0049] In one embodiment of this application, the target key parameters for calculating the target write frequency are determined based on the predefined range in which the overall load value lies. This includes: pre-setting a warning range for the overall load; determining different predefined ranges based on the warning range; and pre-setting the target key parameters corresponding to each predefined range; comparing the overall load value with the warning range; determining the predefined range in which the overall load value lies based on the comparison result; and then determining the target key parameters for calculating the target write frequency. Different predefined ranges correspond to different adjustment intensities to avoid traffic oscillations.
[0050] In this embodiment, based on the comprehensive load value at the target time The adaptive write frequency adjustment algorithm dynamically adjusts the target key parameters used to calculate the target write frequency within different predefined intervals, thereby achieving fine-grained, graded adjustment. These target key parameters include the target load and gain coefficient. The target write frequency is dynamically calculated using the following formula. : Equation (3) in, It is the target write frequency at the target time. It is the actual write frequency at the current moment. It is the comprehensive load value at the target time t. It is the preset ideal target load (i.e., target load), representing the optimal operating point that the system is expected to maintain. This is the variable overall gain coefficient (i.e., the gain coefficient).
[0051] The target write frequency is calculated. The process also includes: determining whether the target write frequency is less than a preset minimum write frequency threshold; if it is less, the target write frequency is forcibly set to the minimum write frequency threshold. This prevents invalid negative values from being calculated under extremely high loads, ensuring the stability of flow control.
[0052] The tiered adjustment strategy can be implemented in the following ways: First, configure the overall load. Warning range Different predefined intervals are determined based on the warning range; for example, intervals smaller than the minimum threshold of the warning range are included. The intervals within the normal range are divided into normal ranges, and the intervals within the warning range are divided into warning ranges, with the intervals exceeding the highest threshold of the warning range being classified as warning ranges. The interval is divided into high-load intervals, and the predefined interval in which the comprehensive load value falls is determined based on the comparison between the comprehensive load value and the warning range. This can also be understood as dividing the comprehensive load value... Divided into normal range ( ), warning interval ( ) and high load range ( ).
[0053] Based on the predefined range of the overall load value, determine the target key parameters used to calculate the target write frequency, including: When the comprehensive load value When the load is within the normal range, actively increase throughput and make full use of resources. The tiered backpressure controller adopts an active strategy parameter, that is, sets a higher local target load. and a large local gain coefficient This triggers strong positive feedback, allowing the write frequency to increase rapidly.
[0054] When the comprehensive load value When the system is in the warning range, it adjusts smoothly to seek a balance between throughput and stability. The tiered back pressure controller switches to robust strategy parameters and sets an appropriate local target load. And lower the local gain coefficient This makes the system's response to load changes smoother, achieving gentle positive feedback or weak negative feedback, effectively preventing system oscillation.
[0055] When the comprehensive load value When in a high-load range, to prioritize system stability, the tiered backpressure controller automatically switches to conservative strategy parameters and sets a lower local target load. The system then restores the local gain coefficient to its default value. At this point, because the actual load is higher than the target load, the formula calculation result will trigger negative feedback, systematically reducing the write frequency and forcibly reducing the voltage on the node. Simultaneously, the system will activate a resource pre-emption mechanism during this phase to ensure the normal execution of critical tasks.
[0056] It is understandable that the local target load in this embodiment That is, the target load in the aforementioned embodiments. The local gain coefficient in this embodiment That is, the gain coefficient in the aforementioned embodiments. Target load and gain coefficient The values in different predefined ranges can be preset according to requirements.
[0057] Through the closed-loop control described above, the actual load value can be automatically and continuously adjusted. To target load By driving closer, a dynamic balance is achieved between the write frequency and the real-time load of the data warehouse, maximizing throughput while ensuring system stability.
[0058] In an exemplary embodiment, the resource allocation method for a data warehouse includes at least steps S1 to S3, which are described in detail below: In step S1, resource node groups are initialized to obtain multiple resource node groups corresponding to the initial business scenario type; data processing requests from clients are received, the target business scenario type of the data processing request is identified, dynamic routing is performed according to the target business scenario type, and it is determined whether resource pre-allocation is required based on the comprehensive load value and the target business scenario type.
[0059] In step S2, the target key parameters for calculating the target write frequency are determined based on the predefined range of the comprehensive load value, and then the target write frequency is calculated. The data write frequency of the upstream system is adjusted according to the target write frequency.
[0060] In step S3, after the target task corresponding to the target business scenario type is completed, the dedicated resource pool is released, triggering a resource release event, so that other running tasks can automatically and elastically expand to utilize the released resources.
[0061] Figure 2 This is a flowchart illustrating a resource allocation method for a data warehouse, as shown in another exemplary embodiment of this application. (Refer to...) Figure 2 As shown, the resource allocation method for a data warehouse specifically includes steps S11 to S32, which are described in detail below: In step S11, a data processing request for a real-time risk control query request (high priority request) arrives at the unified access entry module 401.
[0062] In step S12, the request routing and dynamic allocation module 402 identifies the target business scenario type of the real-time risk control query request. If it determines that the real-time risk control task corresponding to the target business scenario type is a high-priority real-time task, the real-time risk control query request is routed to the real-time computing node group in the data warehouse cluster 403. Additionally, the load monitoring module 407 detects that the system is under high load and notifies the resource pre-allocation and dynamic release module 405.
[0063] In step S13, the resource pre-allocation and dynamic release module 405 checks the global resource catalog. If the service does not yet have pre-allocated resources, it immediately allocates resources from the real-time computing node group according to a preset ratio, and creates and records a dedicated resource pool.
[0064] In step S14, the real-time risk control task is scheduled and executed within a pre-allocated dedicated resource pool, thus obtaining isolated resource protection.
[0065] In step S21, the load monitoring module 407 monitors the overall load value of the batch processing node group in real time. If the value continues to rise and exceeds the lower limit threshold of the warning range, it enters the warning range and the data is pushed to the graded back pressure controller 404.
[0066] In step S22, the graded back pressure controller 404 selects parameters according to the strategy of the warning interval. and Substitute the values into the core algorithm to calculate the target write frequency, which is lower than the current write frequency. .
[0067] In step S23, the graded back pressure controller 404 sends this frequency reduction command to the upstream data writing service, which then reduces the data push rate, thereby controlling and gradually reducing the load on the batch processing node group.
[0068] In step S31, after the aforementioned real-time risk control task is completed, the resource pre-allocation and dynamic release module 405 receives the completion notification, then releases its dedicated resource pool and broadcasts the resource release event; after the resource scheduler listens to the event, it recalculates the priority score of each task to be run; based on the score result, it sends an expansion instruction to the computing engine corresponding to the task with the highest score in order to utilize the released resources.
[0069] In step S32, after the resources are released, the large report job running in the batch processing node group listens to the resource release event, and its computing engine immediately requests more computing resources from the resource manager to achieve automatic elastic expansion, quickly utilize the released resources, and accelerate report generation.
[0070] Through the coordinated operation of the above processes, this embodiment effectively solves the resource contention and interference problems in multi-service scenarios, and achieves performance assurance for critical services and dynamic and efficient utilization of cluster resources. It is understood that step S1 in this embodiment includes at least steps S11 to S14, step S2 includes at least steps S21 to S23, and step S3 includes at least steps S31 to S32.
[0071] In some embodiments, the data warehouse cluster is partitioned into resource node groups for different initial business scenario types; data processing requests are received through a unified access entry module; the target business scenario type of the data processing request is identified, and the request is dynamically routed to the corresponding resource node group for processing; and the overall load value of the data warehouse is continuously monitored. Resource pre-allocation is triggered only when the overall system load exceeds the pre-allocation trigger threshold and a high-priority task arrives, and resources are dynamically released after the high-priority task completes; based on the overall load value... The target key parameters are dynamically adjusted and the target write frequency is calculated based on the different predefined intervals in which it is located. Based on this, the data write frequency of the upstream system is adjusted. Key target parameters include the target load and gain coefficient. That is, when calculating the target write frequency, the target load and gain coefficient are dynamically adjusted within different predefined ranges.
[0072] Figure 3 This is a block diagram illustrating a resource allocation system for a data warehouse, as shown in an exemplary embodiment of this application. Figure 3 As shown, this exemplary resource allocation system for a data warehouse includes a unified access entry module, a resource node partitioning module, a request routing and dynamic allocation module, a load monitoring module, and a resource pre-allocation and dynamic release module.
[0073] The system comprises the following modules: a unified access entry module for receiving data processing requests, each carrying a target business scenario type; a resource node partitioning module for dividing the data warehouse cluster into multiple resource node groups corresponding to the initial business scenario type; a request routing and dynamic allocation module, connected to the unified access entry module and the resource node partitioning module, for dynamically routing data processing requests to the target resource node group based on the matching result between the target business scenario type and the initial business scenario type; a load monitoring module for monitoring the overall load value of the data warehouse cluster at the target time; and a resource pre-allocation and dynamic release module for determining a dedicated resource pool from the target resource node group for the target task corresponding to the preset business scenario type if the overall load value exceeds the pre-allocation trigger threshold and a preset business scenario type exists in the target business scenario type, and dynamically releasing the dedicated resource pool after the target task is completed.
[0074] In this embodiment, the unified access entry module serves as the sole external service endpoint of the data warehouse cluster, receiving data processing requests. The resource node partitioning module divides the nodes of the data warehouse cluster into multiple resource node groups based on both data characteristics and business priorities. The request routing and dynamic allocation module, connected to the unified access entry module and the resource node partitioning module, parses the business scenario characteristics (including data characteristics and business priorities) of the data processing requests and dynamically routes them to the corresponding target resource node groups. The load monitoring module continuously monitors the overall load value of the data warehouse cluster. The resource pre-allocation and dynamic release module pre-allocates a dedicated resource pool from the target resource node group when the overall load value exceeds the pre-allocation trigger threshold and a preset business scenario type (e.g., high-priority business, i.e., high-priority task arrival) exists in the target business scenario type. This pool is used by the target task corresponding to the preset business scenario type (e.g., the high-priority task). When the resource pre-allocation and dynamic release module receives a heartbeat timeout signal or a task completion ACK (Acknowledge character) signal from the target task, it determines that the task is complete and then dynamically releases the dedicated resource pool.
[0075] In this embodiment, the resource pre-allocation and dynamic release module is also used to trigger a resource release event after releasing the dedicated resource pool, so that the low-priority tasks that are running can automatically and elastically expand to utilize the released resources.
[0076] In this embodiment, the comprehensive load value is calculated using equation (2) from the aforementioned embodiment. The specific calculation method is described in the corresponding description in the foregoing embodiments, and will not be repeated here.
[0077] In one embodiment of this application, the exemplary resource allocation system for a data warehouse further includes a hierarchical back pressure controller connected to a load monitoring module. The controller is used to determine the target key parameters for calculating the target write frequency based on the predefined range in which the comprehensive load value is located, calculate the target write frequency based on the actual write frequency at the current moment, the comprehensive load value, and the target key parameters, and adjust the data write frequency according to the target write frequency.
[0078] In this embodiment, the graded back pressure controller is connected to the load monitoring module and is used to dynamically adjust the target key parameters of the target write frequency according to the different predefined intervals in which the comprehensive load value is located, and adjust the data write frequency of the upstream system accordingly. The target key parameters include the target load and the gain coefficient.
[0079] In this embodiment, the graded back pressure controller calculates the target write frequency using the aforementioned equation (3). Furthermore, the graded back pressure controller dynamically adjusts the target load in the formula according to the different predefined ranges in which the comprehensive load value is located. and gain coefficient The value of .
[0080] The tiered strategy of the tiered backpressure controller includes: when the comprehensive load value When within the normal range, use a higher target load. and a large gain coefficient To increase write frequency; when the overall load value When the warning range is in effect, a moderate target load should be adopted. and a smaller gain coefficient To adjust gradually; when the overall load value When in a high-load range, use a lower target load. This is to reduce the write frequency. It's understandable that the normal range, warning range, and high load range are all predefined ranges, with their ranges predefined according to requirements. Furthermore, the target load and gain coefficient values corresponding to each predefined range are also pre-set according to requirements.
[0081] It should be noted that, Figure 3The lines in the diagram indicate that two modules are connected, but the absence of a line does not mean that two modules are not connected.
[0082] In some embodiments, one of the core features of a resource allocation system for a data warehouse is its multi-dimensional dynamic resource partitioning and conflict resolution mechanism. Specifically, the resource node partitioning module is used to split the nodes of the data warehouse cluster into at least three independent resource node groups based on both data characteristics and business priorities. Furthermore, when new business scenario types emerge, nodes can be repartitioned according to data characteristics and business priorities to achieve physical / logical isolation of resources for different business scenarios. Data characteristics include data size, data timeliness, and data processing type (ETL / query); business priorities assign different priority weights to different business lines or task types based on predefined business rules.
[0083] The resource pre-allocation and dynamic release module resolves resource conflicts when the system is under high load (i.e., the overall load exceeds the pre-allocation trigger threshold) and high-priority tasks are present. Its triggering requires two conditions: high system load and the presence of high-priority business. When both conditions are met simultaneously, the system automatically triggers resource pre-allocation in its target resource node group. This mechanism locks a dedicated resource pool for high-priority tasks, ensuring that core business processes receive necessary resources even during periods of system overload. During resource pre-allocation, low-priority batch processing tasks can only use the remaining non-pre-allocated resources. Once a high-priority task completes, the system immediately and dynamically releases its pre-allocated resources. This module avoids resource idleness and improves overall resource utilization.
[0084] The unified access entry module serves as the sole external service endpoint for the data warehouse, receiving all data processing requests and data application query requests from clients or upstream systems.
[0085] The request routing and dynamic allocation module is connected to the unified access entry module and the resource node partitioning module. It is used to parse the received data processing requests and dynamically route and allocate them to the corresponding target resource node groups for processing based on the business scenario characteristics associated with the data processing requests.
[0086] Figure 4 This is a schematic diagram illustrating the module composition and data flow of a resource allocation system for a data warehouse, as shown in an exemplary embodiment of this application. (Reference) Figure 4 As shown, the resource allocation system for the data warehouse includes a unified access entry module 401, a request routing and dynamic allocation module 402, a data warehouse cluster 403, a hierarchical backpressure controller 404, a resource pre-allocation module 405, a resource node partitioning module 406, and a load monitoring module 407. Solid lines represent request and data flows, while dashed lines represent monitoring and control flows.
[0087] The unified access entry module 401 serves as the sole external service endpoint for the data warehouse cluster, configured with a unified network address and port to receive data processing requests from all external clients. The request routing and dynamic allocation module 402, connected to the unified access entry module 401, contains a pre-set routing policy table. This table parses the business scenario characteristics of data processing requests, determines the target business scenario type based on these characteristics, and dynamically distributes the data processing requests to the corresponding resource node group (target resource node group) within the data warehouse cluster 403. The data warehouse cluster 403, composed of multiple physical or logical computing nodes, is the core computing unit for executing data processing and querying tasks. The resource node partitioning module 406, during system initialization or maintenance, partitions the nodes of the data warehouse cluster 403 into different resource node groups based on a pre-defined business scenario plan (considering both data characteristics and business priorities). Based on business needs, the resources can be divided into: a batch processing resource group for handling high-throughput, high-latency full data jobs; a real-time computing group for handling low-latency, short-cycle incremental data jobs; and an interactive query group for handling high-concurrency point queries or complex analytical queries. If other business types emerge later, node groups can be dynamically divided according to their requirements. The resource pre-allocation and dynamic release module 405, connected to the request routing and dynamic allocation module 402 and the data warehouse cluster 403, maintains a global resource directory to record and manage dedicated resource pools pre-allocated for high-priority business tasks. When the resource pre-allocation and dynamic release module 405 receives a high-priority task and the system's overall load value is high (i.e., the overall load value is greater than the pre-allocation trigger threshold), it allocates resources from the target resource node group, forming a dedicated resource pool and locking it. When the high-priority task is completed, the resource pool is released, triggering a resource release event. The load monitoring module 407, through monitoring agents deployed on each node of the data warehouse cluster 403, continuously collects system performance indicators, i.e., the overall load value. The graded back pressure controller 404 is connected to the load monitoring module 407 and subscribes to the comprehensive load value published by the load monitoring module 407. This graded back pressure controller has multiple load thresholds to divide different predefined ranges, according to... Within the predefined range, the graded back pressure controller 404 dynamically selects different target loads. and gain coefficient The parameter combination is used to determine the target load. and gain coefficient The target write frequency is calculated by combining the parameters. This will allow for adjustments to the data writing frequency of the upstream system.
[0088] It should be noted that the resource allocation system for data warehouses provided in the above embodiments and the resource allocation method for data warehouses provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the resource allocation system for data warehouses provided in the above embodiments can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not a limitation here.
[0089] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a resource allocation method for a data warehouse as described in any of the above embodiments.
[0090] Figure 5 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 5 The computer system 500 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of the embodiments of this application.
[0091] like Figure 5 As shown, the computer system 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes, such as executing the methods provided in the various embodiments above, based on a program stored in Read-Only Memory (ROM) 502 or a program loaded from Storage Unit 508 into Random Access Memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An Input / Output (I / O) interface 505 is also connected to the bus 504.
[0092] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.
[0093] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs various functions defined in the system of this application.
[0094] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.
[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0096] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.
[0097] Another aspect of this application provides a computer-readable storage medium storing a computer program that is executed by a processor to implement a resource allocation method for a data warehouse as described in any of the above embodiments. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not incorporated into the electronic device.
[0098] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0099] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the resource allocation method for a data warehouse provided in the various embodiments described above.
[0100] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.
[0101] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.
[0102] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.
Claims
1. A resource allocation method for a data warehouse, characterized in that, include: Obtain a data processing request, wherein the data processing request carries the target business scenario type; Based on the matching result between the target business scenario type and the initial business scenario type, the data processing request is dynamically routed to the target resource node group, wherein the data warehouse cluster is pre-divided into multiple resource node groups corresponding to the initial business scenario type; Determine the overall load value of the data warehouse cluster at the target time; If the overall load value is greater than the pre-emptive trigger threshold, and there is a preset business scenario type among the target business scenario types, a dedicated resource pool for the target task corresponding to the preset business scenario type is determined from the target resource node group. After the target task is completed, the dedicated resource pool is dynamically released.
2. The resource allocation method for a data warehouse according to claim 1, characterized in that, Determining the overall load value of the data warehouse cluster at the target time includes: Obtain load metrics for each resource node in the data warehouse cluster across multiple dimensions at the target time. The load indicators of the multiple dimensions are weighted and summed according to preset weighting coefficients to calculate the comprehensive load value at the target time.
3. The resource allocation method for a data warehouse according to claim 2, characterized in that, Determining the overall load value of the data warehouse cluster at the target time also includes: Obtain historical load metrics across multiple dimensions within a preset time window ending at the target time. Based on the historical load metrics, calculate the average load metrics for each of the multiple dimensions within the preset time window; The average load index of each dimension is weighted and summed according to the preset weighting coefficients to calculate the comprehensive load value at the target time.
4. The resource allocation method for a data warehouse according to claim 1, characterized in that, After determining the overall load value of the data warehouse cluster at the target time, the method further includes: Based on the predefined range in which the comprehensive load value is located, determine the target key parameters used to calculate the target write frequency; The target write frequency is calculated based on the actual write frequency at the current moment, the overall load value, and the target key parameters. Adjust the data writing frequency according to the target writing frequency.
5. The resource allocation method for a data warehouse according to claim 4, characterized in that, Based on the predefined range in which the comprehensive load value falls, the target key parameters used to calculate the target write frequency are determined, including: The warning range of the comprehensive load is preset, different predefined intervals are determined according to the warning range, and the target key parameters corresponding to each predefined interval are preset. The overall load value is compared with the warning range, and the predefined interval in which the overall load value is located is determined based on the comparison result, thereby determining the target key parameters for calculating the target write frequency.
6. The resource allocation method for a data warehouse according to claim 4, characterized in that, The target write frequency is calculated using the following formula: in, It is the target write frequency. It is the actual write frequency at the current moment. This is the overall load value. It is the target load. It is the gain coefficient, and the target key parameters include the target load and the gain coefficient.
7. The resource allocation method for a data warehouse according to any one of claims 1 to 6, characterized in that, After dynamically releasing the dedicated resource pool, the method further includes: Trigger resource release event; In response to the resource release event, the resource quotas of other running tasks are automatically adjusted according to a preset adjustment strategy to utilize the released resources for elastic scaling. The preset adjustment strategy is determined based on task priority and / or the resources required by the task.
8. A resource allocation system for a data warehouse, characterized in that, include: The unified access entry module is used to receive data processing requests, which carry the target business scenario type. The resource node partitioning module is used to divide the data warehouse cluster into multiple resource node groups corresponding to the initial business scenario type; The request routing and dynamic allocation module is connected to the unified access entry module and the resource node partitioning module, and is used to dynamically route the data processing request to the target resource node group according to the matching result of the target business scenario type and the initial business scenario type. The load monitoring module is used to monitor the overall load value of the data warehouse cluster at a target time. The resource pre-allocation and dynamic release module is used to determine a dedicated resource pool from the target resource node group for the target task corresponding to the preset business scenario type if the comprehensive load value is greater than the pre-allocation trigger threshold and there is a preset business scenario type among the target business scenario types, and dynamically release the dedicated resource pool after the target task is completed.
9. The resource allocation system for a data warehouse according to claim 8, characterized in that, It also includes a graded back pressure controller, which is connected to the load monitoring module. It is used to determine the target key parameters for calculating the target write frequency based on the predefined range in which the comprehensive load value is located, calculate the target write frequency based on the actual write frequency at the current moment, the comprehensive load value, and the target key parameters, and adjust the data write frequency according to the target write frequency.
10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the resource allocation method for a data warehouse as described in any one of claims 1 to 7.