Resource scheduling methods under lake warehouse converged architecture
By constructing query feature vectors and machine learning models, and combining model predictive control and adaptive weight adjustment mechanisms, the dynamic adaptability and global optimization problems of scheduling methods under the lake warehouse fusion architecture are solved, achieving efficient resource scheduling and system performance improvement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NORTHEASTERN UNIV CHINA
- Filing Date
- 2026-02-06
- Publication Date
- 2026-05-26
AI Technical Summary
The existing query scheduling methods under the lake warehouse fusion architecture lack dynamic adaptability and global optimization capabilities, making it difficult to achieve real-time resource adaptation for multiple types of query tasks in high-concurrency and load fluctuation scenarios, resulting in suboptimal scheduling decisions and system performance bottlenecks.
By constructing query feature vector representations and machine learning-based performance and resource consumption prediction models, combined with model predictive control mechanisms and contextual multi-armed slot machine models, a rolling optimization decision process is designed to achieve dynamic adjustment of system state perception and resource utilization. A sliding window update mechanism and an automatic scaling mechanism are adopted to improve the adaptability and efficiency of the scheduling strategy.
It achieves intelligent resource scheduling under multiple query tasks, can accurately estimate query runtime and resource consumption, dynamically adapt to load changes, improve system performance and resource utilization, and ensure a balance between query latency and resource utilization.
Smart Images
Figure CN122086935A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of resource scheduling technology, and in particular to a resource scheduling method under a lake-warehouse fusion architecture. Background Technology
[0002] With the continuous development of big data and cloud computing technologies, enterprise-level data management systems are gradually evolving into a lakeware-integrated architecture. This architecture combines the flexible storage capabilities of a data lake with the high-performance query characteristics of a data warehouse, supporting unified management, storage, and analysis of structured, semi-structured, and unstructured data. In a lakeware-integrated architecture, the system needs to handle multiple workload types simultaneously, including online analytical processing, online transaction processing, streaming computing, and batch processing jobs. In this mixed workload environment, different types of query tasks vary significantly in terms of data access patterns, response time requirements, resource consumption characteristics, and execution priorities. Therefore, how to implement query admission control, intelligent query scheduling, and resource allocation adjustment within a unified framework to achieve real-time resource adaptation for various query tasks has become a key issue in improving system performance.
[0003] Existing query scheduling methods can be mainly divided into static rule-based query scheduling methods and learning-based query scheduling methods, as detailed below:
[0004] Query scheduling methods based on static rules determine the execution order of queries through predefined scheduling strategies. Their core lies in using fixed heuristic rules or priorities to sort query tasks. In typical solutions, first-come, first-served (FFS) is the most direct scheduling method, scheduling queries according to their arrival order. Another common method is short query priority: based on static indicators such as estimated execution time or number of operators, queries with shorter execution times are prioritized to reduce average response time. Priority-based scheduling pre-prioritizes queries of different types or business importance, ensuring that critical tasks (such as interactive queries or front-end business processes) are executed before back-end batch processing tasks. Furthermore, resource-type-based classification scheduling statically allocates query operators to different resource pools or execution nodes based on their computational characteristics (such as CPU-intensive or I / O-intensive) to achieve initial load isolation.
[0005] Learning-based query scheduling methods introduce machine learning models to automatically learn the mapping relationship between query features and system performance from historical execution data, thereby achieving intelligent scheduling decisions. Typical methods are as follows: LASER uses a multilayer perceptron to predict the access pattern of queries and designs a two-stage adaptive scheduling algorithm based on this: in the allocation stage, an adaptive greedy strategy is used to balance query similarity (based on L1 distance) and node load; in the selection stage, the buffer state is estimated based on exponential moving average, and buffer-friendly or low-cost queries are executed first; LSched is a reinforcement learning-based scheduler that extracts white-box physical plan features (such as operator type, data block, pipeline state, etc.) from the query plan and combines tree convolution and graph attention mechanisms for efficient encoding to dynamically predict operator scheduling order, pipeline depth and thread allocation strategy; Auto-WLM uses a lightweight XGBoost model to predict query execution time and resource requirements, and then divides queries into short queries and long queries, which are routed to dedicated queues and general queues respectively to achieve differentiated scheduling; Decima is a scheduling scheme based on reinforcement learning and graph neural networks, which is specifically used to optimize DAG-type jobs with stage dependencies. It automatically extracts query structure features through graph embedding technology, dynamically selects the stage to be scheduled and adjusts the job parallelism.
[0006] Rule-based scheduling methods lack dynamic adaptability, relying on predefined static policies as their core mechanism. These methods often struggle to perceive real-time load changes and dynamic interactions between query tasks during system operation. For example, under a short query-first policy, when the system continuously receives a large number of short queries, long queries may be delayed for extended periods, leading to the typical "starvation" phenomenon. Furthermore, a single static rule often fails to simultaneously address diverse scheduling objectives, such as response time, resource utilization, and fairness, thus limiting the overall performance improvement of scheduling.
[0007] In contrast, while learning-based scheduling methods possess a degree of adaptability, they still face numerous challenges. Query tasks are often submitted and executed concurrently, resulting in complex competition and interference relationships among tasks across multiple resources such as CPU, memory, cache, and I / O bandwidth. Existing methods typically struggle to adequately characterize the dynamic impact of resource contention and changes in internal system states on query performance, leading to suboptimal scheduling decisions in real-world high-concurrency, high-load-fluctuation scenarios. Furthermore, many learning-based schedulers still lack the ability to jointly model and optimize task characteristics, system resource states, and global optimization objectives over the long term, making it difficult to maintain optimal decisions. Summary of the Invention
[0008] To address the shortcomings of existing scheduling methods in terms of dynamic adaptability and global optimization capabilities, this invention proposes a resource scheduling method under a lake-warehouse fusion architecture. By introducing a query performance prediction model and a model prediction control mechanism, scheduling decisions can be dynamically adjusted based on the real-time system status and future trends in environments with multiple concurrent queries and resource contention, achieving synergistic optimization of query performance and system resource utilization.
[0009] On the one hand, the present invention provides a resource scheduling method under a lake-warehouse converged architecture, comprising the following steps:
[0010] Step 1: Construct the feature vector representation of the query;
[0011] Assuming a query The physical query plan contains several operator nodes. Where n is the number of operator nodes, and each operator node From a local eigenvector It means that among them If the feature dimension is used, then the query The query feature vector is defined as ,in This indicates that feature aggregation is performed using a vector concatenation function;
[0012] Step 2: Construct a machine learning-based prediction model for query performance and resource consumption;
[0013] Step 2.1: Introduce system and runtime state feature vectors ,in Represents the dimension of the feature vector;
[0014] Specifically, this means considering what is currently being executed in the system. A number of concurrent queries, where the system is a set of compute nodes in a lakeware converging architecture; each query Corresponding to a query feature vector , The concurrent query state feature vector is obtained by weighting and aggregating the feature vectors of each query through an attention mechanism. Considering the current system state, extract the system state feature vector. Concurrent query state feature vector With system state feature vector The data is then concatenated to form a complete system and runtime state feature vector. ;
[0015] Step 2.2: Use XGBoost to build a predictive model for query performance and resource consumption under concurrent conditions;
[0016] Specifically: Let the query performance and resource consumption prediction model be a function The parameter set of the model is The model output is the execution time of the query under concurrent conditions. and resource consumption vector :
[0017] ;
[0018] The model is trained to minimize the error between the predicted output and the actual observations; let the number of training samples be... , No. The observation execution time and resource consumption for each sample are as follows: and The objective function of the training process is:
[0019] ;
[0020] in, The loss function is the mean squared error method. For the optimal set of model parameters, Indicates the query in the nth training sample The query feature vector, This represents the system and runtime state feature vector corresponding to the nth training sample.
[0021] Step 3: Construct a query scoring function to evaluate the query tasks in the waiting queue, thereby forming a priority scheduling candidate set;
[0022] The specific query scoring function is as follows:
[0023] Suppose there are in the waiting queue Candidate queries For each query Its scoring function Defined as:
[0024] ;
[0025] in, and Let be the weighting coefficient, satisfying and ; For resource matching degree, Concurrency influencing factors;
[0026] Among them, resource matching degree This measures the degree of match between query resource requirements and the system's current remaining resources; let the query... The resource demand vector is The system's current remaining resource vector is The resource matching degree is then defined as:
[0027] ;
[0028] Concurrency influencing factors Evaluate the execution of query Q under the current system concurrency conditions. i Performance; design To query the estimated runtime under no concurrent interference, To predict query performance and resource consumption in the current concurrent environment using a model Based on the predicted runtime, the concurrency influencing factors are defined as follows:
[0029] ;
[0030] Ultimately, according to Sort all query tasks in the waiting queue and select queries with scores higher than a set threshold to form a priority scheduling candidate set;
[0031] Step 4: Construct a rolling optimization decision-making process based on a model predictive control mechanism;
[0032] Specifically, a model prediction control mechanism is introduced to perform rolling optimization decisions on the submission order of candidate queries, as designed below:
[0033] Let the current scheduling period be The set of candidate queries in the waiting queue is Each query The arrival time is The prediction time domain is defined as follows: That is, in the future Optimize within each scheduling cycle; in each scheduling cycle Rolling optimization decision is defined as finding the optimal query submission sequence. ,in, Indicates the first The queries submitted from the candidate query set in each scheduling cycle minimize the objective function of the rolling optimization decision:
[0034] ;
[0035] in, Indicates query During the scheduling period The predicted runtime of the submitted execution is determined by the model. Calculations show that Represents the scheduling period Start time, For query Arrival time; and Let be the weighting coefficient, satisfying and ;
[0036] Solving the objective function of the rolling optimization decision yields the optimal sequence. Then, the first query is submitted for execution. Once the query is actually submitted, the system status information is updated. At the same time, if a new query arrives, the system will re-execute steps 3 and 4, iteratively selecting the next optimal query to submit, thus forming a continuously rolling optimization decision-making process.
[0037] Step 5: Construct an adaptive weight adjustment mechanism based on the context-based multi-armed slot machine model;
[0038] The adaptive weight adjustment mechanism uses the system state as context information and dynamically adjusts the weight parameters in the scoring function and the objective function of the rolling optimization decision through a contextual multi-armed slot machine model, as detailed below:
[0039] Define the system state vector As contextual input, t is the decision time, p is the state feature dimension, and S t In order to make decisions The observed values, The state feature dimension is used. The weight configuration action space is a discrete set. Each action This represents a set of weight combinations that satisfy the constraints. , , and ;
[0040] At every decision moment The system performs the following steps:
[0041] Measure the current system state vector ;
[0042] Select an action based on the context of the multi-armed slot machine model. ;
[0043] Select weights Objective functions applied to scoring functions and rolling optimization decisions;
[0044] Execute query scheduling decisions and calculate average query latency and average resource utilization.
[0045] Calculate the reward function And update the context multi-armed slot machine model;
[0046] The reward function The design is as follows: For decision-making moments Average resource utilization rate For decision-making moments The average query latency, then the reward function for:
[0047] ;
[0048] The contextual multi-armed slot machine model learns the expected reward of state-action pairs through historical interaction data and employs an exploration-utilization balancing strategy to select actions to maximize the cumulative reward of discounts. ,in As a discount factor, Indicates relative to the current decision time Time step offset;
[0049] Step 6: Construct a sliding window design method for model updates oriented towards dynamic data;
[0050] To address the dynamic changes in data distribution and query patterns over time within the lake warehouse system, a sliding window-based data sampling and model update mechanism is designed: by setting a sample window, incremental sampling and online training are performed on the most recent W queries and system operation data to achieve continuous adaptive updates of the query performance and resource consumption prediction model;
[0051] Step 7: Build a system utilization monitor and an automatic scaling mechanism;
[0052] The system features a utilization monitor that collects real-time system performance metrics, including query count, CPU utilization, memory utilization, bandwidth utilization, and task queue length. An automatic scaling mechanism is also implemented, dynamically triggering horizontal scaling of computing resources when the task queue duration exceeds a preset threshold. This alleviates query backlog and maintains system processing capacity stability. Furthermore, resource reclamation is supported during load shedding phases, thereby improving resource utilization efficiency.
[0053] On the other hand, this application proposes a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the resource scheduling method under the lakehouse fusion architecture.
[0054] Thirdly, this application proposes a computer program product, including a computer program or instructions that, when executed by a processor, implement the resource scheduling method under the lakeware fusion architecture described above.
[0055] The beneficial effects of adopting the above technical solution are as follows:
[0056] This invention provides a resource scheduling method under a lakeware fusion architecture. Compared with existing technologies, the proposed solution achieves intelligent resource scheduling under this architecture. By introducing a concurrent query performance prediction model with system state awareness, the runtime and resource consumption of a query can be accurately estimated before submission. Based on this, the invention introduces a model predictive control mechanism into the query scheduling process. This mechanism can comprehensively consider system resource constraints, query latency targets, and future load trends within a finite time domain, and continuously solve for the optimal scheduling scheme. Furthermore, this invention designs an adaptive weight adjustment mechanism based on a context-based multi-armed slot machine, which can automatically adjust the optimization weights between resource utilization and query latency according to the real-time system state. Under this self-learning mechanism, the system can continuously optimize itself based on operational feedback, achieving dynamic adaptation of the scheduling strategy to load changes. Finally, this invention implements a dynamic model update and automatic resource expansion mechanism. On the one hand, it adopts an incremental learning mechanism based on a sliding window, enabling the prediction model to adapt to changes in data distribution and query patterns; on the other hand, by monitoring the system load in real time and automatically triggering resource expansion when queue latency exceeds a threshold, the system's elastic adaptability to dynamic loads is improved. Attached Figure Description
[0057] Figure 1 The overall flowchart of the resource scheduling method under the lake-warehouse fusion architecture in this embodiment of the invention. Detailed Implementation
[0058] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.
[0059] Example 1:
[0060] On the one hand, this invention provides a resource scheduling method under a lake-warehouse fusion architecture, such as... Figure 1 As shown, it includes the following steps:
[0061] Step 1: Construct the feature vector representation of the query;
[0062] Assuming a query The physical query plan contains several operator nodes. Where n is the number of operator nodes, and each operator node From a local eigenvector It means that among them This is a feature dimension, including operator type, data distribution, parallelism, data access, etc. Then the query... The query feature vector is defined as ,in This indicates that feature aggregation is performed using a vector concatenation function;
[0063] Step 2: Construct a machine learning-based prediction model for query performance and resource consumption;
[0064] Step 2.1: Obtain the query feature vector Subsequently, in order to perceive the system state, a system and runtime state feature vector is further introduced. ,in Represents the dimension of the feature vector;
[0065] Specifically, this means considering what is currently being executed in the system. A number of concurrent queries, where the system is a set of compute nodes in a lakeware converging architecture; each query Corresponding to a query feature vector , The concurrent query state feature vector is obtained by weighting and aggregating the feature vectors of each query through an attention mechanism. Considering the current system state, extract the system state feature vector. This includes system status metrics such as CPU utilization and memory utilization. The concurrent query status feature vector... With system state feature vector The data is then concatenated to form a complete system and runtime state feature vector. ;
[0066] Step 2.2: Use XGBoost to build a predictive model for query performance and resource consumption under concurrent conditions;
[0067] Specifically: Let the query performance and resource consumption prediction model be a function The parameter set of the model is The model output is the execution time of the query under concurrent conditions. and resource consumption vector :
[0068] ;
[0069] The model is trained to minimize the error between the predicted output and the actual observations; let the number of training samples be... , No. The observation execution time and resource consumption for each sample are as follows: and The objective function of the training process is:
[0070] ;
[0071] in, The loss function is the mean squared error method. For the optimal set of model parameters, Indicates the query in the nth training sample The query feature vector, This represents the system and runtime state feature vector corresponding to the nth training sample.
[0072] Step 3: Construct a query scoring function. This function needs to comprehensively consider resource utilization efficiency and system concurrency characteristics to evaluate the query tasks in the waiting queue, thereby forming a priority scheduling candidate set.
[0073] The specific query scoring function is as follows:
[0074] Suppose there are in the waiting queue Candidate queries For each query Its scoring function Defined as:
[0075] ;
[0076] in, and Let be the weighting coefficient, satisfying and ; For resource matching degree, Concurrency influencing factors;
[0077] Among them, resource matching degree This measures the degree of match between query resource requirements and the system's current remaining resources; let the query... The resource demand vector is The system's current remaining resource vector is The resource matching degree is then defined as:
[0078] ;
[0079] Concurrency influencing factors Evaluate the execution of query Q under the current system concurrency conditions. i Performance; design To query the estimated runtime (estimated using the Stage model) under conditions of no concurrent interference. To predict query performance and resource consumption in the current concurrent environment using a model Based on the predicted runtime, the concurrency influencing factors are defined as follows:
[0080] ;
[0081] Ultimately, according to Sort all query tasks in the waiting queue and select queries with scores higher than a set threshold to form a priority scheduling candidate set;
[0082] Step 4: Construct a rolling optimization decision-making process based on a model predictive control mechanism;
[0083] Specifically, a model predictive control mechanism is introduced to perform rolling optimization decisions on the submission order of candidate queries. The goal of this mechanism is to minimize the query runtime over several future scheduling cycles, as designed below:
[0084] Let the current scheduling period be The set of candidate queries in the waiting queue is Each query The arrival time is The prediction time domain is defined as follows: That is, in the future Optimize within each scheduling cycle; in each scheduling cycle Rolling optimization decision is defined as finding the optimal query submission sequence. ,in, Indicates the first The queries submitted from the candidate query set in each scheduling cycle minimize the objective function of the rolling optimization decision:
[0085] ;
[0086] in, Indicates query During the scheduling period The predicted runtime of the submitted execution is determined by the model. Calculations show that Represents the scheduling period Start time, For query Arrival time; and Let be the weighting coefficient, satisfying and ;
[0087] Solving the objective function of the rolling optimization decision yields the optimal sequence. Then, the first query is submitted for execution. Once the query is actually submitted, the system status information is updated. At the same time, if a new query arrives, the system will re-execute steps 3 and 4, iteratively selecting the next optimal query to submit, thus forming a continuously rolling optimization decision-making process.
[0088] Step 5: Construct an adaptive weight adjustment mechanism based on the context-based multi-armed slot machine model;
[0089] The adaptive weight adjustment mechanism uses the system state as context information and dynamically adjusts the weight parameters in the scoring function and the objective function of the rolling optimization decision through a contextual multi-armed slot machine model, as detailed below:
[0090] Define the system state vector As contextual input, t is the decision time, p is the state feature dimension, and S t Metrics including resource utilization, load, and query queue length are used at the decision-making stage. The observed values, The state feature dimension is used. The weight configuration action space is a discrete set. Each action This represents a set of weight combinations that satisfy the constraints. , , and ;
[0091] At every decision moment The system performs the following steps:
[0092] Measure the current system state vector ;
[0093] Select an action based on the context of the multi-armed slot machine model. ;
[0094] Select weights Objective functions applied to scoring functions and rolling optimization decisions;
[0095] Execute query scheduling decisions and calculate average query latency and average resource utilization.
[0096] Calculate the reward function And update the context multi-armed slot machine model;
[0097] The reward function The design is as follows: For decision-making moments Average resource utilization rate For decision-making moments The average query latency, then the reward function for:
[0098] ;
[0099] The reward function directly reflects the relative change in system performance: the larger the reward value is when resource utilization increases and query latency decreases, and vice versa.
[0100] The contextual multi-armed slot machine model learns the expected reward of state-action pairs through historical interaction data and employs an exploration-utilization balancing strategy to select actions to maximize the cumulative reward of discounts. ,in As a discount factor, Indicates relative to the current decision time The time step offset. Through continuous learning, the system can adaptively adjust the weight configuration to achieve a balance between resource utilization and query latency under different system states.
[0101] Step 6: Construct a sliding window design method for model updates oriented towards dynamic data;
[0102] To address the dynamic changes in data distribution and query patterns over time in the lake warehouse system, a sliding window-based data sampling and model update mechanism is designed: by setting a sample window, incremental sampling and online training are performed on the most recent W queries and system operation data to achieve continuous adaptive updates of the query performance and resource consumption prediction model, thereby maintaining the prediction accuracy and stability of the model under dynamic load environments.
[0103] Step 7: Build a system utilization monitor and an automatic scaling mechanism;
[0104] The system features a utilization monitor that collects real-time system performance metrics, including query count, CPU utilization, memory utilization, bandwidth utilization, and task queue length. An automatic scaling mechanism is also implemented, dynamically triggering horizontal scaling of computing resources when the task queue duration exceeds a preset threshold. This alleviates query backlog and maintains system processing capacity stability. Furthermore, resource reclamation is supported during load shedding phases, thereby improving resource utilization efficiency.
[0105] Example 2:
[0106] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.
[0107] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the resource scheduling method under the lake warehouse converged architecture described in the various embodiments of this application.
[0108] The aforementioned storage media include: flash memory, hard disks, multimedia cards, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disks, optical discs, servers, APP (Application) app stores, and various other media capable of storing program verification codes, on which computer programs are stored. When the computer program is executed by a processor, it can implement the various steps of the resource scheduling method under the lakeware fusion architecture described above.
[0109] Example 3:
[0110] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the resource scheduling method under the lake-warehouse converged architecture.
[0111] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.
[0112] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.
[0113] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of the methods disclosed herein and their equivalents, then the intent of this disclosure also includes such modifications and variations.
Claims
1. A resource scheduling method under a lake-warehouse integrated architecture, characterized in that, Includes the following steps: Step 1: Construct the feature vector representation of the query; Assuming a query The physical query plan contains several operator nodes. Where n is the number of operator nodes, and each operator node From a local feature vector It means that among them If the feature dimension is used, then the query The query feature vector is defined as ,in This indicates feature aggregation using a vector concatenation function; Step 2: Construct a machine learning-based prediction model for query performance and resource consumption; Step 3: Construct a query scoring function to evaluate the query tasks in the waiting queue, thereby forming a priority scheduling candidate set; Step 4: Construct a rolling optimization decision-making process based on a model predictive control mechanism; Specifically, a model prediction control mechanism is introduced to perform rolling optimization decisions on the submission order of candidate queries; Step 5: Construct an adaptive weight adjustment mechanism based on the context-based multi-armed slot machine model; Step 6: Construct a sliding window design method for model updates oriented towards dynamic data; To address the dynamic changes in data distribution and query patterns over time in the lake warehouse system, a sliding window-based data sampling and model update mechanism is designed: by setting a sample window, incremental sampling and online training are performed on the most recent W queries and system operation data to achieve continuous adaptive updates of the query performance and resource consumption prediction model; Step 7: Build a system utilization monitor and an automatic scaling mechanism; The system features a utilization monitor that collects real-time system performance metrics, including query count, CPU utilization, memory utilization, bandwidth utilization, and task queue length. It also incorporates an automatic scaling mechanism that dynamically triggers horizontal scaling of computing resources when the task queue duration exceeds a preset threshold. This alleviates query backlog and maintains system processing capacity stability. Furthermore, it supports resource reclamation during load shedding phases, thereby improving resource utilization efficiency.
2. The resource scheduling method under a lake-warehouse fusion architecture according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2.1: Introduce system and runtime state feature vectors ,in Represents the dimension of the feature vector; Step 2.2: Use XGBoost to build a predictive model for query performance and resource consumption under concurrent conditions.
3. The resource scheduling method under a lake-warehouse fusion architecture according to claim 2, characterized in that, Step 2.1 specifically involves: considering the processes currently being executed in the system. A number of concurrent queries, where the system is a set of compute nodes in a lakeware converging architecture; each query Corresponding to a query feature vector , The concurrent query state feature vector is obtained by weighting and aggregating the feature vectors of each query through an attention mechanism. Considering the current system state, extract the system state feature vector. Concurrent query state feature vector With system state feature vector The data is then concatenated to form a complete system and runtime state feature vector. .
4. The resource scheduling method under a lake-warehouse fusion architecture according to claim 3, characterized in that, Step 2.2 specifically involves: assuming the query performance and resource consumption prediction model is a function... The parameter set of the model is The model output is the execution time of the query under concurrent conditions. and resource consumption vector : ； The model is trained to minimize the error between the predicted output and the actual observations; let the number of training samples be... , No. The observation execution time and resource consumption for each sample are as follows: and The objective function for the training process is: ； in, The loss function is the mean squared error method. For the optimal set of model parameters, Indicates the query in the nth training sample The query feature vector, This represents the system and runtime state feature vector corresponding to the nth training sample.
5. The resource scheduling method under a lake-warehouse fusion architecture according to claim 4, characterized in that, The query scoring function described in step 3 is as follows: Suppose there are in the waiting queue Candidate queries For each query Its scoring function Defined as: ； in, and Let be the weighting coefficient, satisfying and ; For resource matching degree, Concurrency influencing factors; Among them, resource matching degree This measures the degree of match between query resource requirements and the system's current remaining resources; let the query... The resource demand vector is The system's current remaining resource vector is The resource matching degree is then defined as: ； Concurrency influencing factors Evaluate the execution of query Q under the current system concurrency conditions. i Performance; design To query the estimated runtime under no concurrent interference, To predict query performance and resource consumption in the current concurrent environment using a model Based on the predicted runtime, the concurrency influencing factors are defined as follows: ； Ultimately, according to Sort all query tasks in the waiting queue and select queries with scores higher than a set threshold to form a priority scheduling candidate set.
6. The resource scheduling method under a lake-warehouse fusion architecture according to claim 5, characterized in that, Step 4 specifically involves: setting the current scheduling period as... The set of candidate queries in the waiting queue is Each query The arrival time is The prediction time domain is defined as follows: That is, in the future Optimize within each scheduling cycle; in each scheduling cycle Rolling optimization decision is defined as finding the optimal query submission sequence. ,in, Indicates the first The queries submitted from the candidate query set in each scheduling cycle minimize the objective function of the rolling optimization decision: ； in, Indicates query During the scheduling period The predicted runtime of the submitted execution is determined by the model. Calculations show that Represents the scheduling period Start time, For query Arrival time; and Let be the weighting coefficient, satisfying and ; Solving the objective function of the rolling optimization decision yields the optimal sequence. Then, the first query is submitted for execution. Once the query is actually submitted, the system status information is updated. At the same time, if a new query arrives, the system will re-execute steps 3 and 4, iteratively selecting the next optimal query to submit, thus forming a continuously rolling optimization decision-making process.
7. The resource scheduling method under a lake-warehouse fusion architecture according to claim 6, characterized in that, The adaptive weight adjustment mechanism described in step 5 uses the system state as context information and dynamically adjusts the weight parameters in the scoring function and the objective function of the rolling optimization decision through a contextual multi-armed slot machine model, as detailed below: Define the system state vector As contextual input, t is the decision time, p is the state feature dimension, and S t In order to make decisions The observed values, For state feature dimensions; The weight configuration action space is a discrete set. Each of the actions This represents a set of weight combinations that satisfy the constraints. , , and ; At every decision moment The system performs the following steps: Measure the current system state vector ; Select an action based on the context of the multi-armed slot machine model. ; Select weights Objective functions applied to scoring functions and rolling optimization decisions; Execute query scheduling decisions and calculate average query latency and average resource utilization. Calculate the reward function And update the context multi-armed slot machine model; The reward function The design is as follows: For decision-making moments Average resource utilization rate For decision-making moments The average query latency, then the reward function for: ； The contextual multi-armed slot machine model learns the expected reward of state-action pairs through historical interaction data and employs an exploration-utilization balancing strategy to select actions to maximize the cumulative reward of discounts. ,in As a discount factor, Indicates relative to the current decision time The time step offset.
8. A computer-readable storage medium, characterized in that, The system stores executable instructions that, when executed, cause the processor to perform the resource scheduling method under the lake warehouse converged architecture as described in any one of claims 1-7.
9. A computer program product, characterized in that, Includes a computer program or instructions that are executed by a processor using the resource scheduling method under the lakeware fusion architecture as described in any one of claims 1-7.