Dynamic resource scheduling and optimizing method for big data service

Abstract

The invention relates to the field of big data services, in particular to a dynamic resource scheduling and optimizing method for big data services, which comprises the following steps: constructing a time sequence feature set, outputting a resource demand predicted value in a future preset advance time window, calculating a reservation proportion of an elastic buffer pool, and executing container preheating operation before a load peak value arrives; wherein the reservation proportion is dynamically adjusted according to the confidence coefficient of the prediction result, and when the prediction confidence coefficient is higher than a preset threshold value, the scale of the preheating container is automatically expanded to cover the predicted load increment. According to the method, the time sliding window is constructed, the active prediction model is utilized to analyze the periodicity and the burstiness of the load, the container preheating is executed before the peak value arrives based on the pre-reaction principle, and the advanced time window is introduced, so that the cold start time of the traditional passive expansion is eliminated, the resource is used as required, and the resource utilization rate is improved. And the response speed of the system to the tidal flow is obviously improved.

Description

Technical Field

[0001] This application relates to the field of big data services, and in particular to a dynamic resource scheduling and optimization method for big data services. Background Technology

[0002] With the explosive growth of mobile internet and the Internet of Things, data processing operations are characterized by drastic traffic fluctuations, complex and diverse task types, and heterogeneous hardware environments, which places extremely high demands on the response speed and allocation accuracy of resource scheduling systems.

[0003] Currently, existing technologies mainly employ rule-based passive response mechanisms or simple regression prediction models. For example, some mainstream schedulers trigger expansion operations by monitoring whether resource utilization exceeds a fixed threshold, or perform linear extrapolation prediction based on historical averages of a single dimension. However, these technical solutions have significant drawbacks: First, passive response mechanisms have obvious time lags, leading to "cold start" delays in the system during sudden load surges, affecting business continuity. Second, general and single prediction dimensions cannot identify the differentiated needs of heterogeneous tasks, easily leading to resource mismatch phenomena such as high CPU utilization but task failure due to memory overflow.

[0004] Therefore, existing technologies are unable to solve the response lag problem caused by sudden load surges, and are also unable to establish a precise adaptation relationship between heterogeneous tasks and heterogeneous hardware, making it difficult to achieve both resource utilization and service quality. Summary of the Invention

[0005] The purpose of this application is to provide a dynamic resource scheduling and optimization method for big data services, comprising: establishing a time sliding window containing historical resource usage data; constructing a time-series feature set by collecting runtime state data of computing nodes in a big data cluster; inputting the time-series feature set into a preset active prediction model, wherein the active prediction model is based on the principle of pre-reaction and outputs a predicted value of resource demand within a preset lead time window by analyzing the periodicity and burstiness characteristics of the load; calculating the reservation ratio of the elastic buffer pool based on the predicted value of resource demand and the current resource pool status, and performing container preheating operation before the load peak arrives; wherein the reservation ratio is dynamically adjusted according to the confidence level of the prediction result, and when the prediction confidence level is higher than a preset threshold, the size of the preheated container is automatically expanded to cover the expected load increment.

[0006] By adopting the above technical solution, constructing a time sliding window and using an active prediction model to analyze the periodicity and suddenness of the load, and performing container preheating before the peak arrives based on the principle of pre-reaction, the cold start time of traditional passive expansion is eliminated by introducing an advanced time window, realizing the on-demand availability of resources and significantly improving the system's response speed to tidal flow.

[0007] Optionally, the step of constructing the time-series feature set includes a multimodal feature fusion process: collecting system-level indicators, application-level indicators, and business-level indicators respectively; using an adaptive weight network to extract features and perform weighted fusion on the above-mentioned indicators of different dimensions to generate a task resource fingerprint containing multi-dimensional attributes; wherein, the system-level indicators include at least the central processing unit level, memory utilization, and network throughput, the application-level indicators include at least the task directed acyclic graph depth and data shuffling volume, and the business-level indicators include at least the business priority marker and activity cycle identifier.

[0008] Optionally, the active prediction model adopts a spatiotemporal fusion network structure, specifically including: using a one-dimensional convolutional neural network to extract the local fluctuation features of the resource usage curve within the time sliding window; using a long short-term memory network to capture the long-term dependency relationship and periodic change trend of resource load; and mapping the extracted features to the probability distribution of resource demand at future moments through a fully connected layer.

[0009] Optionally, the reservation ratio of the elastic buffer pool specifically includes: calculating the variance of the predicted resource demand value as a basis for measuring the prediction uncertainty; establishing a mapping function between prediction uncertainty and resource redundancy, and nonlinearly increasing resource redundancy when the variance of the predicted resource demand value is large, so as to compensate for the prediction deviation through excessive reservation; and setting a minimum maintenance threshold to ensure that the minimum set of resources to maintain the basic service capacity of the cluster is reserved during periods of low load.

[0010] Optionally, the multimodal feature fusion process further includes a task profile clustering step: based on the resource consumption patterns of historical tasks, an unsupervised clustering algorithm is used to divide the tasks into several resource fingerprint categories; for different resource fingerprint categories, the corresponding feature attention module is activated in the active prediction model to assign high weights to the key dimensions in the input feature vector; wherein, the resource fingerprint categories include at least computationally intensive, input-output intensive, and memory intensive types.

[0011] Optionally, after performing the container preheating operation, a topology-aware scheduling step is further included: constructing a topology mapping graph of heterogeneous physical resources, wherein the topology mapping graph identifies the hardware attributes and real-time interference of physical nodes; calculating the affinity score between the resource requirement vector of the task to be scheduled and the node state vector of the physical node; and mapping the predicted preheating container to the locally optimal physical node according to the affinity score, thereby achieving the adaptation of task characteristics and physical hardware.

[0012] Optionally, the construction of the topology mapping map of heterogeneous physical resources specifically includes: obtaining hardware type labels of physical nodes through static scanning, wherein the hardware type labels distinguish between graphics processor nodes, high-throughput disk nodes, and general-purpose computing nodes; obtaining real-time status labels of physical nodes through dynamic monitoring, wherein the real-time status labels include input / output wait rate and co-location task interference index; and combining the hardware type labels with the real-time status labels to generate three-dimensional resource space coordinates.

[0013] Optionally, the affinity score between the resource requirement vector of the task to be scheduled and the node state vector of the physical node is calculated using a vector matching algorithm: defining a sensitivity weight vector of the task to different hardware resources; calculating the proximity of the task resource requirement vector and the node state vector in the weighted Euclidean distance space; for tasks with high input / output requirements, increasing their matching weight with nodes with non-volatile memory tags, thereby prioritizing the scheduling of such tasks to high-throughput nodes.

[0014] Optionally, it also includes a soft preemption step based on the service level agreement: when the predicted demand of a high-priority task exceeds the currently available resources, a dynamic preemption mechanism is triggered; a target task among low-priority tasks that has enabled the checkpoint saving mechanism is identified; a suspension instruction is sent to the target task, its current execution state is saved to the distributed storage system, and the physical resources it occupies are released for use by high-priority tasks, rather than directly terminating the target task.

[0015] Optionally, it also includes a closed-loop feedback optimization step: real-time collection of actual load data and task completion time after scheduling execution; calculation of the deviation loss between the actual load data and the predicted resource demand; and periodic updating of the network parameters of the active prediction model using a backpropagation algorithm to correct prediction drift caused by changes in business models.

[0016] This application addresses the mismatch between heterogeneous task features and general prediction models by proposing a multimodal feature fusion scheme. It collects indicators from three dimensions: system, application, and business, and uses an adaptive weight network for fusion, solving the problem that a single indicator cannot accurately characterize the resource fingerprint of complex tasks. Based on the principles of segmentation and combination, this application refines and reorganizes task features, enabling the prediction model to accurately identify the real needs of different types of tasks, significantly improving prediction accuracy and avoiding resource mismatch.

[0017] To address the discrepancy between prediction results and heterogeneous physical resource topology adaptation, this application proposes a topology-aware scheduling strategy. By constructing a topology mapping graph that includes static hardware attributes and dynamic interference levels, and calculating affinity scores, logical prediction requirements are mapped to the physical nodes with the best local quality. Based on the principles of local quality and dynamism, this ensures that tasks can run in the hardware environment most suitable for their characteristics, eliminates the long-tail effect of physical resource performance, and further improves the overall throughput and stability of the cluster. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the dynamic resource scheduling and optimization method for big data services used in this application. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0020] like Figure 1 As shown in the figure, this application provides a dynamic resource scheduling and optimization method for big data services, including the following steps.

[0021] S01: Establish a time sliding window containing historical resource usage data, and construct a time-series feature set by collecting runtime status data of computing nodes in the big data cluster.

[0022] S02: Input the time series feature set into the preset active prediction model. The active prediction model is based on the principle of pre-action and outputs the predicted value of resource demand within the preset advance time window by analyzing the periodic and sudden characteristics of the load.

[0023] S03: Based on the predicted resource demand and the current resource pool status, calculate the reservation ratio of the elastic buffer pool and perform container preheating before the load peak arrives; wherein, the reservation ratio is dynamically adjusted according to the confidence level of the prediction results. When the prediction confidence level is higher than the preset threshold, the size of the preheated container is automatically expanded to cover the expected load increment.

[0024] Understandably, this method is mainly applied to scenarios where large-scale distributed computing clusters such as Apache Spark or Hadup are used to process massive user behavior log analysis jobs. In such scenarios, due to the strong tidal effect and suddenness of internet user behavior, such as when e-commerce promotional activities start or when breaking news events occur, log data traffic will show an exponential jump. Traditional passive expansion mechanisms based on resource thresholds often need to take several minutes to complete the startup of nodes and environment initialization, resulting in a "cold start" bottleneck in the early stage of load peak, such as a lack of computing resources, job backlog, or even service crash.

[0025] Specifically, to overcome this technical bias, the embodiments of this application first use lightweight agent programs deployed on each computing node of the cluster to collect runtime state data in real time at a sampling frequency of milliseconds. This runtime state data not only includes basic physical indicators such as CPU utilization and memory usage, but also covers deeper indicators that can reflect the micro-pressure of the system, such as task queue length, garbage collection frequency, and input / output wait time. Subsequently, the system constructs a dynamically long time window, which extracts the collected continuous state data into fixed-length time series segments, for example, setting the window length to the past thirty minutes. This serves as a basic observation unit to capture recent load trends. Based on this, the system cleans and standardizes the raw data within the time window, removing noise caused by network jitter or sensor errors, and constructs a high-fidelity time-series feature set. This feature set is then input into a pre-set active prediction model. The core design of this active prediction model follows the principle of pre-emptive counteraction in application problem-solving theory, that is, before a harmful effect (load burst) occurs, a counteraction (resource expansion) is applied to offset its harm. Internally, the active prediction model uses deep learning algorithms to analyze the implicit cycles in historical load data. The system analyzes both regular patterns (such as daily evening peak hours) and non-linear, sudden characteristics (such as traffic surges triggered by specific events), enabling it to output a predicted resource demand value for a future preset lead time window at the current moment. This predicted resource demand value is a resource demand curve containing a confidence interval. The system compares this predicted resource demand value with the current idle status of the resource pool, calculates the reservation ratio of the elastic buffer pool, and sends a pre-start command to the cluster resource manager a few minutes before the actual arrival of the load peak to start the containers in advance and load the necessary runtime environment and dependent libraries, i.e., to perform container preheating. In particular, to balance resource waste... Regarding service quality risks, the reservation ratio is dynamically adjusted. When the confidence level of the prediction results output by the proactive prediction model is higher than the preset threshold, such as 90%, the system determines that the upcoming load surge has a very high degree of certainty. Therefore, it automatically expands the scale of the preheating container and even reserves 20% more redundant resources than the predicted value to cover possible extreme load increments. This ensures that at the moment the traffic surge arrives, all computing tasks can immediately obtain the computing resources prepared in advance due to the pre-reaction principle, completely eliminating the time lag caused by physical node startup and guaranteeing business continuity and system stability under the impact of data surges.

[0026] It is understandable that the steps to construct a time-series feature set include a multimodal feature fusion process: collecting system-level indicators, application-level indicators, and business-level indicators respectively; using an adaptive weight network to extract features and perform weighted fusion on the above-mentioned indicators of different dimensions to generate a task resource fingerprint containing multi-dimensional attributes; among them, system-level indicators include at least CPU level, memory usage, and network throughput, application-level indicators include at least the depth of the task directed acyclic graph and the amount of data shuffling, and business-level indicators include at least business priority markers and activity cycle identifiers.

[0027] Specifically, this process aims to address the challenge of accurately depicting the resource requirements of heterogeneous tasks using a single-dimensional indicator. In real-world production environments, big data clusters often run a mix of various types of heterogeneous tasks. For example, extraction, transformation, and loading jobs typically involve large-scale data cleaning and format conversion, making them typical input-output intensive tasks with extremely high sensitivity to disk bandwidth and network throughput. Deep learning model training jobs, on the other hand, involve numerous matrix operations, making them typical computationally intensive tasks where the demand for GPU computing power and video memory bandwidth is dominant. Relying solely on the CPU's capacity for prediction can easily lead to an underestimation or misrepresentation of the resource requirements for extraction, transformation, and loading tasks. Ignoring the risk of memory overflow in training tasks, this application introduces a multimodal feature fusion mechanism to collect system-level, application-level, and business-level indicators from all angles, constructing a three-dimensional monitoring perspective. System-level indicators form the underlying framework of the resource profile, including not only conventional CPU usage and memory utilization, but also focusing on monitoring network throughput (including inbound and outbound traffic rates) and disk read / write operations per second to accurately reflect the load pressure on physical facilities. Application-level indicators delve into the logical level of job execution, collecting the depth and breadth of the directed acyclic graph (DAG) of tasks. The depth of the DAG reflects the length of the serial dependency chain between tasks. The degree of data shuffling directly relates to the total execution time of a task, while the amount of intermediate data shuffling quantifies the scale of data transmitted between distributed nodes and is a key factor in assessing network congestion risk. Business-level metrics introduce higher-dimensional semantic information, including business priority markers (e.g., core payment business takes precedence over offline reporting business) and activity cycle identifiers (e.g., during the "Double Eleven" promotion or "Black Friday" period). These metrics help the model understand the external drivers of load changes. After acquiring the aforementioned massive amounts of multidimensional data, the system utilizes a pre-trained adaptive weight network as a feature fusion engine. This network can automatically learn the importance weights of each dimension metric based on the historical performance of different task types. For example, when the current task flow is identified as having high-frequency disk read / write characteristics, the weight of the disk throughput metric in the feature vector is automatically increased, while the weight of the central processing unit (CPU) level is decreased, and vice versa. Finally, the multidimensional data after weighted fusion is compressed and mapped into a task resource fingerprint containing multidimensional attributes. This task resource fingerprint is like a biological gene fragment of the task, uniquely and accurately identifying the resource consumption pattern of the task within a specific time window. This allows the system to not only know "how many resources are needed" but also "what kind of resources are needed," thereby effectively avoiding resource type mismatch caused by missing feature dimensions and ensuring the harmonious coexistence and efficient operation of heterogeneous tasks in a unified cluster.

[0028] In this embodiment, the active prediction model adopts a spatiotemporal fusion network structure, specifically including: using a one-dimensional convolutional neural network to extract the local fluctuation features of the resource usage curve within a time sliding window; using a long short-term memory network to capture the long-term dependency relationship and periodic change trend of resource load; and mapping the extracted features to the probability distribution of resource demand at future moments through a fully connected layer.

[0029] Specifically, the active prediction model is responsible for transforming massive amounts of historical data into accurate predictions of the future. Faced with the complexity of data loads in big data clusters, a single prediction algorithm often struggles to balance local fluctuations and long-term trends. For example, simple linear regression cannot capture non-linear traffic bursts, while ordinary recurrent neural networks are prone to gradient vanishing when dealing with extremely long sequences, leading to the forgetting of early patterns. Therefore, this application's embodiments design a composite spatiotemporal fusion network structure, which deploys dedicated neural processing units in both the spatial and temporal dimensions. First, a one-dimensional convolutional neural network is used as a feature extraction outpost, performing convolution operations on the resource usage curve within the input time window. The one-dimensional convolutional kernel slides across the time series, keenly capturing local fluctuations in resource usage within a short period, such as instantaneous memory jitter triggered by garbage collection or a sudden drop in throughput caused by network packet loss. The convolutional layer filters out high-frequency noise through pooling operations, extracting representative local morphological features, effectively preventing the model from overfitting to random perturbations. Subsequently, the extracted local feature sequences are fed into a long short-term memory network layer. The memory network layer, through its unique forget gate, input gate, and output gate mechanisms, constructs a time series analysis channel with long-term memory capabilities. The Long Short-Term Memory (LSTM) network can capture long-term dependencies and periodic trends in resource load over extended time spans, such as identifying patterns in batch job submissions at the top of each hour or peak data backups every Friday afternoon, thus grasping the overall trajectory of load evolution at a macro level. Finally, the local burst features extracted by the convolutional layer and the long-term periodic features captured by the LSM layer are deeply and non-linearly fused in the fully connected layer. The fully connected layer acts as a mapper, transforming high-dimensional abstract feature vectors into a probability distribution of future resource demands. This probability distribution not only provides the most likely resource demand value but also indicates the degree of certainty in the prediction through its variance. The entire network structure is trained through end-to-end supervised learning, using backpropagation of the error between historical real load data and predicted values ​​to continuously fine-tune the convolutional kernel parameters and gating weights. This allows the model to continuously evolve with the cluster's business model, ultimately achieving high-precision fitting of future load changes and providing a solid basis for dynamic resource scheduling decisions.

[0030] Understandably, calculating the reservation ratio of the elastic buffer pool specifically includes: calculating the variance of the predicted resource demand as a basis for measuring the uncertainty of the prediction; establishing a mapping function between the prediction uncertainty and the resource redundancy, and nonlinearly increasing the resource redundancy when the variance of the predicted resource demand is large, so as to compensate for the prediction deviation through excessive reservation; and setting a minimum maintenance threshold to ensure that the minimum set of resources to maintain the basic service capacity of the cluster is reserved during periods of low load.

[0031] Specifically, although proactive prediction models can provide high-precision load forecasts, in real-world big data production environments, due to external uncertainties such as network cable failures, external attacks, or sudden social media hotspots, the actual load always has the potential to deviate from the predicted trajectory. If resource allocation is rigidly based on the predicted value, resource contention and service degradation will immediately occur if the actual load is slightly higher than the predicted value. Therefore, this application's embodiments introduce a flexible buffer layer based on statistical principles between the prediction result and the execution action. First, while outputting the predicted resource demand value, the system calculates the variance of the predicted value on the probability distribution. This variance directly reflects the degree of hesitation of the proactive prediction model regarding the current prediction result, i.e., the uncertainty of the prediction. The larger the variance, the wider the range of future load fluctuations, and the higher the risk of prediction failure. Next, the system establishes a non-linear mapping function. This mapping function takes the prediction variance as an input variable and resource redundancy as an output variable. Its mathematical characteristics show that as the variance increases, the resource redundancy exhibits an exponential growth rather than a simple linear increase. This means that when the proactive prediction model determines that the future load is extremely unstable, the system will aggressively allocate a large amount of additional resources as a safety margin to compensate for possible prediction deviations through excessive reservation. For example, during a stable period when the proactive prediction model has high prediction accuracy, only 5% of the buffer resources may be reserved, while during a turbulent period with a surge in variance, the reservation ratio may instantly increase to 50%. In addition, to prevent the system from excessively scaling down under extremely low load conditions (such as early morning hours) and causing the cluster to lose its basic rapid response capability, this application embodiment also sets a minimum maintenance threshold. This minimum maintenance threshold defines the minimum number of containers and core service processes that the cluster must retain, ensuring that no matter how low the predicted load is, the cluster always maintains a basic set of services that can cope with cold starts at any time, avoiding shutdowns due to over-optimization. Through this dynamic buffering mechanism based on uncertainty quantification, the system finds a dynamic balance between resource utilization and service reliability. It can save costs by reducing redundancy during stable periods and ensure safety by increasing investment during turbulent periods, reflecting the wisdom of risk management in the control of large-scale complex systems.

[0032] In this embodiment of the application, the multimodal feature fusion process further includes a task profile clustering step: based on the resource consumption patterns of historical tasks, an unsupervised clustering algorithm is used to divide the tasks into several resource fingerprint categories; for different resource fingerprint categories, the corresponding feature attention module is activated in the active prediction model, and high weights are assigned to the key dimensions in the input feature vector; wherein, the resource fingerprint categories include at least computationally intensive, input-output intensive, and memory intensive.

[0033] Understandably, this step achieves intelligent classification and feature focusing of massive heterogeneous tasks through unsupervised learning technology. In big data service scenarios, tens of thousands of jobs are submitted daily. Although the specific code and processed data of each task are different, they often exhibit obvious clustering characteristics in terms of resource consumption patterns. To uncover these potential patterns, this application introduces density-based unsupervised clustering algorithms, such as variants of DBSCAN or K-Means, using time-series data of resource consumption in the historical task library as samples to automatically classify and divide tasks, grouping tasks with similar resource fingerprints into the same cluster. For example, unsupervised clustering algorithms can automatically identify a group of tasks that consume a large amount of memory for a long time and have moderate CPU usage, marking them as memory-intensive; or identify a group of tasks that frequently perform small file read / write operations and have extremely low computational load, marking them as input / output-intensive; and tasks that continuously run arithmetic logic units at full load are marked as computationally intensive. Classification is not merely used for statistical reports, but is directly fed back to the active prediction model to activate the corresponding feature attention module within the model. When a new task arrives and is identified as belonging to a specific resource fingerprint category, the active prediction model automatically adjusts its attention mechanism, assigning higher weights to key dimensions in the input feature vector. For example, for tasks classified as input-output intensive, the active prediction model suppresses interference from CPU-related metrics and focuses intensely on the changing trends of disk queue length and network bandwidth utilization. This allows it to more accurately capture the core bottlenecks of such tasks during prediction, similar to the diagnostic process of human experts who focus on different physiological indicators for different ailments. Through this feature weighting process, the model's generalization ability and prediction accuracy when handling mixed loads are significantly improved. This avoids the averaging error caused by using a set of general parameters to fit all types of tasks, ensuring that both high-throughput data cleaning tasks and high-computing-power model training tasks can obtain resource scheduling strategies that are precisely matched to their characteristics.

[0034] In this embodiment of the application, after performing the container preheating operation, a topology-aware scheduling step is further included: constructing a topology mapping graph of heterogeneous physical resources, wherein the topology mapping graph identifies the hardware attributes and real-time interference of physical nodes; calculating the affinity score between the resource requirement vector of the task to be scheduled and the node state vector of the physical node; and mapping the predicted preheating container to the locally optimal physical node according to the affinity score, thereby achieving the adaptation of task characteristics and physical hardware.

[0035] Understandably, this step primarily addresses the long-tail effect of resource performance caused by the heterogeneity of physical nodes and the complexity of network topology in distributed computing clusters. In large-scale distributed systems such as Apache Spark or Hadup, although the proactive prediction model has accurately calculated the required number and specifications of containers, blindly and randomly allocating these containers to physical nodes often leads to a significant reduction in overall performance due to ignoring the local quality differences in physical hardware. This is just as the principle of local quality reveals, that is, each part of an object should be placed under conditions most conducive to its operation. To this end, the embodiments of this application construct a rigorous input processing... The logical chain is as follows: First, as the system's input, the scheduler receives a list of warm-up containers from the proactive prediction model, containing detailed specifications and requirements. These requirements include not only the number of cores and memory size but also implicit preferences for hardware characteristics. Simultaneously, agents deployed on each node of the Dell PowerEdge server cluster continuously report real-time physical layer status data at a high sampling rate of once per second. This includes, but is not limited to, CPU utilization (accurate to two decimal places, ranging from 0-100%), actual memory usage (in GB, ranging from 0 to 1024 GB), and any indicators reflecting node congestion levels. The system collects massive amounts of data, including queue length (integer count, ranging from 0 to 10000), and even micro-environmental indicators such as ambient temperature (20 to 30 degrees Celsius) and humidity (40% to 60%) within racks and cabinets. This data is aggregated into a real-time physical resource state pool. Next, the system activates the topology-aware scheduling engine. Based on a pre-built topology map, this engine performs location calculations for each container to be deployed. The location calculation process employs a multi-objective optimization algorithm, comprehensively considering the network distance between nodes, the bandwidth load of rack switches, and the current noise interference level of nodes. The system simulates placing containers on different candidate nodes. For example, in terms of performance, for a shuffling task container predicted to be network-intensive, the topology-aware scheduling engine will prioritize scanning idle resources within the same rack or even on the same physical node to minimize data transmission latency by utilizing local loopback networks or high-speed interconnects within the rack. For compute-intensive containers, it will avoid high-load nodes with CPU temperatures approaching thresholds or queue lengths exceeding 5000 to prevent thermal throttling. After complex intermediate calculations, the candidate node list is sorted by affinity and filtered against affinity, eliminating bad nodes with potential hardware failure risks or intense resource contention.Ultimately, as output, the topology-aware scheduling engine generates a container placement list accurate to the physical IP address and port number, and sends this list of container placement instructions to the cluster resource manager, such as YARN ResourceManager or Kubernetes Scheduler. This ensures that each pre-warmed container is precisely placed on the locally optimal physical node, achieving seamless integration and maximizing performance from logical prediction to physical execution.

[0036] In this embodiment of the application, constructing a topology mapping map of heterogeneous physical resources specifically includes: obtaining hardware type labels of physical nodes through static scanning, the hardware type labels distinguishing between graphics processor nodes, high-throughput disk nodes, and general-purpose computing nodes; obtaining real-time status labels of physical nodes through dynamic monitoring, the real-time status labels including input / output wait rates and co-location task interference index; and combining the hardware type labels with the real-time status labels to generate three-dimensional resource space coordinates.

[0037] Understandably, in existing heterogeneous cluster environments, physical nodes often consist of servers procured in different batches, including dedicated computing nodes equipped with high-performance solid-state drives and graphics processors, as well as general-purpose computing nodes mainly used for cold data storage. This hardware inconsistency requires the scheduling system to have extremely strong environmental awareness. This implementation example adopts a dual-track construction strategy of "static scanning + dynamic correction": First, in the static dimension, the system performs a deep hardware asset inventory of all physical nodes in the cluster through out-of-band management interfaces or hardware self-test tools, extracting information such as CPU model, instruction set architecture (e.g., AVX-512 support), and disk type (NVMe). Static attribute tags, including SSDs or SATA HDDs and memory frequency, are used to construct the skeleton structure of the topology mapping, clearly marking which nodes are suitable for high-throughput computing and which nodes are suitable for large-capacity storage. Subsequently, in the dynamic dimension, the system uses the aforementioned lightweight agent to introduce a real-time interference metric. This metric is mainly used to quantify the performance loss caused by the noisy neighbor effect in a multi-tenant environment. Specifically, streaming computation can be performed on the fine-grained monitoring data collected by the agent, focusing on analyzing input / output wait rates and cache miss rates. When a significant increase in the input / output wait time of a node is detected, accompanied by a surge in queue length, the system will derive the co-location task interference index for that node through an algorithm. This co-location task interference index is a floating-point number between 0 and 1; a higher value indicates more intense resource contention for that node. The higher the intensity of the load, the greater the risk of performance degradation when running new tasks. In addition, the system also takes environmental sensor data into consideration, using a weighted fusion algorithm to convert the ambient temperature (20-30 degrees Celsius) and humidity (40-60%) data within the rack into physical stability factors to prevent high-load tasks from being scheduled to poorly cooled areas. These static labels and dynamic indices are uniformly mapped into a high-dimensional vector space to generate unique state coordinates for each physical node in the current time slice. Finally, the system generates a dynamically updated three-dimensional resource topology map, which not only shows the distribution of physical resources but also intuitively presents the real-time load heat and interference level of each node in the form of a heat map. This ensures that scheduling decisions are not only based on "what resources are available" but also on "how well the resources are available," enabling the system structure to adaptively adjust with environmental changes.

[0038] Understandably, the affinity score between the resource requirement vector of the task to be scheduled and the node state vector of the physical node is calculated using a vector matching algorithm: defining a sensitivity weight vector of the task to different hardware resources; calculating the proximity of the task resource requirement vector and the node state vector in the weighted Euclidean distance space; and increasing the matching weight between tasks with high input / output requirements and nodes with non-volatile memory tags, thereby prioritizing the scheduling of such tasks to high-throughput nodes.

[0039] Specifically, in big data processing scenarios, the relationship between tasks and resources is not simply one of presence or absence, but rather one of superiority or inferiority. To find this optimal solution, this application's embodiment introduces vector space model technology: First, in the input phase, the system constructs a task resource requirement vector and a node state vector. The task resource requirement vector is transformed from the multimodal feature fusion result in the preceding steps and includes sensitivity weights for computing power, memory bandwidth, and network latency. For example, a vector for extracting, transforming, and loading tasks might be represented as [processor: 0.3, memory: 0.2, disk: 0.8, ...]. [Network: 0.5], emphasizing its high dependence on disk, while the node state vector is provided by the topology map, representing the node's supply capacity and idle level in the corresponding dimension; then, the vector matching algorithm uses a weighted Euclidean distance or cosine similarity formula to calculate the geometric proximity between the two vectors in the multidimensional feature space. The vector matching algorithm also introduces a set of configurable adjustment factors. For tasks with high input / output requirements, such as log index building, the system will artificially amplify the weight value of the disk throughput dimension, so that the task vector is stretched and transformed in the mathematical space to be closer to high-performance nodes with non-volatile memory tags and a current disk queue length of zero; the system not only calculates the score of a single node, but also considers rack-aware constraints and calculates cross-rack communication. The resulting penalty factor converts physical distance into a negative score, which is then added to the overall affinity score to suppress fragmented resource scheduling across network partitions. Furthermore, the vector matching algorithm introduces a correction coefficient based on historical performance. If a certain type of task has a high historical failure rate on a specific node, the affinity score of that node will be forcibly downgraded. Finally, the vector matching algorithm generates an affinity score list containing all available nodes for each container to be scheduled. The affinity score list is sorted from highest to lowest score, and the scheduler uses this list to perform a greedy selection, prioritizing the encapsulation of tasks through container technology and distributing them to the nodes with the highest scores. By expanding one-dimensional resource values ​​into a multi-dimensional vector space, extremely precise supply and demand matching is achieved, completely solving the resource mismatch problem in heterogeneous environments.

[0040] In this embodiment of the application, a soft preemption step based on the service level agreement is also included: when the predicted demand of a high-priority task exceeds the currently available resources, a dynamic preemption mechanism is triggered; a target task with a checkpoint saving mechanism enabled is identified among low-priority tasks; a suspension instruction is sent to the target task, its current execution state is saved to the distributed storage system, and the physical resources it occupies are released for use by high-priority tasks, rather than directly terminating the target task.

[0041] It is understandable that resource contention is common in hybrid big data clusters. When the forecasting demand of core businesses (such as real-time transaction risk control) suddenly surges and the remaining cluster resources are insufficient to meet its warm-up requirements, resources must be reclaimed from non-core businesses (such as offline report analysis). Traditional direct termination methods would result in the complete loss of progress of the victim task, causing a huge waste of computing power, which violates the principle of sustainable development. To address this, this application's embodiment designs a soft preemption mechanism, which specifically includes: First, the input end monitors a resource gap in a high-priority task, triggering a red alert. The system immediately initiates the preemption process, quickly scanning the list of currently running low-priority tasks. First, a freeze and save instruction is sent to the selected target task. Using the checkpoint mechanism built into the big data computing framework (such as Spark) or the operating system's process snapshot technology, the current memory stack, register state, and intermediate data of the target task are saved. The calculation results are serialized; these serialized state data are divided into data blocks, transmitted at high speed over the network, and persistently stored in a dedicated pending directory of a distributed file system (such as HDFS or S3), ensuring the complete preservation of the task execution progress; subsequently, the system releases the CPU cores and memory pages occupied by the task, transferring its physical resources without any residue to the warm-up container of a high-priority task. At this point, the target task is logically in a suspended state rather than a failed state; finally, when the load peak of high-priority services subsides and the resource pool level recovers, the system automatically retrieves the pending directory, deserializes the previously saved task state, and reloads it back into memory, enabling the task to seamlessly continue execution from the breakpoint without having to recalculate from scratch. This soft preemption mechanism ensures the absolute priority of core services while maximizing the protection of existing computing power investment for offline services, achieving the maximum effective output of the cluster through a time-for-space strategy.

[0042] In this embodiment, a closed-loop feedback optimization step is also included: real-time collection of actual load data and task completion time after scheduling execution; calculation of the deviation loss between actual load data and resource demand prediction; and periodic updating of network parameters of the active prediction model using the backpropagation algorithm to correct prediction drift caused by changes in business model.

[0043] Understandably, any proactive predictive model based on historical data faces the challenge of concept drift, meaning that user behavior patterns and business logic can fundamentally change over time. For example, changes in e-commerce promotional activities or the launch of new business lines can cause existing load patterns to become invalid. To combat this entropy increase, this application's embodiment establishes an automated closed-loop feedback system: First, as an input, after each scheduled execution, the system not only records the final completion time of the task but also uses log aggregation tools (such as ELK) to... The Stack collects detailed actual load curve data, including real CPU utilization, peak memory usage, and network traffic every second, and aligns this real data with previous resource demand predictions over time. Next, the system uses a loss function (such as Mean Squared Error (MSE) or HuberLoss) to calculate the deviation between the predicted and actual values; this deviation is defined as the prediction residual. If the residual exceeds a preset tolerance threshold, the system automatically triggers an online learning process. In this process, the real load data from the most recent period, along with its corresponding multimodal features, is encapsulated into a new training sample set. The backpropagation algorithm is then used to adjust the convolutional kernel weights and the gating parameters of the Long Short-Term Memory network in the active prediction model. To prevent overfitting or catastrophic forgetting caused by noise in new samples, incremental learning techniques are employed, combined with a momentum optimizer to smooth the parameter update trajectory and ensure the stability of model evolution. Finally, the system generates an updated version of the model parameters and seamlessly hot-loads it into the inference engine of the production environment, enabling the proactive prediction model to capture the latest business features in real time, such as quickly learning to identify special traffic pulses brought about by newly launched flash sales. Through this day-to-day closed-loop feedback and self-iteration, the entire resource scheduling system can grow in sync with the development of the enterprise's business, always maintaining high-precision control over resource scheduling, and ensuring the vitality and advancement of the technical solution over a long period of time.

[0044] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0045] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0046] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0047] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0048] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0049] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only one example of a series of equivalent or similar features.

Claims

1. A method for dynamic resource scheduling and optimization for big data services, characterized in that, The application relates to a method for predicting resource demand and preheating containers, comprising the following steps: a time sliding window containing historical resource usage data is established by collecting runtime state data of computing nodes in a big data cluster to build a time series feature set; the time series feature set is input into a preset active prediction model, the active prediction model is based on a pre-impact principle, the periodicity and burstiness characteristics of the load are analyzed, and a resource demand prediction value in a future preset lead time window is output; based on the resource demand prediction value and the current resource pool state, the reservation ratio of the elastic buffer pool is calculated, and a container preheating operation is performed before the load peak arrives; wherein the reservation ratio is dynamically adjusted according to the confidence of the prediction result, and when the prediction confidence is higher than a preset threshold, the size of the preheating container is automatically enlarged to cover the expected load increment.

2. The method for dynamic resource scheduling and optimization for big data services of claim 1, wherein, The step of building the time series feature set includes a multi-modal feature fusion process: system-level indicators, application-level indicators and business-level indicators are collected respectively; an adaptive weight network is used to extract and weight fuse the indicators of different dimensions to generate a task resource fingerprint containing multi-dimensional attributes; wherein the system-level indicators at least include a central processing unit water level, a memory usage rate and a network throughput, the application-level indicators at least include a task directed acyclic graph depth and a data shuffling amount, and the business-level indicators at least include a business priority label and an activity period identifier. 3.The method for dynamic resource scheduling and optimization for big data services of claim 2, wherein, The active prediction model adopts a space-time fusion network structure, specifically comprising: a one-dimensional convolutional neural network is used to extract the local fluctuation characteristics of the resource usage curve in the time sliding window; a long short-term memory network is used to capture the long-term dependence relationship and periodic change trend of the resource load; the extracted features are mapped to the resource demand probability distribution at the future time through a fully connected layer.

4. The method for dynamic resource scheduling and optimization for big data services of claim 2, wherein, The calculation of the reservation ratio of the elastic buffer pool specifically includes: the variance of the resource demand prediction value is calculated as a basis for measuring prediction uncertainty; a mapping relationship function between prediction uncertainty and resource redundancy is established, and when the variance of the resource demand prediction value is large, the resource redundancy is nonlinearly increased to compensate for the prediction deviation through excessive reservation; a minimum maintenance threshold is set to ensure that a minimum resource set maintaining the basic service capability of the cluster is reserved during the load trough period.

5. The method for dynamic resource scheduling and optimization for big data services of claim 2, wherein, The multi-modal feature fusion process further includes a task portrait clustering step: based on the resource consumption mode of historical tasks, an unsupervised clustering algorithm is used to divide the tasks into several resource fingerprint categories; for different resource fingerprint categories, corresponding feature attention modules are activated in the active prediction model, and high weights are given to key dimensions in the input feature vector; wherein the resource fingerprint categories at least include compute-intensive, input-output-intensive and memory-intensive.

6. The method for dynamic resource scheduling and optimization for big data services of claim 1, wherein, After the container preheating operation is performed, a topology-aware scheduling step is further included: a topology mapping diagram of heterogeneous physical resources is constructed, the topology mapping diagram identifies the hardware attributes and real-time interference degree of the physical nodes; the affinity score between the resource demand vector of the task to be scheduled and the node state vector of the physical node is calculated; according to the affinity score, the preheated container generated by prediction is mapped to the physical node with the best local quality, so that the task characteristics and the physical hardware are adapted.

7. The method for dynamic resource scheduling and optimization for big data services of claim 6, wherein, The topology mapping of the constructed heterogeneous physical resources specifically comprises: A hardware type label of the physical node is acquired through static scanning, and the hardware type label distinguishes a graphic processor node, a high-throughput disk node, and a general-purpose computing node; A real-time state label of the physical node is acquired through dynamic monitoring, and the real-time state label comprises an input / output waiting rate and a co-located task interference index; The hardware type label and the real-time state label are combined to generate a three-dimensional resource space coordinate.

8. The method for dynamic resource scheduling and optimization for big data services of claim 6, wherein, An affinity score between a resource requirement vector of a task to be scheduled and a node state vector of a physical node adopts a vector matching algorithm: A sensitivity weight vector of the task to different hardware resources is defined; A closeness of the task resource requirement vector and the node state vector in a weighted Euclidean distance space is calculated; For a task with a high input / output requirement feature, a matching weight with a node with a non-volatile memory label is increased, so that the task is preferentially scheduled to a high-throughput node.

9. The method for dynamic resource scheduling and optimization for big data services of claim 6, wherein, A soft preemption step based on a service level agreement is further included: When a predicted requirement of a high-priority task exceeds a current available resource, a dynamic preemption mechanism is triggered; A target task in a low-priority task with an enabled checkpoint saving mechanism is identified; A suspension instruction is sent to the target task, a current execution state of the target task is saved to a distributed storage system, and a physical resource occupied by the target task is released for use by the high-priority task, instead of directly terminating the target task.

10. The method for dynamic resource scheduling and optimization for big data services of claim 1, wherein, A closed-loop feedback optimization step is further included: Actual load data and a task completion time after scheduling execution are collected in real time; A deviation loss between the actual load data and a resource requirement prediction value is calculated; A network parameter of the active prediction model is regularly updated by using a back propagation algorithm to correct a prediction drift caused by a change in a business mode.

Images