Method for dynamically allocating resources on carbon satellite application cloud based on data volume prediction

Abstract

The application provides a carbon satellite application cloud resource dynamic allocation method based on data volume prediction, relates to the cross field of cloud computing and remote sensing satellites, and collects process performance data and task characteristic labels of multiple processes of multiple data processing tasks in a carbon satellite application system through a fine-grained process monitoring module and an application feature labeling module, respectively, performs cloud resource dynamic optimization, outputs an elastic cloud resource optimization strategy, and dynamically adjusts the resource allocation of a basic computing resource pool to multiple data processing tasks for pre-allocation. The technical problems of the carbon satellite business system in the existing resource allocation method under the cloud environment, that is, the GPU / CPU utilization rate is in an extreme imbalance state due to the imbalance between supply and demand of computing resources, and high-priority tasks are interrupted. In the implementation of accurate matching of resource supply and demand, the technical effect of improving resource utilization rate while guaranteeing zero interruption of high-priority carbon emission data analysis tasks.

Description

Technical Field

[0001] This invention relates to the intersection of cloud computing and remote sensing satellites, and in particular to a method for dynamic allocation of cloud resources for carbon satellite applications based on data volume prediction. Background Technology

[0002] The carbon satellite service system faces significant technical bottlenecks in the cloud environment: the traditional static resource allocation model is difficult to adapt to dynamic changes in computing load, resulting in a serious imbalance between the supply and demand of computing resources.

[0003] Specifically, the business system exhibits significant periodicity in resource usage peaks and troughs. When satellite data arrives in batches or when there are sudden high-time-sensitivity tasks, computing resources are severely insufficient, while during low-load periods, resources are largely idle. The traditional static allocation model cannot match dynamic load changes, leading to a continuous imbalance between supply and demand.

[0004] In addition, the manual operation and maintenance response cycle is as long as several hours, which makes it impossible for the system to achieve elastic scaling at the minute level. At the same time, due to the lack of a task priority guarantee mechanism, high-time-sensitive critical tasks are frequently blocked by computationally intensive processes in a resource-competitive environment, resulting in task execution delays or even interruptions.

[0005] More notably, historical task execution data has not been effectively mined and utilized, and similar resource configurations have been repeatedly tested and erroneous. This requires both over-allocating 30% redundant resources to cope with sudden loads and increasing operating costs due to idle resources.

[0006] In summary, the carbon satellite service system suffers from technical problems under the existing resource allocation methods in the cloud environment, such as extreme imbalance in GPU / CPU utilization and interruption of high-priority tasks due to the imbalance between supply and demand of computing resources. Summary of the Invention

[0007] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a dynamic resource allocation method for carbon satellite applications in the cloud based on data volume prediction. This method solves the technical problem of extreme imbalance in GPU / CPU utilization and interruption of high-priority tasks caused by the imbalance between supply and demand of computing resources in the existing resource allocation method in the cloud environment of carbon satellite service systems.

[0008] To achieve the above objectives, this invention provides a method for dynamic resource allocation in a carbon satellite application cloud based on data volume prediction. This method is applied to a carbon satellite cloud resource optimization system, which includes a fine-grained process monitoring module, an application feature annotation module, a resource allocation decision module, and a resource dynamic adjustment module. The method includes:

[0009] The fine-grained process monitoring module performs millisecond-level process monitoring during the execution of multiple data processing tasks in the carbon satellite application system, obtaining multiple process performance data. The application feature annotation module performs automated multi-dimensional feature annotation during the execution of the multiple data processing tasks, obtaining multiple task feature labels. The data acquisition cycles of the fine-grained process monitoring module and the application feature annotation module are synchronized. The resource allocation decision module receives and, based on the multiple process performance data and multiple task feature labels uploaded synchronously by the fine-grained process monitoring module and the application feature annotation module, performs dynamic optimization of cloud resources and outputs a real-time cloud resource optimization strategy. The resource dynamic adjustment module, based on the ultra-short-term task incremental prediction results, elastically compensates the real-time cloud resource optimization strategy and outputs an elastic cloud resource optimization strategy. According to the elastic cloud resource optimization strategy, the resource allocation of the basic computing power resource pool is dynamically adjusted and pre-allocated to the multiple data processing tasks.

[0010] In one implementation, the following processing is also performed:

[0011] The carbon satellite cloud-based resource optimization system is monitored at the process level to obtain a real-time response efficiency vector, which is composed of task queue length, feature synchronization latency, strategy generation latency, and resource delivery latency. Preset module response latency thresholds are provided, including queue backlog thresholds, feature synchronization timeout thresholds, strategy generation timeout thresholds, and resource delivery timeout thresholds. A real-time response deviation vector is output by calculating the deviation between the real-time response efficiency vector and the module response latency thresholds. Based on the real-time response deviation vector, a resource replenishment mechanism is triggered to dynamically calibrate the module-level response coordination of the carbon satellite cloud-based resource optimization system.

[0012] In one implementation, the following processing is also performed:

[0013] The resource dynamic adjustment module is driven to update the ultra-short-term task incremental prediction results in a time sequence, and output the task incremental time sequence prediction results; the resource demand decline rate is calculated on the task incremental time sequence prediction results to obtain the resource utilization decline sequence; if the resource utilization decline sequence is less than the preset demand threshold for M consecutive prediction periods, the redundant elastic resources of the multiple data processing tasks are adjusted and the gradient reduction is recovered according to the real-time cloud resource optimization strategy, where M≥3 and M is a positive integer.

[0014] In one implementation, the resource allocation decision module receives and, based on the performance data of multiple processes and the feature tags of multiple tasks uploaded in time sequence by the fine-grained process monitoring module and the application feature labeling module, performs dynamic optimization of cloud resources, outputs a real-time cloud resource optimization strategy, and also performs the following processing:

[0015] A dual-channel resource decision network is pre-constructed and deployed to the resource allocation decision module. The dual-channel resource decision network includes a parallel emergency resource allocation decision sub-channel and a steady-state resource allocation decision sub-channel. First process performance data and first task feature labels are mapped and loaded into the emergency resource allocation decision sub-channel and the steady-state resource allocation decision sub-channel of the dual-channel resource decision network, and resource allocation decisions are executed in parallel, outputting a first real-time emergency allocation strategy and a first steady-state adaptive allocation strategy. The first process performance data and the first task feature labels are used as two-dimensional search conditions, and historical optimal strategies are queried based on a preset execution time reduction scale. A first deviation rate and a second deviation rate are calculated between the first real-time emergency allocation strategy and the first steady-state adaptive allocation strategy and the historical optimal strategy. The first real-time emergency allocation strategy and the first steady-state adaptive allocation strategy are weighted and fused according to the first deviation rate and the second deviation rate to output a first fused resource optimization strategy. The dynamic optimization of cloud resources based on multiple process performance data and multiple task feature labels is executed analogously to output multiple fused resource optimization strategies, constituting the real-time cloud resource optimization strategy.

[0016] In one implementation, the following processing is also performed in the emergency resource allocation decision subchannel:

[0017] Step a1: Use predefined multi-dimensional anomaly features to traverse the performance data of the first process for anomaly detection and locate the abnormal process occupancy features; Step b1: Perform emergency action matching based on the abnormal process occupancy features and output an emergency response instruction set; Step c1: Perform emergency resource rematching based on the emergency response instruction set and output the first real-time emergency allocation strategy.

[0018] In one implementation, the following processing is also performed in the steady-state resource allocation decision subchannel:

[0019] Step a2: Based on the label quantization rule, convert the first task feature label quantization into a first label quantization combination; Step b2: Based on the steady-state resource matching rule, allocate steady-state resource quotas to the first label quantization combination and output the first steady-state adaptation allocation strategy.

[0020] In one implementation, if the first deviation rate and / or the second deviation rate are greater than a preset deviation scale, the historical optimal strategy is output as the first fusion resource optimization strategy.

[0021] In one implementation, the resource dynamic adjustment module elastically compensates the real-time cloud resource optimization strategy based on the ultra-short-term task increment prediction results, outputs an elastic cloud resource optimization strategy, and also performs the following processing:

[0022] By analyzing the load characteristic parameters of the ultra-short-term task incremental prediction results through time series analysis, the incremental preprocessing task, the incremental GPU demand of the inversion task, and the peak load period are extracted; incremental over-limit scenarios are matched based on the incremental preprocessing task, the incremental GPU demand of the inversion task, and the peak load period; gradient resource supplementation is performed on the incremental over-limit scenarios, and incremental resource quotas are output; the incremental resource quotas are used to elastically compensate the real-time cloud resource optimization strategy, and the elastic cloud resource optimization strategy is output.

[0023] In one implementation, the resource dynamic adjustment module elastically compensates the real-time cloud resource optimization strategy based on the ultra-short-term task increment prediction results and outputs the elastic cloud resource optimization strategy. Before that, the following processing is also performed:

[0024] Based on the task types of the multiple data processing tasks, the historical data storage module is driven to retrieve historical incremental average time-series data; the fine-grained process monitoring module is interacted with to retrieve the disk I / O queue length change rate, and the task queue growth slope is calculated based on the disk I / O queue length change rate; the application feature labeling module is interacted with to retrieve periodic load labels as periodic load coefficients; the task queue growth slope and periodic load coefficients are used as a joint feature vector, and based on a preset prediction time window, the historical incremental average time-series data is matched and segmented to obtain an ultra-short-term task incremental average sequence; the ultra-short-term task incremental average sequence is decoupled by task type to obtain the preprocessing task increment, the inversion task GPU demand increment, and the peak load period; the preprocessing task increment, the inversion task GPU demand increment, and the peak load period are structurally integrated to obtain the ultra-short-term task increment prediction result.

[0025] In one implementation, the process performance data includes CPU utilization, memory usage, disk I / O throughput, network I / O traffic, and GPU usage metrics.

[0026] In one implementation, the task feature labels include computational complexity, data dependency, time sensitivity, and GPU dependency.

[0027] One or more technical solutions provided in this invention have at least the following technical effects or advantages:

[0028] The method provided in this invention drives the fine-grained process monitoring module to perform millisecond-level process monitoring during the execution of multiple data processing tasks in a carbon satellite application system, obtaining multiple process performance data; drives the application feature annotation module to perform multi-dimensional feature automatic annotation during the execution of the multiple data processing tasks, obtaining multiple task feature labels, wherein the data acquisition cycles of the fine-grained process monitoring module and the application feature annotation module are synchronized; the resource allocation decision module receives and, based on the multiple process performance data and multiple task feature labels uploaded in time sequence by the fine-grained process monitoring module and the application feature annotation module, performs dynamic optimization of cloud resources and outputs a real-time cloud resource optimization strategy; the resource dynamic adjustment module, based on the ultra-short-term task incremental prediction results, elastically compensates the real-time cloud resource optimization strategy and outputs an elastic cloud resource optimization strategy; according to the elastic cloud resource optimization strategy, the resource allocation of the basic computing power resource pool is dynamically adjusted to pre-allocate to the multiple data processing tasks. Through the dynamic adaptation of millisecond-level monitoring and reinforcement learning decision-making, the technical effect of zero interruption of high-priority carbon emission data analysis tasks is achieved while realizing precise matching of resource supply and demand and improving resource utilization. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This invention illustrates a flowchart of a cloud-based dynamic resource allocation method for carbon satellite applications based on data volume prediction, provided by the present invention.

[0031] Figure 2 A block diagram of the carbon satellite cloud resource optimization system provided by the present invention is shown. Detailed Implementation

[0032] This invention provides a dynamic resource allocation method for carbon satellite applications in the cloud based on data volume prediction. It is used to solve the technical problems of extreme imbalance in GPU / CPU utilization and interruption of high-priority tasks caused by the imbalance between supply and demand of computing resources in the existing resource allocation method in the cloud environment of carbon satellite service systems.

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0034] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0035] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0036] The flowchart of the cloud-based dynamic resource allocation method for carbon satellite applications based on data volume prediction provided in this embodiment of the invention is shown below. Figure 1 The method is applied to a carbon satellite cloud resource optimization system. This system includes a fine-grained process monitoring module, an application feature annotation module, a resource allocation decision module, and a resource dynamic adjustment module. A diagram illustrating the module composition of the carbon satellite cloud resource optimization system is provided below. Figure 2 .

[0037] The cloud-based dynamic resource allocation method for carbon satellite applications based on data volume prediction provided in this embodiment of the invention specifically includes:

[0038] A100: Drives the fine-grained process monitoring module to perform millisecond-level process monitoring during the execution of multiple data processing tasks in the carbon satellite application system, and obtains multiple process performance data.

[0039] In one implementation, the process performance data includes CPU utilization, memory usage, disk I / O throughput, network I / O traffic, and GPU usage metrics.

[0040] Specifically, in this embodiment, the fine-grained process monitoring module is driven to perform millisecond-level monitoring of multiple concurrently executed data processing tasks in the carbon satellite application system in order to collect key performance indicators and obtain multiple process performance data, including CPU utilization, memory usage, disk I / O throughput, network I / O traffic and GPU usage indicators.

[0041] In the fine-grained process monitoring module, CPU utilization represents the rationality of computing resource allocation. Server probes collect single-core and multi-core usage distribution data of processes at millisecond frequencies to analyze for resource skew or idleness. Memory usage represents the actual consumption level of physical and virtual memory, covering indicators such as used memory, memory fragmentation, and OS system usage. Real-time collection by a monitoring agent accurately locates memory leaks or abnormally occupied processes. Disk I / O throughput represents storage read / write performance, focusing on disk read / write rates and request queue length indicators. Data captured by the monitoring subsystem identifies system lag caused by I / O blocking. Network I / O traffic represents process communication overhead, tracking process uplink and downlink data packet rates in real time to provide a basis for dynamic network bandwidth allocation. GPU usage indicators represent the heterogeneous computing load status, acquiring GPU usage indicators (including usage rate, power consumption, memory usage, temperature, and health status data) through a resident monitoring agent.

[0042] A200: Drives the application feature annotation module to perform multi-dimensional feature automatic annotation during the execution of the multiple data processing tasks, and obtains multiple task feature labels. The data acquisition cycles of the fine-grained process monitoring module and the application feature annotation module are synchronized.

[0043] In one implementation, the task feature labels include computational complexity, data dependency, time sensitivity, and GPU dependency.

[0044] Specifically, in this embodiment, the application feature annotation module automatically performs multi-dimensional feature annotation on the data processing task, and its annotation cycle is strictly synchronized with the fine-grained process monitoring module to ensure spatiotemporal consistency.

[0045] The output of the application feature annotation module is multiple task feature labels, which include computational complexity, data dependency, time sensitivity, and GPU dependency.

[0046] Computational complexity characterizes the intensity of data processing tasks' consumption of computing resources. It is quantified through algorithm complexity analysis and instruction execution statistics. For example, tracking the number of iterative computation operations and data scale can determine time complexity and space complexity, thereby distinguishing between high computationally intensive tasks (such as carbon dioxide inversion) and low computationally intensive tasks.

[0047] Data dependency characterizes the constraints on the execution order between tasks. It is labeled by tracing the data generation, transmission and consumption paths. For example, it can be made clear that the radiation transmission simulation task that depends on its results can only be started after the data preprocessing task is completed, so as to avoid process interruption or version error.

[0048] Time sensitivity characterizes the urgency and periodicity of task execution, and time thresholds are set according to business rules for labeling: strict response time limits are set for high-priority tasks such as real-time data transmission, and time ranges and patterns are labeled for the periodic peaks and troughs of business systems.

[0049] GPU dependency characterizes the degree to which a task requires heterogeneous computing power. It is labeled by analyzing the parallelism characteristics of the algorithm. For example, the inversion algorithm is labeled as a GPU-required task because of its matrix operation requirements, while log processing is labeled as not requiring GPU support.

[0050] The features covered by the task feature labels are all generated through the automatic analysis capabilities of the application feature annotation module. The computational complexity is based on instruction statistical modeling, the data dependency is analyzed through data flow topology, the time sensitivity is associated with the business rule base, and the GPU dependency is identified based on parallel computing features. The entire process is conducted without human intervention.

[0051] A300: The resource allocation decision module receives and, based on the performance data of multiple processes and the feature tags of multiple tasks uploaded in time sequence by the fine-grained process monitoring module and the application feature labeling module, performs dynamic optimization of cloud resources and outputs real-time cloud resource optimization strategies.

[0052] In one implementation, the resource allocation decision module receives and, based on the performance data of multiple processes and multiple task feature tags synchronously uploaded by the fine-grained process monitoring module and the application feature labeling module, performs dynamic optimization of cloud resources and outputs a real-time cloud resource optimization strategy. Step A300 of the method provided by this invention further includes:

[0053] A310: Pre-construct a dual-channel resource decision network and deploy the dual-channel resource decision network to the resource allocation decision module, wherein the dual-channel resource decision network includes a parallel emergency resource allocation decision sub-channel and a steady-state resource allocation decision sub-channel.

[0054] A320: The first process performance data and the first task feature label are mapped and loaded into the emergency resource allocation decision sub-channel and the steady-state resource allocation decision sub-channel of the dual-channel resource decision network, and the resource allocation decision is executed in parallel to output the first real-time emergency allocation strategy and the first steady-state adaptive allocation strategy.

[0055] A330: Use the first process performance data and the first task feature label as two-dimensional search conditions, and perform historical optimal strategy query based on the preset execution time shortening scale.

[0056] A340: Calculate the first deviation rate and the second deviation rate of the first real-time emergency allocation strategy and the first steady-state adaptive allocation strategy from the historical optimal strategy.

[0057] A350: Based on the weighted fusion of the first real-time emergency allocation strategy and the first steady-state adaptive allocation strategy according to the first deviation rate and the second deviation rate, output the first fusion resource optimization strategy.

[0058] A360: By analogy, it performs dynamic optimization of cloud resources based on the performance data of multiple processes and the feature tags of multiple tasks, and outputs multiple integrated resource optimization strategies to form the real-time cloud resource optimization strategy.

[0059] In one implementation, in the emergency resource allocation decision subchannel:

[0060] Step a1: Use predefined multidimensional anomaly features to traverse the performance data of the first process to perform anomaly detection and locate the characteristics of the abnormal process occupying the data.

[0061] Step b1: Based on the characteristics of the abnormal process occupation, perform emergency action matching and output an emergency response instruction set.

[0062] Step c1: Perform emergency resource rematching based on the emergency response instruction set, and output the first real-time emergency allocation strategy.

[0063] In one implementation, in the steady-state resource allocation decision subchannel:

[0064] Step a2: Based on the label quantization rules, convert the first task feature label quantization into a first label quantization combination.

[0065] Step b2: Based on the steady-state resource matching rules, perform steady-state resource quota allocation on the first tag quantization combination and output the first steady-state adaptation allocation strategy.

[0066] In one implementation, if the first deviation rate and / or the second deviation rate are greater than a preset deviation scale, the historical optimal strategy is output as the first fusion resource optimization strategy.

[0067] Specifically, a dual-channel resource decision-making network is pre-built and deployed to the resource allocation decision-making module. This network consists of parallel emergency resource allocation decision-making sub-channels and steady-state resource allocation decision-making sub-channels. The emergency sub-channel is dedicated to real-time response to abnormal events, while the steady-state sub-channel is responsible for resource matching for routine tasks. The two work together to form a dynamically complementary decision-making architecture.

[0068] The dual-channel resource decision network is deployed to the resource allocation decision module to realize the resource allocation decision function of the module.

[0069] After inputting the first process performance data and the first task feature label into the resource allocation decision module, they are mapped and loaded into the emergency resource allocation decision sub-channel and the steady-state resource allocation decision sub-channel of the dual-channel resource decision network. The resource allocation decision is executed in parallel, and the first real-time emergency allocation strategy and the first steady-state adaptive allocation strategy are output. The dual-channel parallel mechanism ensures high timeliness (millisecond-level response).

[0070] Specifically, in the emergency resource allocation decision-making sub-channel:

[0071] Step a1 uses predefined multidimensional anomaly features to scan process performance data. When it detects that the GPU memory usage exceeds 95%, the CPU utilization is above 90% for 10 consecutive seconds, or the disk I / O request queue has more than 5 backlogged tasks, it automatically locates the memory leaking process, the computationally intensive blocking process, or the high I / O stuttering task.

[0072] Step b1 matches handling instructions according to the anomaly type to form the emergency handling instruction set. Specifically, if video memory overflows, low-priority tasks are terminated to release resources; if the CPU is overloaded, new task scheduling is paused and the quota of non-real-time tasks is reduced; if the disk is blocked, tasks are migrated to high-speed solid-state storage.

[0073] Step c1 is used to perform resource reallocation and output the first real-time emergency allocation strategy. Specifically, the released GPU is immediately transferred to a high-priority inversion task, the reclaimed CPU resources are allocated to a real-time data transmission task, and the migrated I / O tasks are bound to high-performance storage devices, forming an emergency strategy to ensure uninterrupted task execution.

[0074] Specifically, in the steady-state resource allocation decision sub-channel:

[0075] Step a2: Based on the label quantization rules (for example, in the computational complexity score, high computational intensive tasks are scored 3 points, medium tasks are scored 2 points, and low-load tasks are scored 1 point; in the GPU dependency score, GPU acceleration is required and scored 2 points, preferred tasks are scored 1 point, and no GPU acceleration is scored 0 points; in the time sensitivity classification, real-time tasks are scored 3 points, near-real-time tasks are scored 2 points, and offline tasks are scored 1 point), the first task feature label quantization is converted into a first label quantization combination in numerical combination form.

[0076] Step b2: Based on the steady-state resource matching rules, process the first tag quantization combination, allocate dedicated acceleration cards and reserve redundant computing power for computationally intensive tasks that require GPU acceleration; delay the start of tasks with strong data dependencies until the preprocessing is completed; grant resource preemption rights to high real-time tasks, and output the first steady-state adaptation allocation strategy.

[0077] Using the first set of process performance data and task feature tags as joint search conditions, the resource allocation scheme with the largest reduction in execution time is queried in the historical database. This scheme must meet a preset time optimization threshold (e.g., the historical task execution time is reduced by more than 20% compared to the average). The best historical strategy for similar tasks is selected as a benchmark reference.

[0078] Calculate the percentage deviation of the first real-time emergency allocation strategy (such as terminating low-priority tasks to release GPUs) output by the emergency decision sub-channel from the historical best strategy in terms of key resource indicators; simultaneously calculate the percentage deviation of the first steady-state adaptive allocation strategy (such as binding 4 GPU cards) output by the steady-state decision sub-channel from the historical best strategy; generate two sets of quantitative values: the first deviation rate (emergency strategy offset) and the second deviation rate (steady-state strategy offset).

[0079] Based on the weighted fusion of the first real-time emergency allocation strategy and the first steady-state adaptive allocation strategy using the first deviation rate and the second deviation rate, a first fusion resource optimization strategy is output that simultaneously satisfies immediate anomaly handling and long-term efficiency optimization.

[0080] For the process performance data and feature tags of all remaining tasks, the A330-A350 process is executed cyclically to output multiple fusion resource optimization strategies, which constitute the real-time cloud resource optimization strategy.

[0081] In one implementation, if the first deviation rate and / or the second deviation rate are greater than a preset deviation scale, the historical optimal strategy is output as the first fusion resource optimization strategy.

[0082] When the deviation rate of the emergency strategy or steady-state strategy exceeds the preset critical value (e.g., 20%), it indicates that there is a significant difference between the current decision and the efficient solution verified in history. At this time, the historical optimal strategy is directly adopted as the fusion result to avoid the risk of insufficient resource allocation.

[0083] This embodiment achieves precise insight into the status of heterogeneous resources through millisecond-level process monitoring and time-series synchronization of feature annotation. By leveraging dual-channel parallel decision-making to dynamically adapt to task fluctuations, it significantly improves the accuracy of resource allocation. It also balances emergency response and steady-state optimization effects by relying on historical deviation calibration and multi-dimensional strategy fusion mechanisms.

[0084] A400: The resource dynamic adjustment module, based on the ultra-short-term task incremental prediction results, elastically compensates the real-time cloud resource optimization strategy and outputs the elastic cloud resource optimization strategy.

[0085] In one implementation, the resource dynamic adjustment module elastically compensates the real-time cloud resource optimization strategy based on the ultra-short-term task increment prediction results and outputs an elastic cloud resource optimization strategy. The method step A400 provided by this invention further includes:

[0086] A410: By analyzing the load characteristic parameters of the ultra-short-term task increment prediction results through time series analysis, the preprocessed task increment, the inverted task GPU demand increment, and the peak load period are extracted.

[0087] A420: Match incremental over-limit scenarios based on the preprocessing task increment, inversion task GPU demand increment, and peak load period.

[0088] A430: Perform gradient resource replenishment for the incremental over-limit scenario and output incremental resource quota.

[0089] A440: The real-time cloud resource optimization strategy is compensated by the incremental resource quota, and the elastic cloud resource optimization strategy is output.

[0090] In one implementation, the resource dynamic adjustment module elastically compensates the real-time cloud resource optimization strategy based on the ultra-short-term task increment prediction results and outputs an elastic cloud resource optimization strategy. Previously, the method step A400 provided by this invention further includes:

[0091] A400-1: Based on the task type of the multiple data processing tasks, drive the historical data storage module to retrieve historical incremental average time series data.

[0092] A400-2: Interact with the fine-grained process monitoring module to retrieve the disk I / O queue length change rate, and calculate the task queue growth slope based on the disk I / O queue length change rate.

[0093] A400-3: The interactive application feature annotation module retrieves the periodic load label as the periodic load coefficient.

[0094] A400-4: Using the growth slope of the task queue and the periodic load coefficient as a joint feature vector, and based on a preset prediction time window, the ultra-short-term task incremental mean sequence is obtained by matching and segmenting the historical incremental mean time series data.

[0095] A400-5: Decouple the ultra-short-term task increment mean sequence by task type to obtain the preprocessing task increment, the inversion task GPU demand increment, and the peak load period.

[0096] A400-6: The preprocessing task increment, the inversion task GPU demand increment, and the peak load period are structurally integrated to obtain the ultra-short-term task increment prediction result.

[0097] Specifically, based on the task type classification of the multiple data processing tasks, the historical data storage module is driven to retrieve historical data on the growth of the number of tasks in the corresponding category, extract the time-series average change curve as a benchmark reference sequence, and output the historical incremental average time-series data.

[0098] The real-time data interface of the fine-grained process monitoring module is connected to extract the instantaneous change value stream of the disk I / O request queue length. By calculating the rate difference of queue length change within the continuous monitoring window, the task queue growth slope, which represents the growth rate of the real-time request backlog trend, is generated. A positive value indicates that the task backlog is aggravated, and a negative value indicates that the load is relieved. This slope directly reflects the intensity of real-time load fluctuations.

[0099] The interactive application feature annotation module retrieves periodic load labels as periodic load coefficients.

[0100] The dynamically generated task queue growth slope is combined with the static periodic coefficient to form a multi-dimensional feature vector. Based on a preset future time period prediction window (such as the next 5 minutes), the corresponding time segment is located in the historical incremental mean time series data, and the task incremental mean sequence within that time period is cut and extracted to form the benchmark data unit for ultra-short-term prediction, namely the ultra-short-term task incremental mean sequence.

[0101] The task type is decoupled from the average incremental sequence of ultra-short-term tasks. The incremental preprocessing tasks that mainly consume CPU resources are decomposed, and the incremental GPU demand of inversion tasks that rely on GPU acceleration is parsed. The peak time interval of resource demand marked in the periodic coefficient is extracted simultaneously to obtain the peak load period.

[0102] The preprocessing task increment, the inversion task GPU demand increment, and the peak load period are structurally integrated to obtain the ultra-short-term task increment prediction result.

[0103] By comparing whether the incremental preprocessing task exceeds the CPU resource capacity limit, whether the GPU demand of the inversion task exceeds the number of available graphics cards, and whether the peak period coincides with the system idle period, the specific scenario type of insufficient resource supply (such as computing resource shortage / accelerator card shortage / time period conflict) is determined, and the incremental over-limit scenario is obtained.

[0104] For the incremental over-limit scenario, perform phased resource replenishment. For example, add 30% of the predicted demand as basic resources for the first time (such as adding 4 CPU cores + 1 GPU), and add 20% for the second time. Output a list of the specific resource types and quantities that need to be expanded to obtain the incremental resource quota.

[0105] The incremental resource quota is arithmetically added to the real-time cloud resource optimization strategy to output the elastic cloud resource optimization strategy. This embodiment achieves the technical effect of precise matching of resource supply and demand and reducing the waste of idle resources through elastic scaling.

[0106] A500: Based on the elastic cloud resource optimization strategy, dynamically adjust the resource allocation of the basic computing power resource pool to pre-allocate to the multiple data processing tasks.

[0107] This embodiment dynamically adjusts the resource allocation ratio of the basic computing power resource pool based on the resource increment quota and effective time period instructions specified in the elastic cloud resource optimization strategy: the CPU resource pool core allocation ratio is expanded according to the incremental value of preprocessing tasks, the GPU card allocation number is expanded according to the incremental value of GPU demand of inversion tasks, and the expanded resources are pre-allocated to the corresponding data processing tasks during peak load periods to achieve real-time matching between resource supply and task demand.

[0108] In one implementation, the method provided by the present invention further includes:

[0109] A610: Perform process-level response efficiency monitoring on the carbon satellite cloud resource optimization system to obtain a real-time response efficiency vector, wherein the real-time response efficiency vector consists of task queue length, feature synchronization delay, strategy generation delay, and resource delivery delay.

[0110] A620: Preset module response delay threshold, wherein the module response delay threshold includes queue backlog threshold, feature synchronization timeout threshold, policy generation timeout threshold, and resource delivery timeout threshold.

[0111] A630: Output the real-time response deviation vector by calculating the deviation between the real-time response efficiency vector and the module response delay threshold.

[0112] A640: Based on the real-time response deviation vector, trigger the resource replenishment mechanism to perform module-level response coordination dynamic calibration of the carbon satellite cloud resource optimization system.

[0113] Specifically, in this embodiment, the carbon satellite cloud resource optimization system performs process-level efficiency monitoring by collecting four indicators in real time: task queue backlog length, feature annotation data synchronization delay, resource allocation strategy generation time, and resource quota delivery delay, forming a real-time vector dataset characterizing the system's response efficiency.

[0114] Preset latency tolerance thresholds for each stage. Specifically, set the maximum backlog of the task queue as the queue threshold, the maximum allowed time for feature synchronization as the synchronization threshold, the policy generation time limit as the generation threshold, and the resource delivery deadline as the delivery threshold to build a module-level response baseline standard.

[0115] Calculate the absolute deviation of each indicator in the real-time response vector from its corresponding threshold. Specifically, the task queue deviation is the actual backlog minus the threshold, the feature synchronization deviation is the actual latency minus the threshold, the strategy generation deviation is the actual time consumed minus the threshold, and the resource delivery deviation is the actual latency minus the threshold. Generate a real-time response deviation vector containing four-dimensional deviation values.

[0116] Based on the four-dimensional numerical triggering of real-time response deviation vector, dynamic resource replenishment is achieved. Specifically, when any deviation value exceeds the zero point, a corresponding computing unit is added to the resource pool; when the queue deviation exceeds the standard, the process processing node is expanded; when the synchronous deviation exceeds the standard, the data transmission bandwidth is increased; when the deviation exceeds the standard, the decision computing power is improved; and when the delivery deviation exceeds the standard, the resource scheduling link is accelerated, thereby realizing dynamic collaborative calibration of the response efficiency of each module.

[0117] In one implementation, the method provided by the present invention further includes:

[0118] A710: Drives the resource dynamic adjustment module to update the ultra-short-term task increment prediction results in a timely manner, and outputs the task increment time-series prediction results.

[0119] A720: Calculate the resource demand decline rate based on the incremental time-series prediction results of the task to obtain the resource utilization decline sequence.

[0120] A730: If the resource utilization rate decrease sequence is less than the preset demand threshold for M consecutive prediction periods, then the redundant elastic resources of the multiple data processing tasks are adjusted and the recovery is carried out in a gradient manner according to the real-time cloud resource optimization strategy, where M≥3 and M is a positive integer.

[0121] Specifically, in this embodiment, the drive resource dynamic adjustment module updates the ultra-short-term task increment prediction results at fixed time intervals (such as every minute), and regenerates the task quantity growth prediction value for future periods based on the latest collected process performance data and feature labels, forming a continuous time-series prediction data sequence output.

[0122] Perform resource demand variation analysis on the time-series forecast data sequence. Specifically, calculate the percentage decrease in task increment within adjacent forecast periods (current period increment / previous period increment - 1). When the result is negative, it indicates a decrease in resource demand. Output the resource utilization rate decrease curve for multiple consecutive periods.

[0123] When the decline in resource utilization rate is consistently below the preset demand threshold (e.g., -10%) for three or more consecutive forecast periods, the redundant resource recovery procedure is initiated: initially, 20% of the idle computing units in the elastic resource pool are reduced (e.g., releasing 4 CPU cores), and in subsequent periods, the resource recovery is gradually carried out according to the gradient reduction rule until the downward trend is reversed, ensuring that the pace of resource release is synchronized with the decline in demand.

[0124] This embodiment achieves the technical effect of effectively eliminating the waste of idle computing resources through a prediction-driven gradient resource recycling mechanism.

[0125] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for dynamic allocation of cloud resources for carbon satellite applications based on data volume prediction, characterized in that, The method is applied to a carbon satellite cloud-based resource optimization system, which includes a fine-grained process monitoring module, an application feature annotation module, a resource allocation decision module, and a resource dynamic adjustment module. The method includes: The fine-grained process monitoring module is driven to perform millisecond-level process monitoring during the execution of multiple data processing tasks in the carbon satellite application system, thereby obtaining multiple process performance data. The application feature annotation module is driven to perform multi-dimensional feature automatic annotation during the execution of the multiple data processing tasks, thereby obtaining multiple task feature labels. The data acquisition cycles of the fine-grained process monitoring module and the application feature annotation module are synchronized. The resource allocation decision module receives and dynamically optimizes cloud resources based on the performance data of multiple processes and the feature tags of multiple tasks that are uploaded in time sequence by the fine-grained process monitoring module and the application feature labeling module, and outputs real-time cloud resource optimization strategies. The resource dynamic adjustment module, based on the ultra-short-term task incremental prediction results, elastically compensates the real-time cloud resource optimization strategy and outputs the elastic cloud resource optimization strategy. Based on the elastic cloud resource optimization strategy, the resource allocation ratio of the basic computing power resource pool is dynamically adjusted and pre-allocated to the multiple data processing tasks; The resource allocation decision module receives and dynamically optimizes cloud resources based on the performance data of multiple processes and the feature tags of multiple tasks, which are uploaded in real time by the fine-grained process monitoring module and the application feature labeling module, and outputs real-time cloud resource optimization strategies, including: A dual-channel resource decision network is pre-constructed and deployed to the resource allocation decision module. The dual-channel resource decision network includes a parallel emergency resource allocation decision sub-channel and a steady-state resource allocation decision sub-channel. The first process performance data and the first task feature label are mapped and loaded into the emergency resource allocation decision sub-channel and the steady-state resource allocation decision sub-channel of the dual-channel resource decision network, and the resource allocation decision is executed in parallel to output the first real-time emergency allocation strategy and the first steady-state adaptive allocation strategy. Using the first process performance data and the first task feature label as two-dimensional search conditions, historical optimal strategy is queried based on a preset execution time shortening scale. Calculate the first deviation rate and the second deviation rate of the first real-time emergency allocation strategy and the first steady-state adaptive allocation strategy from the historical optimal strategy; Based on the weighted fusion of the first real-time emergency allocation strategy and the first steady-state adaptive allocation strategy according to the first deviation rate and the second deviation rate, the first fusion resource optimization strategy is output. By analogy, the cloud resource dynamic optimization is performed based on the performance data of multiple processes and the feature tags of multiple tasks, and multiple integrated resource optimization strategies are output to form the real-time cloud resource optimization strategy.

2. The method for dynamic allocation of cloud resources for carbon satellite applications based on data volume prediction as described in claim 1, characterized in that, Also includes: The carbon satellite cloud resource optimization system is monitored at the process level to obtain a real-time response efficiency vector, which is composed of task queue length, feature synchronization delay, strategy generation delay and resource delivery delay. A preset module response latency threshold is provided, wherein the module response latency threshold includes a queue backlog threshold, a feature synchronization timeout threshold, a policy generation timeout threshold, and a resource delivery timeout threshold. The real-time response deviation vector is output by calculating the deviation between the real-time response efficiency vector and the module response delay threshold. Based on the real-time response deviation vector, a resource replenishment mechanism is triggered to perform module-level response coordination dynamic calibration of the carbon satellite cloud resource optimization system.

3. The method for dynamic allocation of cloud resources for carbon satellite applications based on data volume prediction as described in claim 1, characterized in that, Also includes: The resource dynamic adjustment module is driven to update the ultra-short-term task increment prediction results in a time sequence, and the task increment time sequence prediction results are output. The resource demand decline rate is calculated based on the incremental time-series prediction results of the task to obtain the resource utilization decline rate sequence; If the decrease in resource utilization rate is less than the preset demand threshold for M consecutive prediction periods, then the redundant elastic resources of the multiple data processing tasks will be gradually reduced and recovered according to the real-time cloud resource optimization strategy, where M≥3 and M is a positive integer.

4. The method for dynamic allocation of cloud resources for carbon satellite applications based on data volume prediction as described in claim 1, characterized in that, In the emergency resource allocation decision sub-channel: Step a1: Use predefined multidimensional anomaly features to traverse the performance data of the first process to perform anomaly detection and locate the characteristics of the abnormal process occupying the data. Step b1: Based on the characteristics of the abnormal occupancy process, perform emergency action matching and output an emergency response instruction set; Step c1: Perform emergency resource rematching based on the emergency response instruction set, and output the first real-time emergency allocation strategy.

5. The method for dynamic allocation of cloud resources for carbon satellite applications based on data volume prediction as described in claim 1, characterized in that, In the steady-state resource allocation decision sub-channel: Step a2: Based on the label quantization rules, convert the first task feature label quantization into a first label quantization combination; Step b2: Based on the steady-state resource matching rules, perform steady-state resource quota allocation on the first tag quantization combination and output the first steady-state adaptation allocation strategy.

6. The method for dynamic allocation of cloud resources for carbon satellite applications based on data volume prediction as described in claim 1, characterized in that, If the first deviation rate and / or the second deviation rate are greater than the preset deviation scale, then the historical optimal strategy will be output as the first fusion resource optimization strategy.

7. The method for dynamic allocation of cloud resources for carbon satellite applications based on data volume prediction as described in claim 1, characterized in that, The resource dynamic adjustment module, based on the ultra-short-term task incremental prediction results, elastically compensates the real-time cloud resource optimization strategy and outputs an elastic cloud resource optimization strategy, including: By analyzing the load characteristic parameters of the ultra-short-term task increment prediction results in time series, the preprocessing task increment, the inverted task GPU demand increment, and the peak load period are extracted. Based on the incremental preprocessing task, the incremental GPU demand for the inversion task, and the peak load period, the incremental over-limit scenario is matched; For the incremental over-limit scenario, gradient resource replenishment is performed, and incremental resource quota is output; The real-time cloud resource optimization strategy is compensated by the incremental resource quota, and the elastic cloud resource optimization strategy is output.

8. The method for dynamic allocation of cloud resources for carbon satellite applications based on data volume prediction as described in claim 7, characterized in that, The resource dynamic adjustment module, based on the ultra-short-term task incremental prediction results, elastically compensates the real-time cloud resource optimization strategy and outputs an elastic cloud resource optimization strategy, which includes: Based on the task types of the multiple data processing tasks, the historical data storage module is driven to retrieve historical incremental average time series data; The fine-grained process monitoring module retrieves the disk I / O queue length change rate and calculates the task queue growth slope based on the disk I / O queue length change rate. The interactive application feature annotation module retrieves periodic load labels as periodic load coefficients; Using the task queue growth slope and periodic load coefficient as a joint feature vector, and based on a preset prediction time window, the ultra-short-term task incremental mean sequence is obtained by matching and segmenting the historical incremental mean time series data. The task type is decoupled from the ultra-short-term task increment mean sequence to obtain the preprocessing task increment, the inversion task GPU demand increment, and the peak load period; The preprocessing task increment, the inversion task GPU demand increment, and the peak load period are structurally integrated to obtain the ultra-short-term task increment prediction result.

9. The method for dynamic allocation of cloud resources for carbon satellite applications based on data volume prediction as described in claim 1, characterized in that, The process performance data includes CPU utilization, memory usage, disk I / O throughput, network I / O traffic, and GPU usage metrics, and the task feature labels include computational complexity, data dependency, time sensitivity, and GPU dependency.

Images