Distributed computing power scheduling method and system based on AIGC

Abstract

The present application discloses a distributed computing power scheduling method and system based on AIGC, which relates to the field of computing power scheduling. First, the GPU indicator stream and request indicator stream are obtained in real time, and time series alignment and feature engineering are performed on them. At the same time, semantic feature extraction is performed on the business intent text, and the above-mentioned processed features are fused into vectorized system states and input into the generative AI scheduling agent to predict future model requirements and output the model requirement prediction JSON. Subsequently, the planner module generates an optimized cache action plan based on this prediction and the current cache status to guide the scheduling executor to pre-load or intelligently replace the model. Finally, when the request router receives a new request, it can query the updated cache status and directly distribute the request to the GPU node that has loaded the target model, thereby significantly reducing the cold start delay and improving the computing power resource utilization efficiency and user experience.

Description

Technical Field

[0001] The present application relates to the field of computing power scheduling, and more specifically, to a distributed computing power scheduling method and system based on AIGC. Background Art

[0002] The rapid development of artificial intelligence (AI) technology, particularly the widespread application of AIGC (artificial intelligence generated content) models, has posed unprecedented challenges to the underlying computing infrastructure. These large AIGC models, such as text generation and image generation models, are often enormous, taking seconds or even tens of seconds to load into GPU memory, resulting in significant cold start latency. In a distributed computing environment, allocating GPU resources exclusively to each model for a long period of time is not only costly but also wastes significant resources. Therefore, sharing computing resources is essential. However, when multiple business departments or users generate unpredictable demands for diverse AIGC models at different times, efficiently and intelligently scheduling limited GPU resources to avoid frequent cold starts becomes a critical bottleneck that impacts user experience and service level agreements (SLAs).

[0003] Traditional computing scheduling methods typically rely on preset rules or simple statistical predictions based on historical data, making them difficult to cope with the dynamic, diverse, and highly unpredictable nature of AIGC model requests. For example, business demand can vary significantly depending on the time of day: in the morning, the focus might be on image generation, while in the afternoon, it might shift to text analysis. Existing systems often cannot anticipate these changes in patterns. When all GPU nodes have loaded models and are busy, a new model request arrives, and the scheduler must make a model replacement decision. If this decision lacks accurate predictions of future demand and blindly evicts currently frequently accessed models, it will lead to new cold starts, further increasing latency and reducing resource utilization efficiency. Therefore, existing technologies are unable to effectively balance resource utilization and user experience when handling this complex resource allocation problem that involves both prediction and decision-making.

[0004] In view of the above challenges, there is an urgent need for a more intelligent and adaptive distributed computing power scheduling solution. Summary of the Invention

[0005] Taking into account the above limitations in application, according to one aspect of the present application, a distributed computing power scheduling method based on AIGC is provided, which includes: obtaining a GPU indicator stream, a request indicator stream and a business intent text; performing time series alignment and feature engineering on the GPU indicator stream and the request indicator stream to obtain GPU indicator timing features, request indicator timing features and GPU indicator-request indicator cross-domain correlation features; performing semantic feature extraction on the business intent text to obtain business intent text semantic embedding; fusing the GPU indicator timing features, the request indicator timing features, the GPU indicator-request indicator cross-domain correlation features and the business intent text semantic embedding to obtain a vectorized system state; inputting the vectorized system state into a generative AI scheduling agent to obtain a model demand prediction JSON; inputting the model demand prediction JSON and the current cache state into a planner module to obtain a cache action plan, wherein the current cache state is used to indicate which GPU node has currently loaded which model; a scheduling executor executes the cache action plan to obtain an updated cache state; after the request router receives a newly arrived inference request, the request router queries the updated cache state to generate a distribution decision.

[0006] According to another aspect of the present application, a distributed computing power scheduling system based on AIGC is provided, which includes: a data acquisition module for acquiring a GPU indicator stream, a request indicator stream and a business intention text; a data feature processing module for performing time series alignment and feature engineering on the GPU indicator stream and the request indicator stream to obtain GPU indicator timing features, request indicator timing features and GPU indicator-request indicator cross-domain correlation features; a business intention text encoding module for performing semantic feature extraction on the business intention text to obtain business intention text semantic embedding; a system state fusion module for fusing the GPU indicator timing features, the request indicator timing features, the GPU indicator-request indicator Cross-domain correlation features and the business intent text semantics are embedded to obtain a vectorized system state; a model demand prediction module is used to input the vectorized system state into a generative AI scheduling agent to obtain a model demand prediction JSON; a data cache module is used to input the model demand prediction JSON and the current cache state into a planner module to obtain a cache action plan, and the current cache state is used to indicate which GPU node has currently loaded which model; a cache state update module is used to schedule an executor to execute the cache action plan to obtain an updated cache state; a distribution module is used to request a router to query the updated cache state to generate a distribution decision after receiving a newly arrived inference request.

[0007] Compared with the existing technology, the present application provides a distributed computing power scheduling method and system based on AIGC, which realizes accurate prediction of future model requirements and proactive resource management by deeply integrating multi-source heterogeneous data. Specifically, the system obtains GPU indicator streams and request indicator streams in real time, and performs time series alignment and feature engineering on them, while extracting semantic features from business intent texts. These multi-dimensional information are integrated into vectorized system states and input into the generative AI scheduling agent. The agent uses its generation capability to predict future model requirements and outputs model requirement prediction JSON. Subsequently, the planner module generates an optimized cache action plan based on this prediction and the current cache status, guiding the scheduling executor to pre-load or intelligently replace the model. Finally, when the request router receives a new request, it can query the updated cache status and directly distribute the request to the GPU node that has loaded the target model, thereby significantly reducing cold start delay, improving computing power resource utilization efficiency and user experience, and effectively overcoming the limitations of traditional scheduling solutions in a dynamic AIGC environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0009] Figure 1 This is a flowchart of a distributed computing power scheduling method based on AIGC according to an embodiment of the present application.

[0010] Figure 2 This is a data flow diagram of the distributed computing power scheduling method based on AIGC according to an embodiment of the present application.

[0011] Figure 3 This is a flowchart of step S2 in the distributed computing power scheduling method based on AIGC according to an embodiment of the present application.

[0012] Figure 4 This is a flowchart of step S6 in the distributed computing power scheduling method based on AIGC according to an embodiment of the present application.

[0013] Figure 5 This is a block diagram of an AIGC-based distributed computing power scheduling system according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. While the drawings illustrate certain embodiments of the present disclosure, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0015] In response to the limitations of existing background technologies, this application proposes a distributed computing power scheduling method based on AIGC. Figure 1 This is a flowchart of a distributed computing power scheduling method based on AIGC according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of the distributed computing power scheduling method based on AIGC according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to an embodiment of the present application, the distributed computing power scheduling method based on AIGC includes: S1, obtaining a GPU indicator stream, a request indicator stream and a business intent text; S2, performing time series alignment and feature engineering on the GPU indicator stream and the request indicator stream to obtain GPU indicator timing features, request indicator timing features and GPU indicator-request indicator cross-domain correlation features; S3, performing semantic feature extraction on the business intent text to obtain business intent text semantic embedding; S4, fusing the GPU indicator timing features, the request indicator timing features, the GPU indicator-request indicator cross-domain correlation features and the business intent text semantic embedding to obtain a vectorized system state; S5, inputting the vectorized system state into a generative AI scheduling agent to obtain a model demand prediction JSON; S6, inputting the model demand prediction JSON and the current cache state into a planner module to obtain a cache action plan, wherein the current cache state is used to indicate which GPU node has loaded which model; S7, a scheduling executor executes the cache action plan to obtain an updated cache state; S8, after the request router receives the newly arrived inference request, the request router queries the updated cache state to generate a distribution decision.

[0016] In step S1, the GPU metric stream, request metric stream, and business intent text are obtained. It should be understood that obtaining the GPU metric stream, request metric stream, and business intent text is a key foundation for building an AIGC-based distributed computing scheduling solution, aiming to overcome the inherent shortcomings of existing technologies in addressing the high cold start latency, low resource utilization, and unpredictable demand of large AIGC models. The GPU metric stream provides insights into the real-time health and availability of underlying computing resources, which is crucial for assessing the current GPU node load and identifying idle or overloaded nodes. It is the physical foundation for effective resource allocation and avoiding resource waste. The request metric stream reveals historical and real-time model request patterns. This data is the core basis for understanding the dynamic changes in model demand, predicting future request trends, and quantifying scheduling effectiveness. As a forward-looking, unstructured information source, business intent text can capture explicit instructions and future plans from operations personnel or business departments. This enables the scheduling solution to go beyond pure historical data analysis and proactively predict upcoming business peaks or specific model demand. This allows for model preloading or resource reservation before actual requests occur, fundamentally alleviating cold start issues and significantly improving user experience and meeting service level agreements.

[0017] Specifically, step S1 is performed as follows: The GPU metric stream is acquired by deploying lightweight monitoring agents on each distributed GPU node. These agents continuously monitor and collect real-time telemetry data from their GPUs, including but not limited to the GPU node's unique identifier (ID), video memory usage (VRAM usage), and computing power utilization. For example, an agent deployed on a GPU node will obtain the current GPU's VRAM usage (e.g., 75%) and computing power utilization (e.g., 60%) at preset fixed intervals (e.g., every 5 seconds) by calling underlying hardware interfaces such as the NVIDIA Management Library (NVML). These collected data points are then encapsulated into structured data packets and streamed in real time to a central data aggregation service, forming a continuous GPU metric stream.

[0018] The acquisition of the request metrics stream depends on the entry point of the inference request and the model service execution layer. When a new inference request arrives at the request router, its metadata, such as the request timestamp, the requested model name, and the user ID that initiated the request, are recorded immediately. After the model completes inference and returns a response, relevant performance metrics, such as response latency and whether a model cold start has occurred (i.e., whether the model needs to be loaded from storage to video memory when the request arrives), are also captured. These real-time request logs and performance data, such as a request for the Llama-2-70B model initiated by user A at 16:00:05, which incurs a response latency of 120 milliseconds and is marked as a cold start, are immediately pushed to a high-performance data stream processing pipeline, such as a stream based on Apache Kafka, to build a continuous request metrics stream.

[0019] The purpose of capturing business intent text is to capture unstructured, forward-looking business demand information. This can be achieved in two main ways: first, by providing a dedicated text input box in the scheduling management interface, allowing operations personnel or business leaders to directly enter future plans, such as preparing the financial report analysis model fin-llama-v2 for next week; second, by integrating with the API interfaces of enterprise collaboration tools such as Jira and Slack. By configuring these interfaces, the scheduling plan can monitor the text content in specific projects, channels, or tags. When text containing preset keywords, such as preloading and deploying new models, is detected, the relevant text content is automatically extracted. This captured unstructured text information is then transmitted to the subsequent processing module as business intent text input.

[0020] In step S2, the GPU indicator stream and the request indicator stream are time series aligned and feature engineered to obtain GPU indicator timing features, request indicator timing features and GPU indicator-request indicator cross-domain correlation features. Accordingly, since the original GPU indicator stream and request indicator stream are high-frequency, discrete and heterogeneous real-time data, it is difficult to capture the deep dynamic laws of computing resources and model requests by direct use. The problems of high cold start latency, low resource utilization and demand unpredictability mentioned in the background technology are rooted in the lack of a comprehensive and quantitative understanding of the complex relationship between computing power status and model requirements. To this end, the present application can convert these raw data into structured and meaningful timing features through time series alignment and feature engineering, providing high-quality, high-dimensional input for subsequent generative AI scheduling agents, thereby achieving accurate prediction of future model requirements and more intelligent resource scheduling decisions, effectively alleviating cold start problems and improving overall computing power efficiency.

[0021] Specifically, in one embodiment, Figure 3 FIG is a flow chart of step S2 in the distributed computing power scheduling method based on AIGC according to an embodiment of the present application. Figure 3 As shown, step S2, performing time series alignment and feature engineering on the GPU indicator flow and the request indicator flow to obtain GPU indicator timing characteristics, request indicator timing characteristics and GPU indicator-request indicator cross-domain correlation characteristics, including: S21, performing time window aggregation on the GPU indicator flow and the request indicator flow, and performing statistical analysis on the GPU indicator flow and the request indicator flow within the time window to obtain initial GPU indicator timing characteristics, initial request indicator timing characteristics and initial GPU indicator-request indicator cross-domain correlation characteristics, wherein the initial GPU indicator timing characteristics include the average idle time in the past hour The percentage of video memory and the maximum computing power utilization in the past 5 minutes; the initial request indicator timing characteristics include the number of requests of each model in the past hour, the p95 latency and cold start incidence rate of each model in the past hour; the initial GPU indicator-request indicator cross-domain correlation characteristics include the average load of the GPU when model X is active; S22, normalizing the initial GPU indicator timing characteristics, the initial request indicator timing characteristics, and the initial GPU indicator-request indicator cross-domain correlation characteristics to obtain the GPU indicator timing characteristics, the request indicator timing characteristics, and the GPU indicator-request indicator cross-domain correlation characteristics.

[0022] Specifically, step S2 is performed using the following method: First, in step S21. For the GPU indicator stream, first, two sliding time windows are set: one is the past 1-hour window, and the other is the past 5-minute window, so as to capture the long-term trend and short-term instantaneous peak of computing resources and request patterns at the same time. For example, if the current time point is 10:00:00, the past 1-hour window covers data from 09:00:00 to 10:00:00, and the past 5-minute window covers data from 09:55:00 to 10:00:00. For each GPU node, within the past 1-hour window, all its video memory usage data points are collected, and its average idle video memory percentage is calculated, that is, 1 minus the average video memory usage; within the past 5-minute window, all its computing power utilization data points are collected, and the maximum value among them is calculated as the maximum computing power utilization of the GPU node in the past 5 minutes. For example, if a GPU's average memory utilization over the past hour was 70%, its average idle memory percentage is 30%. If its compute utilization peaked at 95% over the past five minutes, its maximum compute utilization is 95%. The scheduling method also uses a sliding window over the past hour for aggregation and statistical analysis of request metrics. For each known model, the total number of requests for that model within the past hour is counted to obtain the number of requests for each model over the past hour. For example, the Stable-Diffusion-XL model was requested 200 times over the past hour. Response latency data for all requests to this model over the past hour is collected, and its p95 latency is calculated. This is the value at which 95% of the request latencies are less than or equal to this value, reflecting the model's performance under peak load. For example, if the p95 latency for Stable-Diffusion-XL is 450 milliseconds, this means that 95% of requests are responded to within 450 milliseconds. Furthermore, the number of cold start requests for this model over the past hour is counted and divided by the total number of requests to obtain the cold start rate. For example, if fin-llama-v2 had 100 requests in the past hour, 20 of which were cold starts, then its cold start rate is 20%. Finally, to obtain the initial GPU metric-request metric cross-domain correlation feature, the GPU metric stream and the request metric stream are aligned and correlated by time series analysis. Specifically, for each model X, the scheduling method identifies on which GPU nodes the model X was activated and processed requests in the past hour. Then, the average value of the computing power utilization data of these specific GPU nodes during the period when model X was active is calculated to obtain the average GPU load when model X was active. For example, if the Llama-2-70B model was mainly run on GPU002 in the past hour, and the average computing power utilization of GPU002 during this period was 85%, then the feature value is 85%.Finally, the initial feature set is calculated, including the initial GPU indicator timing features, the initial request indicator timing features, and the initial GPU indicator-request indicator cross-domain correlation features.

[0023] Then, in step S22. It is easy to know that the purpose of normalization is to eliminate the differences in dimensions and numerical ranges between different features, ensure that all features have the same weight or influence in the subsequent generative AI scheduling agent, and avoid features with larger values ​​dominating model training. This method adopts the Min-Max normalization method to linearly scale the value of each feature to a preset fixed interval, such as [0,1]. The specific calculation formula is: normalized value = (original value - minimum value) / (maximum value - minimum value). Among them, the minimum and maximum values ​​of each feature can be obtained based on historical data statistics or preset based on experience. For example, for the average idle video memory percentage in the past hour, the minimum value can be set to 0 and the maximum value can be set to 1; for the number of requests of each model in the past hour, if the historical maximum number of requests is 10,000, the minimum value is set to 0 and the maximum value is set to 10,000. In this way, for example, a request number feature with an original value of 200 will be normalized to 0.02 when the maximum value is 10,000. After normalization, all features are on a unified numerical scale, and GPU indicator timing features, request indicator timing features, and GPU indicator-request indicator cross-domain correlation features are obtained.

[0024] In step S3, semantic features are extracted from the business intent text to obtain a semantic embedding of the business intent text. It should be understood that the background technology clearly points out that traditional scheduling methods are difficult to cope with the diversity and unpredictability of AIGC model requests, especially when business demands are completely different in different time periods and there is a lack of ability to predict future patterns. The business intent text contains clear indications or potential trends of future model requirements, which is valuable information that cannot be obtained by traditional scheduling solutions based on historical data and real-time indicators. However, this text information is unstructured and cannot be directly used for numerical calculations and model inputs. Therefore, through semantic feature extraction, these texts can be converted into high-dimensional numerical vectors (i.e., semantic embeddings) to capture the deep semantic meaning and contextual information of the text, so that the scheduling solution can understand and utilize these forward-looking intentions, thereby achieving smarter and more predictive scheduling decisions.

[0025] Specifically, in one embodiment, step S3, performing semantic feature extraction on the business intent text to obtain semantic embedding of the business intent text, includes: inputting the business intent text into a pre-trained sentence vector model to obtain the semantic embedding of the business intent text.

[0026] Specifically, step S3 is performed as follows: the pre-trained sentence vector model adopts an encoder model based on the Transformer architecture, such as a variant of BERT or RoBERTa, or a model specially designed for generating sentence vectors such as Sentence-BERT. Taking Sentence-BERT as an example, its basic architecture is based on the pre-trained BERT model and is fine-tuned through a twin network or a triplet network structure. The BERT model itself is composed of a stack of multiple layers of Transformer encoders, each layer of which contains a multi-head self-attention mechanism and a feedforward network, which can capture the complex dependencies and contextual information between words in the input text. The weights and bias parameters of the model are pre-trained through self-supervised learning tasks on a large-scale unlabeled text corpus, which gives the model powerful language understanding capabilities.

[0027] After obtaining the business intent text, such as the model fin-llama-v2 preparing for next week's financial report analysis, the text will first be segmented and converted into an input sequence acceptable to the model, for example, using WordPiece or Byte-PairEncoding. These segmented words are then input into the pre-trained sentence vector model along with special tags such as [CLS] and [SEP]. The model processes the input sequence through its multi-layer Transformer encoder to generate a context-related representation of each word. In order to obtain the semantic embedding of the entire sentence, an average pooling operation is performed on the output vectors of all words except for special tags. For example, for the input text fin-llama-v2 preparing for next week's financial report analysis, the model will output a fixed-dimensional floating-point number vector, which is the semantic embedding of the business intent text.

[0028] In step S4, the GPU indicator timing features, the request indicator timing features, the GPU indicator-request indicator cross-domain correlation features, and the business intent text semantic embedding are fused to obtain the vectorized system state. It should be understood that the background technology clearly points out that traditional scheduling methods struggle to cope with the dynamic, diverse, and highly unpredictable nature of AIGC model requests, and cannot effectively balance the conflict between resource utilization and user experience. A single type of feature, whether it's the GPU's real-time load, the model's historical request volume, or the business's future intent, cannot independently depict a complete picture of the distributed computing environment. For example, the GPU's free memory alone cannot determine whether it is suitable for loading a model, as the model may not have any requests in a short period of time; nor can historical request volume alone predict sudden business demand. Therefore, by fusing these features from different dimensions, granularities, and even different modalities (numerical and textual), a vectorized system state can be constructed that comprehensively reflects the current computing resource status, model demand trends, and future business planning, providing data support for subsequent decision-making.

[0029] Specifically, in one embodiment, step S4, fusing the GPU indicator timing characteristics, the request indicator timing characteristics, the GPU indicator-request indicator cross-domain correlation characteristics and the business intention text semantic embedding to obtain the vectorized system state, includes: S41, fusing the GPU indicator timing characteristics, the request indicator timing characteristics, the GPU indicator-request indicator cross-domain correlation characteristics to obtain the real-time state timing characteristics; S42, fusing the real-time state timing characteristics and the business intention text semantic embedding to obtain the vectorized system state.

[0030] Specifically, step S4 is performed using the following method: First, in step S41. This fusion process aims to integrate the current and historical computing resource status with the model request pattern. In specific implementation, a feature splicing method can be adopted. After normalization in step S22, the GPU indicator timing characteristics, request indicator timing characteristics and GPU indicator-request indicator cross-domain correlation characteristics are obtained, and these characteristics are all represented in the form of vectors. In order to obtain the real-time state timing characteristics, these feature vectors are first spliced. Since GPU indicators and request indicators may involve multiple GPU nodes and multiple models, it is necessary to flatten or aggregate the features of all related nodes and models. A common practice is to splice the GPU indicator timing characteristics of all GPU nodes to form a GPU state vector; splice the request indicator timing characteristics of all models to form a model request vector; and splice the cross-domain correlation characteristics of all models and GPUs to form a correlation vector. Finally, these three large vectors are spliced ​​to obtain a unified real-time state timing feature vector as the real-time state timing feature.

[0031] Next, in step S42, feature splicing can also be used to embed the real-time state time series features and the business intent text semantics into the two vectors for splicing to form the final vectorized system state.

[0032] In step S5, the vectorized system state is input into the generative AI scheduling agent to obtain the model demand prediction JSON. Accordingly, through the aforementioned steps, the heterogeneous GPU indicators, request indicators and business intent texts have been integrated into a unified vectorized system state. This vector contains comprehensive information about the current computing environment. However, to convert this information into an actionable forecast of future model demand, an intelligent entity that can understand complex patterns, perform nonlinear mapping and generate structured outputs is required. The generative AI scheduling agent, with its powerful pattern recognition and generation capabilities, can learn and infer the probability distribution and specific model list of future model requirements from high-dimensional system states, thereby transforming passive response into active prediction. This predictive capability enables the scheduling solution to pre-load models or adjust resources before the request actually occurs.

[0033] Specifically, step S5 is performed using the following method: the generative AI scheduling agent adopts an encoder-decoder model based on the Transformer architecture. Its architecture includes an encoder and a decoder. The encoder is responsible for converting the input vectorized system state into a compact, context-rich intermediate representation. Since the vectorized system state is a high-dimensional numerical vector, for example, with a dimension of 1024, the encoder first maps it into a series of fixed-dimensional embedding vector sequences through a linear projection layer, for example, mapping a 1024-dimensional vector into 16 64-dimensional embedding vectors. This sequence is then input into a multi-layer Transformer encoder block, each of which contains a multi-head self-attention mechanism and a feedforward network to capture the complex relationship between different features in the input sequence and generate a global encoder output representation, namely the system state regulation encoding feature vector.

[0034] The decoder receives the system state adjustment encoded feature vector output by the encoder as context information and generates a text sequence of the model demand forecast JSON in an autoregressive manner. The decoder is also composed of multiple layers of Transformer decoder blocks. Each decoder block contains a masked multi-head self-attention mechanism to ensure that only the generated historical tokens are focused when generating the current token; the cross-attention mechanism is used to fuse the context information output by the encoder with the feedforward network. The decoder predicts the next token at each time step until the end symbol is generated. The training process of this generative AI scheduling agent adopts supervised learning. During training, the weights and bias parameters within the model are iteratively updated through the backpropagation algorithm and optimizer. The initial values ​​of these parameters are randomly set and gradually learned through training data, enabling it to accurately map the system state to future model demand forecasts.

[0035] During the inference phase, upon receiving the current vectorized system state input, the encoder first processes it to generate a compact contextual representation, namely the system state conditioning encoding feature vector. This representation condenses all key information about current computing resources, model request trends, and business intent. The decoder then conditions this contextual representation and gradually generates JSON text starting with a special start symbol, such as the [SOS] marker indicating the start of a sequence. At each generation step, the decoder predicts the next most likely token based on the currently generated sequence and the contextual information provided by the encoder. This process continues until an end symbol, such as the [EOS] marker indicating the end of the sequence, is generated or the preset maximum output length, for example, set to 256 tokens, is reached. Finally, the generated token sequence is concatenated and parsed into a structured model demand prediction JSON format. For example, the output might be {"predicted_models": [{"name": "fin-llama-v2", "probability": 0.95, "horizon_minutes": 15}, {"name": "Stable-Diffusion-XL", "probability": 0.7, "horizon_minutes": 15}]}, indicating that the probability of demand for the fin-llama-v2 and Stable-Diffusion-XL models is high in the next 15 minutes.

[0036] It is understandable that in the above example, when fusing the real-time state time series features and the business intent text semantic embedding, a structured JSON object is constructed by merging the processed structured features and the embedded semantic vectorized representation of the unstructured text. This is then input into the generative AI scheduling agent to obtain the quantitative model demand forecast JSON. Therefore, within the generative AI scheduling agent encoding and decoding framework, the structured JSON object serves as a structured attribute dimension, while the quantitative JSON object serves more as a numerical dimension, which requires optimizing the adaptive dimension transfer between the two.

[0037] Based on this, in a preferred embodiment, step S5, inputting the vectorized system state into the generative AI scheduling agent to obtain the model demand forecast JSON, includes: first, inputting the vectorized system state into the encoder of the generative AI scheduling agent to obtain the system state encoding feature vector. It should be understood that the encoder, as the first part of the generative AI scheduling agent, is used to convert these heterogeneous information into a more compact and semantically rich low-dimensional representation, namely, the system state encoding feature vector. This encoding feature vector can capture the essential characteristics of the current and future states of the system, and provide high-quality input for subsequent prediction and generation tasks. In particular, the specific architecture and processing process of this encoder are the same as those of the encoder in the above embodiment, and will not be described in detail here.

[0038] Next, each eigenvalue in the system state encoding feature vector is probabilistically adaptively adjusted to obtain a system state adjustment encoding feature vector, namely: ;in, is the probability adaptability adjustment coefficient, which is used to represent the adaptability strength of the input structured attribute encoding feature in the probability domain. is the probability deterministic evaluation coefficient, which can represent the certainty of the system state regulation encoding feature vector in the probability domain. This can be determined as a system weight parameter, for example, simply taking ,and is equal to the mean of the system state encoding eigenvector, is each eigenvalue in the system state encoding feature vector. Accordingly, the system state encoding feature vector is a continuous numerical value that may not directly reflect probability or confidence. Therefore, probabilistic adaptive adjustment of the eigenvalues ​​aims to map these values ​​to a range more suitable for expressing probability or confidence, giving them more probabilistic semantics. This allows each component in the system state encoding feature vector to better represent the likelihood of an event occurring or the certainty of a certain state, providing more intuitive and effective input for subsequent probabilistic predictions. Through this nonlinear transformation, the eigenvalues ​​are compressed to a range between 0 and 1, making them probabilistically interpretable.

[0039] Then, the information entropy of each eigenvalue in the system state regulation coding feature vector is compared with the inverse of the overall information entropy of the system state regulation coding feature vector to obtain an optimized system state regulation coding feature vector; wherein, in response to the information entropy of the eigenvalue being greater than the inverse of the overall information entropy of the system state regulation coding feature vector, the eigenvalue remains unchanged; in response to the information entropy of the eigenvalue being less than the inverse of the overall information entropy of the system state regulation coding feature vector, the eigenvalue is subjected to cross-dimensional transfer calibration to obtain a calibrated eigenvalue, that is: ;in, is the logarithmic function value with base 2, is the number of eigenvalues ​​of the system state regulation encoding eigenvector, It is to optimize each eigenvalue in the system state adjustment encoding feature vector. Accordingly, in the AIGC distributed computing power scheduling scenario, some system state features may be more accurate in predicting model requirements than other features, or some features may contain more noise at specific moments. For example, when the business intent text clearly indicates to prepare the financial report analysis model fin-llama-v2 for next week, the feature information entropy related to fin-llama-v2 may be higher, while the feature information entropy related to irrelevant models or inactive GPUs may be lower. This step aims to determine whether the structured attribute encoding features have sufficient numerical quantifiable reliability in the probability domain mapping to achieve effective cross-dimensional transfer from structured dimensions to numerical dimensions. It is to ensure that the feature vector of the input decoder is of high quality and high reliability, and can accurately reflect the real information of the system state, thereby improving the generation effect of the generative AI scheduling agent from structured JSON objects to quantitative JSON objects, that is, improving the accuracy and reliability of model demand prediction. In specific implementation, for each eigenvalue in the system state adjustment encoding feature vector , first calculate its information entropy contribution (or information amount), expressed as Then, the inverse of the overall information entropy of the entire system state regulation encoding feature vector is calculated, that is, The inverse of this overall information entropy can be considered a threshold for measuring the overall uncertainty or average information content of the vector. A comparison is then performed: if the information entropy contribution of the eigenvalue is greater than the inverse of the overall information entropy of the system state adjustment encoding eigenvector, it indicates that the eigenvalue contains above-average information and has high reliability. Therefore, the eigenvalue remains unchanged and is directly used as the optimized eigenvalue. If the information entropy contribution of the eigenvalue is less than the inverse of the overall information entropy of the system state adjustment encoding eigenvector, it indicates that the eigenvalue contains below-average information and may be noisy or unreliable. At this point, the eigenvalue is calibrated across dimensions to obtain a calibrated eigenvalue. This calibration is achieved using the Euclidean distance of the reliability reference, adjusting the eigenvalue with lower information content to a more reliable, overall average information content level to ensure that the regression process has sufficient numerical attribute information for accurate regression. For example, if the percentage of free video memory on a GPU has fluctuated significantly over the past hour, resulting in a low information entropy contribution, the eigenvalue is calibrated so that it does not introduce excessive uncertainty when predicting model requirements. In particular, the information entropy value is a continuous quantity calculated based on probability and is usually represented as a floating-point number. Due to the precision limitations of floating-point numbers in computer storage and calculation, as well as their inherent continuity, the probability that two independent floating-point numbers will be exactly equal in calculation results is theoretically almost zero.

[0040] Finally, the optimized system state adjustment encoding feature vector is input into the decoder of the generative AI scheduling agent to obtain the model demand forecast JSON. That is, the decoder receives the optimized system state adjustment encoding feature vector that has been refined and optimized, and uses it as context information for generating model demand forecasts. The decoder uses an autoregressive method to gradually generate a text sequence in JSON format, thereby directly guiding the subsequent planner module to perform efficient caching action plans, and ultimately achieving effective mitigation of the AIGC model cold start problem and optimized utilization of computing resources. In particular, the specific architecture and implementation process of the decoder are the same as those of the decoder in the above embodiment.

[0041] In step S6, the model demand prediction JSON and the current cache status are input into the planner module to obtain a cache action plan, and the current cache status is used to indicate which GPU node has loaded which model. It is worth mentioning that although the generative AI scheduling agent in the previous step can accurately predict future model requirements, this prediction itself is not a directly executable instruction. The problems mentioned in the background technology such as high latency in cold start, low resource utilization, and unpredictable demand require an intelligent decision-making layer that can convert abstract predictions into specific resource operations. To this end, the present application introduces a planner module, which can identify the status gap between current resource configuration and future demand by combining the prediction of future demand (model demand prediction JSON) and the understanding of the actual occupancy of current resources (current cache status), thereby making room for more urgent or higher priority models, while significantly reducing the delay perceived by users, thereby effectively solving the challenges described in the background technology.

[0042] Specifically, in one embodiment, Figure 4 FIG is a flow chart of step S6 in the distributed computing power scheduling method based on AIGC according to an embodiment of the present application. Figure 4 As shown, step S6, inputting the model demand prediction JSON and the current cache status into the planner module to obtain a cache action plan, including: S61, the planner module traverses each model in the model demand prediction JSON and identifies the status gap of each model to obtain a list of models to be loaded and a list of loaded models; S62, the planner module searches for available nodes for each model to be loaded in the list of models to be loaded and adds them to the cache action plan.

[0043] Specifically, step S6 is performed using the following method: First, in step S61, it should be understood that state gap identification involves comparing the ideal future state depicted by the model demand forecast JSON with the actual resource configuration reflected by the current cache state, thereby clarifying which models need to be loaded (models to be loaded) and which loaded models may no longer be the highest priority (loaded models). The model demand forecast JSON is structured data that contains the models that may be requested within a certain time window in the future (for example, the next 15 minutes) and their predicted demand probability. The current cache state is a mapping table that clearly represents the models currently loaded on each GPU node. This mapping table of loaded models on each GPU node is updated and maintained in real time by the scheduler executor after executing the cache action plan. For example: {"GPU001": "Llama-2-70B", "GPU002": "Stable-Diffusion-XL", "GPU003": "Idle", "GPU004": "ImageGen-v3"}. For each model predicted in the JSON, the planner module queries the current cache state to determine whether the model is already loaded on a GPU node. If a model is listed in the model demand forecast JSON but is not loaded in the current cache state, this constitutes a state gap that needs to be filled, and it is added to the list of models to be loaded. For example, if fin-llama-v2 is in the model demand forecast JSON but not in the current cache state, it is added to the list of models to be loaded, along with its predicted probability. At the same time, the planner module builds a list of loaded models based on the current cache state, which contains the name of each loaded model, the GPU node it is on, and its corresponding need probability. If the model is also in the forecast JSON, the predicted probability is used; if not, a default low probability can be set or a baseline probability can be assigned based on historical usage frequency, such as 0.1, indicating that its current demand is unclear or low. For example, the list of models to be loaded might be [{"name": "fin-llama-v2", "probability": 0.95}], while the list of loaded models might be [{"name": "Llama-2-70B", "gpu_id": "GPU001", "need_probability": 0.3}, {"name":"Stable-Diffusion-XL", "gpu_id": "GPU002", "need_probability": 0.7}, {"name":"ImageGen-v3", "gpu_id": "GPU004", "need_probability": 0.1}].

[0044] Secondly, in step S62, the planner module searches for available nodes for each model to be loaded in the list of models to be loaded and adds them to the cache action plan. Specifically, in one embodiment, step S62, the planner module searches for available nodes for each model to be loaded in the list of models to be loaded and adds them to the cache action plan, including: S621, the planner module searches for the best idle node for each model to be loaded in the list of models to be loaded; S622, if the best idle node exists, the best idle node is regarded as the available node; S623, if the best idle node does not exist, the planner module selects a victim from the list of loaded models and uses the node released by the victim as the available node.

[0045] Specifically, in S621. The planner module will traverse all GPU nodes marked as Idle in the current cache state and evaluate whether they meet the video memory requirements of the model to be loaded. For example, if the fin-llama-v2 model requires 140GB of video memory, the planner module will look for a GPU that is idle and has a video memory capacity greater than or equal to 140GB. Among multiple idle nodes that meet the conditions, the definition of the best can be based on a preset strategy, such as selecting the node with the lowest current load, the node with the lowest temperature, or simply the first node that meets the conditions. For example, if GPU003 is the only idle node that is currently idle and has 160GB of video memory, it will be identified as the best idle node.

[0046] In S622, if the best idle node exists, the planner module regards the best idle node as the available node. At this time, the planner module generates a load instruction and adds it to the cache action plan. For example, if the best idle node of fin-llama-v2 is GPU003, an item will be added to the cache action plan: {"action": "load", "model": "fin-llama-v2", "target_gpu": "GPU003"}. At the same time, the planner module updates its internally maintained copy of the current cache state, updating the state of GPU003 from Idle to fin-llama-v2 to reflect the upcoming changes.

[0047] In S623, if the best idle node does not exist, that is, all GPU nodes have loaded models and there is not enough free video memory, the planner module needs to select a victim model from the loaded model list to release the node it occupies as an available node. Specifically, in one embodiment, the victim is the model with the lowest probability of need in the loaded model list. The planner module will traverse the loaded model list, compare the need probability of each model therein, and select the model with the lowest probability as the victim. For example, in the above-mentioned loaded model list, the need probability of ImageGen-v3 is 0.1, which is the lowest. The planner module will first generate an unload instruction and add it to the cache action plan, for example: {"action": "unload", "model": "ImageGen-v3", "source_gpu": "GPU004"}. The planner module then marks the GPU004 node occupied by ImageGen-v3 as available and generates a load instruction, assigning the model to be loaded, such as fin-llama-v2, to that node and adding it to the cache action plan: {"action": "load", "model": "fin-llama-v2", "target_gpu": "GPU004"}. Similarly, the planner module updates its internal copy of the current cache state, updating the state of GPU004 from ImageGen-v3 to fin-llama-v2. Finally, after processing all models in the list of models to be loaded, the planner module outputs a complete cache action plan, which contains a series of ordered load and unload operations.

[0048] In step S7, the scheduler executes the cache action plan to obtain an updated cache state. That is, although the aforementioned steps achieve accurate predictions of future model demand through the generative AI scheduling agent and the planner module generates an optimized cache action plan, these remain at the logical level. To truly address the issues mentioned in the background technology, such as high cold start latency, low resource utilization, and demand unpredictability of the AIGC model, it is necessary to change the model loading state of the GPU node through physical operations. As a bridge connecting logical decisions and physical resources, the scheduler executes the abstract instructions generated by the planner, such as loading a model to a specific GPU or unloading a model, into specific underlying operations, such as transferring model files and allocating and releasing video memory. By performing these operations, the scheduler executes the GPU resources to dynamically adjust according to the predictions and plans, thereby keeping the required model in a hot-loaded state before the inference request actually arrives, significantly reducing cold start latency and optimizing the overall utilization efficiency of distributed computing resources.

[0049] Specifically, step S7 is performed as follows: the scheduler executor receives a cache action plan from the planner module. The plan is an ordered list of instructions, including two types of operations: load and unload. Each instruction explicitly specifies the target model and the GPU node involved. For example, a cache action plan may contain: [{"action": "unload", "model": "ImageGen-v3", "source_gpu": "GPU004"}, {"action": "load","model": "fin-llama-v2", "target_gpu": "GPU004"}]. The scheduler executor executes these instructions one by one in the order specified in the plan.

[0050] For each planned unloading action, the scheduling executor sends an unloading instruction to the designated GPU node, such as GPU004. This instruction triggers the model management component on the GPU node to completely remove the currently loaded ImageGen-v3 model from its video memory and release the corresponding video memory resources. The scheduling executor waits for a confirmation signal that the unloading operation is complete. For example, after the unloading instruction is sent, the scheduling executor sets a timeout period, for example, preset to 30 seconds. If no confirmation is received within this time, the error handling mechanism is triggered and the unloading failure is recorded. Once the confirmation is received, it means that ImageGen-v3 has been successfully unloaded from GPU004 and the video memory resources of the GPU are released.

[0051] Next, for each scheduled load action, the scheduler executes a series of operations. First, it retrieves the target model, such as the fin-llama-v2 binary file and associated configuration, from a distributed model repository, such as an object storage service or distributed file system. This process may involve network transmission, so the scheduler ensures the integrity and efficiency of the data transfer. After retrieving the model file, the scheduler sends a load command to the designated GPU node, such as GPU004, and transfers the model file. This command instructs the model management component on the GPU node to load the fin-llama-v2 model into its memory and perform the necessary initialization operations to prepare it for inference. The scheduler also waits for confirmation that the load operation is complete and sets a timeout, for example, 120 seconds, to prevent large models from taking too long to load. Once confirmation is received, fin-llama-v2 has been successfully loaded onto GPU004 and is ready for use.

[0052] After each load or unload operation, the scheduler updates its internal cache state copy in real time. For example, after ImageGen-v3 is unloaded from GPU004, GPU004's state is updated to Idle; after fin-llama-v2 is loaded onto GPU004, GPU004's state is updated to fin-llama-v2. This internally updated cache state copy is the final updated cache state, accurately reflecting the actual model loaded on each GPU node after all scheduled operations have completed.

[0053] In step S8, after receiving a newly arrived inference request, the requesting router queries the updated cache state to generate a dispatch decision. This ensures that when an inference request actually arrives, it is immediately routed to the GPU node that has already loaded the target model, completely avoiding the delays caused by cold starts. By querying and utilizing the updated cache state of the scheduler executor in real time, the requesting router can make immediate and optimal dispatch decisions, significantly improving the user experience, meeting service level agreements, and maximizing the utilization efficiency of distributed computing resources, transforming forward-looking scheduling into actual performance advantages.

[0054] Specifically, step S8 is performed as follows: the request router, acting as the front-end entry point for the distributed computing service, continuously listens for and receives new inference requests from users or upstream services. Each incoming inference request contains explicit request information, the most critical of which is the name of the target AIGC model. For example, a request might specify that inference be performed on the fin-llama-v2 model.

[0055] When the request router receives such a newly arrived inference request, it immediately executes the query operation. The query target is the latest updated cache state provided by the scheduling executor. The updated cache state is a real-time mapping table that accurately records the model information loaded on each GPU node in the current distributed computing cluster. For example, the state may be represented as: {"GPU001": "fin-llama-v2", "GPU002": "Stable-Diffusion-XL", "GPU003": "Idle", "GPU004": "ImageGen-v3"}.

[0056] The request router parses the target model name (e.g., fin-llama-v2) from the received inference request and then searches the updated cache state based on the model name.

[0057] If the query results indicate that the target model, such as fin-llama-v2, is already loaded on one or more GPU nodes, such as GPU001, the request router generates a dispatch decision. If the target model is loaded on multiple available nodes, for example, for high availability or load balancing, the same model may be loaded on multiple GPUs. The request router selects the optimal GPU node based on a pre-defined load balancing strategy. Common load balancing strategies include round-robin, which dispatches requests to different available nodes in turn; least connections, which selects the node with the fewest connections currently processing requests; or real-time GPU load-based strategies, such as querying the GPU metrics stream to obtain current compute utilization or memory idleness, which selects the node with the lowest load. For example, if fin-llama-v2 is loaded on both GPU001 and GPU005, and GPU001 is currently less loaded, the request router will decide to dispatch the inference request to GPU001. The output of this dispatch decision is a specific network routing instruction that directs the inference request packet to the inference service interface on the selected GPU node. Since the model has been pre-loaded, this process does not need to wait for the model to load, thus achieving extremely low inference latency and effectively avoiding cold start.

[0058] If the query result shows that the target model is not loaded on any GPU node in the updated cache state, this means that the prediction of the generative AI scheduling agent failed to cover this specific request, or an unexpected situation occurred during the planning and execution process. In this case, the request router will make a distribution decision based on the preset fault tolerance or fallback mechanism. For example, it can temporarily put the request into a waiting queue and trigger an urgent loading process; or route it to a default GPU pool that may require a cold start; or return an error message that the model is not ready.

[0059] In summary, the distributed computing power scheduling method based on AIGC based on the embodiment of the present application is explained, which realizes accurate prediction of future model requirements and proactive resource management by deeply integrating multi-source heterogeneous data. Specifically, the system obtains GPU indicator streams and request indicator streams in real time, and performs time series alignment and feature engineering on them, while extracting semantic features from business intent texts. These multi-dimensional information are fused into vectorized system states and input into the generative AI scheduling agent. The agent uses its generation capability to predict future model requirements and outputs a model requirement prediction JSON. Subsequently, the planner module generates an optimized cache action plan based on this prediction and the current cache status, guiding the scheduling executor to pre-load or intelligently replace the model. Finally, when the request router receives a new request, it can query the updated cache status and directly distribute the request to the GPU node that has loaded the target model, thereby significantly reducing cold start delays, improving computing power resource utilization efficiency and user experience, and effectively overcoming the limitations of traditional scheduling schemes in dynamic AIGC environments.

[0060] Figure 5 FIG is a block diagram of a distributed computing power scheduling system based on AIGC according to an embodiment of the present application. Figure 5 As shown, according to the embodiment of the present application, the distributed computing power scheduling system 100 based on AIGC includes: a data acquisition module 110 for acquiring a GPU indicator flow, a request indicator flow and a business intention text; a data feature processing module 120 for performing time series alignment and feature engineering on the GPU indicator flow and the request indicator flow to obtain GPU indicator timing features, request indicator timing features and GPU indicator-request indicator cross-domain correlation features; a business intention text encoding module 130 for performing semantic feature extraction on the business intention text to obtain business intention text semantic embedding; a system state fusion module 140 for fusing the GPU indicator timing features, the request indicator timing features, the GPU indicator-request indicator cross-domain correlation features, the ... fusing the GPU indicator timing features, the request indicator timing features, the GPU indicator-request indicator cross-domain correlation features, the business intention text encoding module 130 for fusing the GPU indicator timing features, the request indicator timing features, the GPU indicator-request indicator cross-domain correlation features, the business intention text encoding module 130 for fusing the GPU indicator timing features, the request indicator timing features, the GPU indicator-request indicator cross-domain correlation features, the business intention text encoding module 130 for fusing the GPU indicator timing features, the request indicator timing features, Domain-related features and the business intent text semantics are embedded to obtain a vectorized system state; a model demand prediction module 150 is used to input the vectorized system state into a generative AI scheduling agent to obtain a model demand prediction JSON; a data cache module 160 is used to input the model demand prediction JSON and the current cache state into a planner module to obtain a cache action plan, wherein the current cache state is used to indicate which GPU node has currently loaded which model; a cache state update module 170 is used to schedule an executor to execute the cache action plan to obtain an updated cache state; a distribution module 180 is used to request a router to query the updated cache state to generate a distribution decision after receiving a newly arrived inference request.

[0061] Here, those skilled in the art will understand that the specific operations of each step in the above-mentioned distributed computing power scheduling system based on AIGC have been referred to above. Figures 1 to 4 It has been introduced in detail in the description of the AIGC-based distributed computing power scheduling method, and therefore, its repeated description will be omitted.

Claims

1. A distributed computing power scheduling method based on AIGC, characterized in that: include: Obtain GPU indicator stream, request indicator stream, and business intent text; Performing time series alignment and feature engineering on the GPU indicator stream and the request indicator stream to obtain GPU indicator timing features, request indicator timing features, and GPU indicator-request indicator cross-domain correlation features; Performing semantic feature extraction on the business intent text to obtain semantic embedding of the business intent text; Fusing the GPU indicator timing feature, the request indicator timing feature, the GPU indicator-request indicator cross-domain correlation feature, and the business intent text semantic embedding to obtain a vectorized system state; Input the vectorized system state into a generative AI scheduling agent to obtain a model demand forecast JSON; Input the model demand prediction JSON and the current cache status into the planner module to obtain a cache action plan, wherein the current cache status is used to indicate which GPU node has loaded which model; The scheduling executor executes the cache action plan to obtain an updated cache state; After the requesting router receives the newly arrived inference request, the requesting router queries the updated cache state to generate a distribution decision; The vectorized system state is input into the generative AI scheduling agent to obtain the model demand forecast JSON, including: Inputting the vectorized system state into an encoder of a generative AI scheduling agent to obtain a system state encoding feature vector; Probabilistically adaptively adjusting each eigenvalue in the system state encoding feature vector to obtain a system state adjustment encoding feature vector; Comparing the information entropy of each eigenvalue in the system state regulation coding feature vector with the inverse of the overall information entropy of the system state regulation coding feature vector to obtain an optimized system state regulation coding feature vector; wherein, in response to the information entropy of the eigenvalue being greater than the inverse of the overall information entropy of the system state regulation coding feature vector, the eigenvalue remains unchanged; in response to the information entropy of the eigenvalue being less than the inverse of the overall information entropy of the system state regulation coding feature vector, performing cross-dimensional transfer calibration on the eigenvalue to obtain a calibrated eigenvalue; Inputting the optimized system state adjustment encoded feature vector into a decoder of a generative AI scheduling agent to obtain the model demand forecast JSON; The model demand prediction JSON and the current cache status are input into the planner module to obtain a cache action plan, including: The planner module traverses each model in the model demand forecast JSON and identifies the status gap of each model to obtain a list of models to be loaded and a list of loaded models; The planner module searches for an available node for each model to be loaded in the list of models to be loaded and adds the node to the cache action plan.

2. The distributed computing power scheduling method based on AIGC according to claim 1 is characterized in that: Performing time series alignment and feature engineering on the GPU indicator stream and the request indicator stream to obtain GPU indicator timing features, request indicator timing features, and GPU indicator-request indicator cross-domain correlation features, including: Performing time window aggregation on the GPU indicator flow and the request indicator flow, and performing statistical analysis on the GPU indicator flow and the request indicator flow within the time window to obtain initial GPU indicator timing characteristics, initial request indicator timing characteristics, and initial GPU indicator-request indicator cross-domain correlation characteristics, wherein the initial GPU indicator timing characteristics include the average idle video memory percentage and the maximum computing power utilization in the past 1 hour and the past 5 minutes, the initial request indicator timing characteristics include the number of requests of each model in the past 1 hour, the p95 latency and cold start incidence rate of each model in the past 1 hour, and the initial GPU indicator-request indicator cross-domain correlation characteristics include the average GPU load when model X is active; The initial GPU indicator timing characteristics, the initial request indicator timing characteristics, and the initial GPU indicator-request indicator cross-domain correlation characteristics are normalized to obtain the GPU indicator timing characteristics, the request indicator timing characteristics, and the GPU indicator-request indicator cross-domain correlation characteristics.

3. The distributed computing power scheduling method based on AIGC according to claim 2 is characterized in that: Performing semantic feature extraction on the business intent text to obtain semantic embedding of the business intent text includes: inputting the business intent text into a pre-trained sentence vector model to obtain the semantic embedding of the business intent text.

4. The distributed computing power scheduling method based on AIGC according to claim 3 is characterized in that: The GPU indicator timing feature, the request indicator timing feature, the GPU indicator-request indicator cross-domain correlation feature, and the business intent text semantic embedding are integrated to obtain the vectorized system state, including: Fusion of the GPU indicator timing characteristics, the request indicator timing characteristics, and the GPU indicator-request indicator cross-domain correlation characteristics to obtain real-time state timing characteristics; The real-time state temporal features and the business intent text semantic embedding are fused to obtain the vectorized system state.

5. The distributed computing power scheduling method based on AIGC according to claim 1 is characterized in that: The planner module searches for an available node for each model to be loaded in the list of models to be loaded and adds the node to the cache action plan, including: The planner module searches for the best idle node for each model to be loaded in the list of models to be loaded; If the best idle node exists, regard the best idle node as the available node; If the best idle node does not exist, the planner module selects a victim from the loaded model list and uses the node released by the victim as the available node.

6. The distributed computing power scheduling method based on AIGC according to claim 5 is characterized in that: The victim is the model with the lowest required probability in the loaded model list.

7. A distributed computing power scheduling system based on AIGC, used to execute the distributed computing power scheduling method based on AIGC according to any one of claims 1 to 6, characterized in that: include: Data acquisition module, used to obtain GPU indicator stream, request indicator stream and business intent text; a data feature processing module, configured to perform time series alignment and feature engineering on the GPU indicator stream and the request indicator stream to obtain GPU indicator timing features, request indicator timing features, and GPU indicator-request indicator cross-domain correlation features; A business intention text encoding module is used to extract semantic features of the business intention text to obtain semantic embedding of the business intention text; A system status fusion module is used to fuse the GPU indicator timing characteristics, the request indicator timing characteristics, the GPU indicator-request indicator cross-domain correlation characteristics and the business intent text semantic embedding to obtain a vectorized system status; A model demand prediction module, configured to input the vectorized system state into a generative AI scheduling agent to obtain a model demand prediction JSON; A data cache module is used to input the model demand prediction JSON and the current cache status into the planner module to obtain a cache action plan, wherein the current cache status is used to indicate which GPU node has loaded which model; A cache status update module, configured to schedule an executor to execute the cache action plan to obtain an updated cache status; The distribution module is configured to request the router to query the updated cache state to generate a distribution decision after the request router receives a newly arrived inference request.

Images