AI model deployment request scheduling method and device, equipment, medium and product

Abstract

The invention discloses an AI model deployment request scheduling method and device, equipment, a medium and a product, and relates to the technical field of communication, and the method comprises the steps: obtaining an AI model deployment request forwarded by a scheduler, the request comprising a target model identifier; according to the target model identifier, querying parameter cache states of all computing nodes in the cluster to the target model; if the first node with the cache target model is queried, informing a scheduler of first node information, so that the scheduler schedules the request to the first node; if the target model does not exist, a second node is selected, a parameter pulling instruction is sent to a node cache agent of the second node, and the node cache agent is indicated to pull the parameters of the target model to the local of the second node from remote storage; and after a cache ready notification returned by the node cache agent is received, notifying the scheduler, so that the scheduler schedules the AI model deployment request to the second node. According to the method, the delay of a parameter loading link in coping with AI task deployment can be reduced.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to an AI model deployment request scheduling method, apparatus, device, medium and product. Background Technology

[0002] As large-scale model technology is adopted in industrial applications, cloud-native platforms have become the standard infrastructure for deploying AI services. In this architecture, models typically run as containers on Kubernetes clusters, while model parameters are stored in remote object storage systems, achieving separation of storage and computation. While this deployment method improves the flexibility of resource management, it also introduces significant performance bottlenecks: whenever an AI service instance is started or scaled up, a massive model parameter file needs to be pulled from the remote endpoint to the local node and loaded into memory. This process is time-consuming and severely limits the service's startup speed and elastic scaling capabilities.

[0003] Currently, optimizations for model loading efficiency are mostly localized improvements. Examples include optimizing model storage formats to support efficient memory mapping, using container image caching to avoid repeatedly downloading the same model, or scheduling tasks based on resource availability. While these methods can reduce latency to some extent, they haven't yet formed an end-to-end collaborative optimization mechanism encompassing global scheduling, intelligent caching, and data provisioning.

[0004] However, existing technical solutions suffer from high latency in the parameter loading stage when deploying AI tasks. Summary of the Invention

[0005] This application provides an AI model deployment request scheduling method, apparatus, device, medium, and product that can reduce the delay in the parameter loading stage when dealing with AI task deployment.

[0006] To achieve the above objectives, this application adopts the following technical solution: Firstly, this application provides an AI model deployment request scheduling method, including: Obtain the AI ​​model deployment request forwarded by the scheduler, wherein the AI ​​model deployment request contains the target model identifier; Based on the target model identifier, query the parameter caching status of all computing nodes in the cluster for the target model; If a query finds at least one first node whose parameter cache status is cached as the target model, then the first node information is notified to the scheduler so that the scheduler can schedule the AI ​​model deployment request to the first node; If no first node with the parameter cache status of caching the target model is found, then the second node is selected, and a parameter retrieval instruction is sent to the node cache agent of the second node. The parameter retrieval instruction is used to instruct the node cache agent to retrieve the parameters of the target model from the remote storage to the local second node. Upon receiving the cache ready notification returned by the node cache agent, the cache ready information is notified to the scheduler so that the scheduler can schedule the AI ​​model deployment request to the second node.

[0007] Optionally, the parameter fetching command is determined in the following way: The parameter retrieval instruction is generated based on the heat prediction of the target model and the resource level of the second node.

[0008] Optionally, generating the parameter retrieval instruction based on the heat prediction of the target model and the resource level of the second node includes: Based on heat prediction, determine whether the target model is a real-time hotspot model; based on resource level, determine whether the available memory of the second node is greater than the memory threshold. If the target model is determined to be a real-time hotspot model and the available memory of the second node is greater than the memory threshold, a parameter retrieval instruction is generated to pull the parameters of the target model to the local memory of the second node. If both conditions are not met simultaneously, a parameter retrieval instruction is generated to retrieve the parameters of the target model to the local disk of the second node.

[0009] Optionally, the heat prediction of the target model is obtained in the following way: Obtain the prediction model trained based on historical deployment logs; By inputting the target model's identifier and the current time context into the prediction model, the future popularity prediction of the target model can be obtained.

[0010] Optionally, selecting the second node includes: Collect resource status data for all nodes in the cluster that have no valid cache. Based on the resource status data, calculate the cache pull adaptation score for each node; Select the node with the highest fit score as the second node.

[0011] Optionally, the resource status data includes CPU / GPU utilization, remaining memory capacity, disk I / O load, peak network bandwidth, and network latency with remote storage.

[0012] Secondly, this application provides an AI model deployment request scheduling device, comprising: The acquisition module is used to acquire AI model deployment requests forwarded by the scheduler, wherein the AI ​​model deployment requests contain a target model identifier; The processing module is used to query the parameter caching status of all computing nodes in the cluster for the target model based on the target model identifier. The scheduling module is configured to: if a first node is found to have at least one parameter cache status indicating that the target model is cached, notify the scheduler of the first node information so that the scheduler can schedule the AI ​​model deployment request to the first node; if no first node is found to have a parameter cache status indicating that the target model is cached, select a second node and send a parameter retrieval instruction to the node cache agent of the second node, the parameter retrieval instruction instructing the node cache agent to retrieve the parameters of the target model from remote storage to the local storage of the second node; and after receiving a cache ready notification returned by the node cache agent, notify the scheduler of the cache ready information so that the scheduler can schedule the AI ​​model deployment request to the second node.

[0013] Thirdly, this application provides a computing device, including a memory and a processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of the first aspects.

[0014] Fourthly, this application provides a computer-readable storage medium for storing a computer program for performing the method as described in any one of the first aspects.

[0015] Fifthly, this application provides a computer program product comprising one or more computer instructions, wherein when the computer instructions are executed by a computer, the computer performs the method as described in any one of the first aspects.

[0016] As can be seen from the above technical solution, this application has at least the following beneficial effects: In this application, on the one hand, by performing a global query on the parameter cache status of all computing nodes in the cluster before deployment, the deployment request is prioritized to the first node that already holds the target model cache, which completely avoids the lengthy process of repeatedly pulling huge parameter files from the remote end in the traditional solution, greatly shortens the AI ​​service startup time, allows the service to respond quickly to business needs, and fully releases the elastic scaling potential of the cloud-native platform. On the one hand, for scenarios without cache nodes, parameter retrieval instructions are dynamically generated based on target model popularity prediction and node resource level. This enables a differentiated caching strategy that prioritizes storing hot model parameters in memory and rationally allocates non-hot model parameters to disk. This ensures high response efficiency for hot services while maximizing the use of node storage resources and avoiding resource waste. On the other hand, by collecting multi-dimensional node resource status data to calculate the adaptation score and select the second node, efficient network transmission and balanced node load are ensured during parameter retrieval, further reducing parameter loading latency.

[0017] Compared to existing local optimization methods, this approach establishes a collaborative link between global scheduling, intelligent caching, and data supply, fundamentally improving the overall efficiency and resource utilization of AI service deployment, and providing reliable support for the large-scale, highly flexible deployment needs in the industrial application of large models.

[0018] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0019] Figure 1 A flowchart illustrating an AI model deployment request scheduling method provided in this application embodiment; Figure 2 A schematic diagram of an AI model deployment request scheduling device provided in an embodiment of this application; Figure 3 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0020] The terms "first," "second," and "third," etc., used in this application specification and accompanying drawings are used to distinguish different objects, not to limit a specific order.

[0021] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0022] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first: Model parameter caching is a technique that pre-downloads model parameter files stored remotely and stores them locally on the compute node (in memory or on disk). When deploying the same model later, the local cache can be reused directly, avoiding repeated fetching from the remote node.

[0023] As large-scale model technology enters industrial applications, the scale of AI model parameters is growing explosively. While the cloud-native platform's storage-compute separation deployment mode brings flexibility in resource management, it also exposes major technical problems: When starting up and scaling up AI services, it is necessary to pull GB or even TB-level model parameter files from remote object storage to local nodes and load them into memory. This process is time-consuming, resulting in excessive parameter loading latency, which severely limits the service startup speed and the elastic scaling capability of the cloud-native platform. At the same time, it causes expensive computing resources to idle for a long time during the loading process, resulting in low resource utilization and hindering the large-scale implementation of large-scale model industrial applications.

[0024] The main reason for this problem lies in the limitations of existing solutions: on the one hand, the parameter retrieval path in the traditional deployment process is lengthy, and the cost of transmitting large-scale parameters across networks is extremely high, becoming the main bottleneck for service startup; on the other hand, current optimization methods are mostly local improvements, focusing only on single-point links such as model storage format optimization, container image caching, or single resource indicator scheduling, failing to form an end-to-end collaborative mechanism from global cache status awareness and intelligent node selection to differentiated parameter storage, and lacking an intelligent adaptation strategy that combines model popularity and node resource status, thus failing to fundamentally solve the problem of parameter loading efficiency across nodes and links.

[0025] In view of this, embodiments of this application provide an AI model deployment request scheduling method, which can be executed by a processing device. The processing device can be a terminal or a server. Terminals include, but are not limited to, smartphones, tablets, laptops, personal digital assistants, or smart wearable devices. The server can be a cloud server, such as a central server in a central cloud computing cluster or an edge server in an edge cloud computing cluster. Of course, the server can also be a server in a local data center. A local data center refers to a data center directly controlled by the user.

[0026] In this method, to address the issues of high latency in loading AI model parameters and lack of global coordination in existing optimization techniques under a cloud-native compute-splitting architecture, this application first performs a global query on the target model cache status of all nodes in the cluster, prioritizing scheduling to nodes with existing caches to avoid repeated remote fetching. For scenarios without cache, the optimal fetching node is intelligently selected by combining model popularity prediction results with real-time node resource levels, and the parameter storage medium (memory / disk) is dynamically decided to achieve accurate matching between model popularity and node resources. At the same time, the scientific nature of node selection is ensured by quantifying the adaptation score. Ultimately, this method breaks through the limitations of single-point optimization, constructs an end-to-end collaborative mechanism, shortens parameter loading time from the source, and improves the deployment efficiency and resource utilization of AI services.

[0027] To make the technical solution of this application clearer and easier to understand, the following description, in conjunction with the accompanying drawings, introduces an AI model deployment request scheduling method provided by an embodiment of this application. Figure 1 As shown, this figure is a flowchart of an AI model deployment request scheduling method provided in an embodiment of this application. The method includes: S201. The cache controller obtains the AI ​​model deployment request forwarded by the scheduler. The AI ​​model deployment request contains the target model identifier.

[0028] The cache controller is the decision-making component in this application. It has functions such as global cache status management, intelligent node selection, and parameter retrieval instruction generation. It is the carrier for realizing global cache awareness and intelligent decision scheduling, and is responsible for coordinating the query, preparation and optimization of model cache within the cluster.

[0029] The scheduler refers to the task scheduling component in a cloud-native platform (such as a Kubernetes cluster). Its responsibility is to receive AI model deployment requests and allocate deployment nodes according to the resource status of cluster nodes (such as CPU / GPU load, memory usage, etc.). In this application, the scheduler needs to work in conjunction with the cache controller to adjust the scheduling logic based on the cache status.

[0030] An AI model deployment request is an instruction initiated by a user or business system to start an AI model service in a cloud-native cluster. This request must explicitly tell the cluster the target model to be deployed so that the cluster can allocate resources and load the corresponding parameters.

[0031] The target model identifier is a feature information used to uniquely identify the AI ​​model to be deployed, such as model name, version number, unique ID, etc. It is the basis for the cache controller to query the cache status of the model in the cluster and pull the corresponding parameters from remote storage, ensuring accurate location of the target model.

[0032] After a user or business system initiates an AI model deployment request, the scheduler of the cloud-native platform (such as a Kubernetes cluster) does not directly decide on the deployment node. Instead, the scheduler forwards the request to the cache controller, which has global cache management and intelligent decision-making functions. This design breaks through the limitation of the traditional scheduler that only focuses on node resources and is unaware of the cache status. Meanwhile, the forwarded request carries a target model identifier, such as model name, version number, and unique ID, to uniquely identify the AI ​​model to be deployed. This ensures that the cache controller can accurately locate the model to be deployed, avoiding cache query errors or parameter retrieval deviations. After receiving the request, the cache controller will immediately start the subsequent process to query the target model cache status of all computing nodes in the cluster. This provides the premise for subsequent branch decisions to prioritize scheduling to existing cache nodes or select nodes to pull cache, thereby initiating an intelligent scheduling link based on cache status.

[0033] S202. The cache controller queries the parameter caching status of all compute nodes in the cluster for the target model based on the target model identifier.

[0034] Parameter cache status refers to the status information of whether the computing node stores the complete parameter file of the target model locally (memory or disk) and whether the parameter file is valid (such as whether the version matches, whether the data is complete, and whether it is within the validity period). It is divided into three categories: cached (valid), not cached, and cached but invalid (such as version expired or data corrupted).

[0035] A cluster refers to a cloud-native deployment cluster composed of multiple computing nodes, such as a Kubernetes cluster. All nodes are interconnected through a network and work together to carry out tasks such as AI model deployment, parameter storage, and inference computation. It is the overall hardware resource pool for running AI services.

[0036] Compute nodes are physical or virtual machines with computing capabilities within a cluster. They serve as the specific carriers for deploying AI models, possessing hardware resources such as local storage (memory, disk), CPU / GPU, and capable of performing operations such as parameter retrieval, cache storage, and model inference.

[0037] After obtaining the AI ​​model deployment request and target model identifier, the cache controller initiates a global cache probing step. Specifically, based on the unique target model identifier, the cache controller queries the local storage status of all compute nodes in the cloud-native cluster, focusing on verifying whether each node stores the parameter file for the target model, and whether the parameter file is complete and valid. For example, it checks whether the parameter file version is consistent with the target model in the deployment request, whether the data is intact, and whether it has exceeded the preset validity period. The purpose of this step is to comprehensively understand the cache distribution of the target model within the cluster and determine whether there are any directly reusable local caches.

[0038] If a node with a valid cache is found, the deployment request can be directly scheduled to that node, thus avoiding the time-consuming process of repeatedly fetching parameters from the remote end. If no valid cache is found on any node, the process of selecting a node and fetching parameters needs to be initiated. This step provides a basis for subsequent branch decisions and is a prerequisite for achieving intelligent scheduling based on cache status.

[0039] S203. If the cache controller finds that there is a first node with at least one parameter cache status as the cache target model, it will notify the scheduler of the first node information so that the scheduler can schedule the AI ​​model deployment request to the first node.

[0040] The first node is the computing node in the cloud-native cluster that has cached the valid parameters of the target model. It is the node that the cache controller selects after querying and can directly reuse the local cache, and it is also the preferred choice for subsequent AI model deployment.

[0041] Node information is data used to identify the first node, including the node's unique identifier, network address, hardware configuration summary, cache media type (memory / disk), etc., so that the scheduler can accurately locate and schedule deployment requests.

[0042] After the cache controller completes the parameter cache status query for all compute nodes in the cluster, if it finds at least one first node in the cluster that has cached valid parameters of the target model, it will immediately synchronize the node information of the first node (such as the node's unique identifier and network address) to the cloud-native platform's scheduler. Upon receiving the first node information, the scheduler will adjust its deployment decisions based on this information. Instead of simply allocating deployment tasks based on the node's CPU, GPU, or other hardware resources, it will directly schedule AI model deployment requests to the first node. In this way, model deployment does not require fetching parameters from remote storage; it can directly reuse the first node's local cache, significantly shortening parameter loading time, fully leveraging the advantages of cache reuse, and improving the startup efficiency of AI services.

[0043] S204. If the cache controller finds that the first node with the parameter cache status of the target model does not exist, it selects the second node and sends a parameter retrieval instruction to the node cache agent of the second node. The parameter retrieval instruction is used to instruct the node cache agent to retrieve the parameters of the target model from the remote storage to the local storage of the second node.

[0044] The second node is selected as follows: The cache controller collects resource status data from all nodes in the cluster that have no valid cache; based on the resource status data, it calculates the cache pull adaptation score for each node; and selects the node with the highest adaptation score as the second node.

[0045] A node without a valid cache refers to a computing node in the cluster that does not store valid cached parameters of the target model, or whose stored cache has problems such as version mismatch, data corruption, or expired validity (determined to be invalid). It is a candidate for selecting the second node.

[0046] Resource status data is a set of metrics used to evaluate the ability of nodes without effective cache to adapt to parameter fetching tasks. This data includes CPU / GPU utilization, remaining memory capacity, disk I / O load, peak network bandwidth, and network latency with remote storage. CPU / GPU utilization refers to the proportion of resources occupied by the central processing unit (CPU) or graphics processing unit (GPU) of a compute node, reflecting the current computing power load of the node. The lower the utilization, the more computing power the node has to handle parameter fetching and model deployment tasks. Remaining memory capacity refers to the amount of unused physical memory or GPU memory on the compute node, directly determining whether the node can provide sufficient memory-level storage space for model parameters. Larger remaining capacity is more suitable for storing model parameters with high loading speed requirements. Disk I / O load refers to the read / write activity of the compute node's disk devices (such as SSDs and NVMe). The lower the load, the more efficient the node is in writing and reading parameters, reducing the time spent on parameter fetching and subsequent loading. Peak network bandwidth refers to the maximum data transfer rate that the network link between the computing node and the remote storage can carry. The higher the peak, the faster the model parameters can be retrieved from the remote storage, which can shorten the time cost of parameter transmission.

[0047] Network latency with remote storage refers to the time interval between a compute node sending a data request to remote storage and receiving a response. The lower the latency, the higher the communication efficiency between the node and the remote storage, and the more time can be saved during parameter retrieval.

[0048] The cache pull adaptation score is a quantitative evaluation value calculated based on node resource status data. The logic is that the higher the node resource idleness and the better the network transmission conditions, the higher the adaptation score. It is used to objectively measure the node's ability to adapt to parameter pull tasks.

[0049] The second node is selected from all nodes without valid cache and has the highest cache pull adaptation score. It is a node specifically used to pull the target model parameters and undertake subsequent AI model deployment tasks.

[0050] When the cache controller queries and finds that no node in the cluster holds a valid cache for the target model, it initiates the selection process for the second node. First, it comprehensively collects resource status data for all nodes without valid caches in the cluster. This data includes indicators such as CPU / GPU utilization, remaining memory capacity, disk I / O load, peak network bandwidth, and network latency with remote storage, fully reflecting the hardware resource idleness and network transmission capabilities of each candidate node. Next, based on preset calculation rules, such as the weighting of resource idleness and network condition scoring standards, it quantifies the resource status data of each node without valid caches to obtain its respective cache fetch adaptation score. The score directly corresponds to the node's ability to handle parameter fetching tasks. Finally, the node with the highest adaptation score is selected from all candidate nodes as the second node.

[0051] The expression for calculating the cache pull fit score is:

[0052] in, This represents the cache fetch fit score for a single node without a valid cache. Indicates the first Weighting coefficients of class indicators Indicates the first Standardized scores of class indicators This indicates the number of resource status indicators that are included in the scoring.

[0053] This selection logic ensures that the second node has sufficient resource margins and efficient network transmission conditions, enabling it to quickly and stably pull target model parameters from remote storage, laying an efficient foundation for subsequent model deployment, while avoiding problems such as excessive parameter retrieval latency or node overload due to improper node selection.

[0054] The parameter fetch command is determined in the following way: The cache controller generates parameter retrieval instructions based on the target model's popularity prediction and the resource level of the second node.

[0055] The heat prediction of the target model is obtained in the following way: Obtain the prediction model trained based on historical deployment logs; input the target model's identifier and the current time context into the prediction model to obtain the target model's future popularity prediction.

[0056] Popularity prediction refers to a quantitative assessment of the deployment frequency and access demand of a target AI model in the future. The assessment results are used to distinguish whether a model is a real-time hot model and serve as the basis for subsequent parameter storage media decisions.

[0057] Historical deployment logs are a complete record of all past AI model deployment tasks in the cloud-native cluster, including information such as model identifier, deployment time, deployment frequency, scaling up and down times, and peak access times. They are the basic data source for training and predicting models.

[0058] The predictive model is an intelligent model trained based on historical deployment logs through statistical analysis or machine learning algorithms (such as time series models, decision trees, etc.). It has the ability to take model identifiers and time information as input and output heat assessment results.

[0059] The current time context refers to the time-related information when the target model deployment request is initiated, including date, time period, weekday / holiday, peak / off-peak business period, etc., which is an important factor affecting the popularity of the model.

[0060] Specifically, firstly, the cache controller needs to retrieve a pre-trained prediction model. This model is obtained by mining the correlation between model deployment frequency and time through algorithms using a large amount of historical deployment logs accumulated by the cluster, and it has the ability to predict the future popularity of the model. Secondly, the cache controller inputs the unique identifier of the model to be deployed (used to accurately locate the historical deployment characteristics of the model) and the current time context (used to judge the popularity trend by combining time patterns) into the prediction model. The prediction model will then output the popularity assessment result of the target model in the future period based on the built-in pattern algorithm. Through this process, this solution can overcome the limitations of experience-based judgment and achieve objective and quantitative prediction of model popularity, providing a basis for subsequent differentiated parameter storage strategies.

[0061] The resource level here specifically refers to the idle status of the hardware resources of the second node, with a core focus on the available memory capacity, which is the basis for judging whether the node can support memory-level parameter storage.

[0062] After the cache controller selects the second node, it doesn't directly generate a fixed parameter fetch instruction. Instead, it customizes the instruction content through a two-dimensional conditional judgment: the first dimension is the target model's popularity prediction result, used to clarify the model's access requirements and determine whether it needs high-speed memory storage to ensure low-latency loading; the second dimension is the second node's resource level, used to clarify the node's hardware capacity and determine whether it has sufficient available memory to store model parameters. The cache controller combines the results of these two dimensions to ultimately generate a parameter fetch instruction that accurately matches the model's needs and the node's capabilities. This ensures that the parameter storage location meets the model's access efficiency requirements while maximizing the utilization of node resources, avoiding memory waste or disk storage inadequate for hot models.

[0063] The specific steps for determining the parameter fetch command are as follows: Based on popularity prediction, determine whether the target model is a real-time hotspot model; based on resource level, determine whether the available memory of the second node is greater than the memory threshold; if the target model is determined to be a real-time hotspot model and the available memory of the second node is greater than the memory threshold, generate a parameter retrieval instruction to pull the parameters of the target model to the local memory of the second node; if neither condition is met at the same time, generate a parameter retrieval instruction to pull the parameters of the target model to the local disk of the second node.

[0064] Real-time hotspot models are AI models that are frequently accessed and urgently needed in the current and future period, derived from popularity prediction. These models have high requirements for parameter loading speed and low-latency access should be prioritized.

[0065] The memory threshold is a preset quantitative standard for judging whether a node has sufficient memory, such as 50% of the total memory of the node or a fixed capacity of 80GB. It is dynamically adjusted by the system according to the model parameter scale and cluster resource configuration, and is the basis for determining whether parameters can be stored in memory.

[0066] Local memory is the physical memory (RAM) or GPU video memory (VRAM) of the second node. It features fast read and write speeds and is suitable for storing real-time hotspot model parameters to shorten loading time.

[0067] Local disks are high-speed storage media (such as NVMe and SSD) for the second node. They have lower read and write speeds than memory but larger storage capacity. They are used to store non-hotspot model parameters, balancing storage efficiency and resource utilization.

[0068] First, based on the popularity prediction results obtained from the prediction model trained by historical deployment logs, it is determined whether the target model to be deployed belongs to the real-time hot model, that is, the model with high access and high deployment demand in the near future. At the same time, combined with the resource level of the second node, it is focused on checking whether its available memory capacity exceeds the preset memory threshold, so as to determine whether the node memory is sufficient to support the target model parameters.

[0069] If both conditions are met, the target model is determined to be a real-time hot model, and the available memory of the second node is greater than the memory threshold, indicating that the node has the conditions to store parameters in memory, and the hot model needs to be loaded with low latency. At this time, the cache controller will generate a parameter fetch instruction, instructing the node cache agent to directly fetch the parameters of the target model to the local memory (RAM / VRAM) of the second node. If the above two conditions are not met at the same time (including non-real-time hotspot models and real-time hotspot models but the node's available memory is insufficient to meet the threshold), a parameter fetch instruction is generated, instructing the node cache agent to fetch the parameters of the target model to the local disk (NVMe / SSD) of the second node.

[0070] This logic ensures the rapid loading of real-time hotspot models while preventing non-hotspot models from consuming valuable memory resources, thus achieving a balance between resource utilization efficiency and deployment speed.

[0071] S205. After receiving the cache ready notification returned by the node cache agent, the cache controller notifies the scheduler of the cache ready information so that the scheduler can schedule the AI ​​model deployment request to the second node.

[0072] The cache ready notification is a status feedback message sent to the cache controller after the node cache agent completes the parameter retrieval and storage operations. The content includes the second node identifier, parameter storage location (memory / disk), parameter integrity verification result, etc., and is used to inform the cache controller that the parameter caching is complete and can be used for model deployment.

[0073] The cache readiness information is the information that the cache controller forwards to the scheduler after integrating the contents of the cache readiness notification. It includes the unique identifier of the second node and the parameter cache status, ensuring that the scheduler can accurately locate the target node.

[0074] After the cache controller selects the second node and issues a parameter retrieval command, the second node's node cache agent initiates the parameter retrieval process. This process involves retrieving parameters from remote storage, storing them on the specified medium, and verifying parameter integrity. The cache controller then sends a cache-ready notification to the cache controller. Upon receiving this notification, the cache controller confirms that the second node is ready to host model deployment. It then compiles cache-ready information containing the second node's identifier and cache status and forwards it to the scheduler. Based on this information, the scheduler bypasses the usual resource filtering and directly schedules the AI ​​model deployment request to the second node. At this point, the model deployment can directly reuse locally cached parameters without needing to retrieve them from the remote storage. This ultimately optimizes parameter loading efficiency in cache-free scenarios and completes the seamless integration of cache preparation and deployment scheduling.

[0075] Based on the above description, this application has the following beneficial effects: In this application, on the one hand, by performing a global query on the parameter cache status of all computing nodes in the cluster before deployment, the deployment request is prioritized to the first node that already holds the target model cache, which completely avoids the lengthy process of repeatedly pulling huge parameter files from the remote end in the traditional solution, greatly shortens the AI ​​service startup time, allows the service to respond quickly to business needs, and fully releases the elastic scaling potential of the cloud-native platform. On the one hand, for scenarios without cache nodes, parameter retrieval instructions are dynamically generated based on target model popularity prediction and node resource level. This enables a differentiated caching strategy that prioritizes storing hot model parameters in memory and rationally allocates non-hot model parameters to disk. This ensures high response efficiency for hot services while maximizing the use of node storage resources and avoiding resource waste. On the other hand, by collecting multi-dimensional node resource status data to calculate the adaptation score and select the second node, efficient network transmission and balanced node load are ensured during parameter retrieval, further reducing parameter loading latency.

[0076] Compared to existing local optimization methods, this approach establishes a collaborative link between global scheduling, intelligent caching, and data supply, fundamentally improving the overall efficiency and resource utilization of AI service deployment, and providing reliable support for the large-scale, highly flexible deployment needs in the industrial application of large models.

[0077] The above text combined Figure 1 The AI ​​model deployment request scheduling method provided in the embodiments of this application has been described in detail. The apparatus and equipment provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0078] like Figure 2 As shown in the figure, this is a schematic diagram of an AI model deployment request scheduling device provided in an embodiment of this application. The device includes: The acquisition module 301 is used to acquire an AI model deployment request forwarded by the scheduler, wherein the AI ​​model deployment request includes a target model identifier; Processing module 302 is used to query the parameter caching status of all computing nodes in the cluster for the target model based on the target model identifier; The scheduling module 303 is configured to: if a first node is found to have at least one parameter cache status that caches the target model, notify the scheduler of the first node information so that the scheduler can schedule the AI ​​model deployment request to the first node; if no first node is found to have a parameter cache status that caches the target model, select a second node and send a parameter retrieval instruction to the node cache agent of the second node, the parameter retrieval instruction being used to instruct the node cache agent to retrieve the parameters of the target model from remote storage to the local storage of the second node; and after receiving a cache ready notification returned by the node cache agent, notify the scheduler of the cache ready information so that the scheduler can schedule the AI ​​model deployment request to the second node.

[0079] Optionally, the processing module 302 is specifically used to generate the parameter retrieval instruction based on the heat prediction of the target model and the resource level of the second node.

[0080] Optionally, the scheduling module 303 is specifically used to determine whether the target model is a real-time hotspot model based on heat prediction; and to determine whether the available memory of the second node is greater than the memory threshold based on resource level. If the target model is determined to be a real-time hotspot model and the available memory of the second node is greater than the memory threshold, a parameter retrieval instruction is generated to pull the parameters of the target model to the local memory of the second node. If both conditions are not met simultaneously, a parameter retrieval instruction is generated to retrieve the parameters of the target model to the local disk of the second node.

[0081] Optionally, the processing module 302 is specifically used to obtain the prediction model trained based on historical deployment logs; input the identifier of the target model and the current time context into the prediction model to obtain the future popularity prediction of the target model.

[0082] Optionally, the processing module 302 is specifically used to collect resource status data of all nodes in the cluster that do not have valid caches; Based on the resource status data, calculate the cache pull adaptation score for each node; Select the node with the highest fit score as the second node.

[0083] The AI ​​model deployment request scheduling device according to the embodiments of this application can correspondingly execute the method described in the embodiments of this application, and the other operations and / or functions of each module / unit of the AI ​​model deployment request scheduling device are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the illustrated embodiments will not be described in detail here.

[0084] This application also provides a computing device. For example... Figure 3 As shown in the figure, this is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 700 includes a bus 701, a processor 702, a communication interface 703, and a memory 704. The processor 702, the memory 704, and the communication interface 703 communicate with each other via the bus 701.

[0085] The 701 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0086] The processor 702 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0087] The communication interface 703 is used for communication with external devices.

[0088] Memory 704 may include volatile memory, such as random access memory (RAM). Memory 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0089] The memory 704 stores executable code, and the processor 702 executes the executable code to perform the aforementioned AI model deployment request scheduling method.

[0090] Specifically, in achieving Figure 2 In the case of the illustrated embodiment, and Figure 2 When the modules or units of the AI ​​model deployment request scheduling device described in the embodiment are implemented through software, the execution... Figure 2 The software or program code required for the functions of each module / unit can be partially or wholly stored in memory 704. Processor 702 executes the program code corresponding to each unit stored in memory 704 and executes the aforementioned AI model deployment request scheduling method.

[0091] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned AI model deployment request scheduling method.

[0092] This application also provides a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0093] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0094] When the computer program product is executed by a computer, the computer executes any of the aforementioned AI model deployment request scheduling methods. The computer program product can be a software installation package; when any of the aforementioned AI model deployment request scheduling methods needs to be used, the computer program product can be downloaded and executed on the computer.

[0095] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0096] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. A method for scheduling AI model deployment requests, characterized in that, The method includes: Obtain the AI ​​model deployment request forwarded by the scheduler, wherein the AI ​​model deployment request contains the target model identifier; Based on the target model identifier, query the parameter caching status of all computing nodes in the cluster for the target model; If a query finds at least one first node whose parameter cache status is cached as the target model, then the first node information is notified to the scheduler so that the scheduler can schedule the AI ​​model deployment request to the first node; If no first node with the parameter cache status of caching the target model is found, then the second node is selected, and a parameter retrieval instruction is sent to the node cache agent of the second node. The parameter retrieval instruction is used to instruct the node cache agent to retrieve the parameters of the target model from the remote storage to the local second node. Upon receiving the cache ready notification returned by the node cache agent, the cache ready information is notified to the scheduler so that the scheduler can schedule the AI ​​model deployment request to the second node.

2. The method according to claim 1, characterized in that, The parameter retrieval command is determined in the following way: The parameter retrieval instruction is generated based on the heat prediction of the target model and the resource level of the second node.

3. The method according to claim 2, characterized in that, The step of generating the parameter retrieval instruction based on the heat prediction of the target model and the resource level of the second node includes: Based on heat prediction, determine whether the target model is a real-time hotspot model; based on resource level, determine whether the available memory of the second node is greater than the memory threshold. If the target model is determined to be a real-time hotspot model and the available memory of the second node is greater than the memory threshold, a parameter retrieval instruction is generated to pull the parameters of the target model to the local memory of the second node. If both conditions are not met simultaneously, a parameter retrieval instruction is generated to retrieve the parameters of the target model to the local disk of the second node.

4. The method according to claim 2, characterized in that, The heat prediction of the target model is obtained through the following method: Obtain the prediction model trained based on historical deployment logs; By inputting the target model's identifier and the current time context into the prediction model, the future popularity prediction of the target model can be obtained.

5. The method according to claim 1, characterized in that, The selection of the second node includes: Collect resource status data for all nodes in the cluster that have no valid cache. Based on the resource status data, calculate the cache pull adaptation score for each node; Select the node with the highest fit score as the second node.

6. The method according to claim 5, characterized in that, The resource status data includes CPU / GPU utilization, remaining memory capacity, disk I / O load, peak network bandwidth, and network latency with remote storage.

7. An AI model deployment request scheduling device, characterized in that, The device includes: The acquisition module is used to acquire AI model deployment requests forwarded by the scheduler, wherein the AI ​​model deployment requests contain a target model identifier; The processing module is used to query the parameter caching status of all computing nodes in the cluster for the target model based on the target model identifier. The scheduling module is configured to: if a first node is found to have at least one parameter cache status indicating that the target model is cached, notify the scheduler of the first node information so that the scheduler can schedule the AI ​​model deployment request to the first node; if no first node is found to have a parameter cache status indicating that the target model is cached, select a second node and send a parameter retrieval instruction to the node cache agent of the second node, the parameter retrieval instruction instructing the node cache agent to retrieve the parameters of the target model from remote storage to the local storage of the second node; and after receiving a cache ready notification returned by the node cache agent, notify the scheduler of the cache ready information so that the scheduler can schedule the AI ​​model deployment request to the second node.

8. A computing device, characterized in that, Including memory and processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes one or more computer instructions that, when executed by a computer, perform the method as described in any one of claims 1 to 6.

Images