Service scheduling method and device, equipment, storage medium and program product

Abstract

The invention provides a service scheduling method and device, equipment, a storage medium and a program product, and relates to the field of artificial intelligence. The method comprises the steps that when a user request is received, whether an idle activated state instance exists or not is detected; wherein the activated state instance is an instance which is bound with graphics processing unit (GPU) resources and provides a target service; if yes, the user request is routed to the idle activated state instance; if not, selecting a first target instance from the dormant state instances, loading a video memory snapshot of the first target instance to restore the first target instance to an activated state, and routing the user request to the first target instance of which the activated state is restored; wherein the dormant state instance is an instance in which a model state is stored through a video memory snapshot, GPU resources are released and services are not provided temporarily. According to the method provided by the invention, efficient utilization of resources can be realized while the real-time performance of the service is ensured.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a service scheduling method, apparatus, device, storage medium, and program product. Background Technology

[0002] In the field of artificial intelligence, especially in large-scale model inference scenarios, service processing relies on massively parallel computing capabilities. Graphics Processing Units (GPUs), with their multi-core architecture and efficient parallel processing characteristics, have become the core hardware resource supporting these high-computing-power-demand services. As business scale expands and user needs diversify, the requirements for service response speed and resource utilization efficiency are increasing.

[0003] Currently, known service scheduling methods typically employ a fixed number of instance deployments. When faced with sudden traffic surges, insufficient instance resources can easily lead to response delays or even service interruptions. Conversely, during periods of low traffic, instances remaining idle for extended periods result in wasted hardware resources such as GPUs, making dynamic resource adaptation and efficient utilization difficult. Therefore, related technologies have proposed elastic scheduling schemes based on serverless architectures, which create and destroy instances on demand to match traffic changes.

[0004] However, in the above-mentioned elastic scheduling scheme, the binding of GPU resources and model loading take too long during the instance state switching (startup / destruction) process, which cannot achieve efficient resource utilization while ensuring service real-time performance. Summary of the Invention

[0005] This application provides a service scheduling method, apparatus, device, storage medium, and program product for a recommendation model, which can ensure service real-time performance and achieve efficient resource utilization.

[0006] Firstly, this application provides a service scheduling method, the method comprising:

[0007] Upon receiving a user request, the system checks whether there are any idle active instances; wherein, the active instance is an instance that is bound to a graphics processing unit (GPU) resource and provides the target service.

[0008] If it exists, the user request will be routed to an idle active instance;

[0009] If it does not exist, select the first target instance from the dormant instances, load the memory snapshot of the first target instance to restore the first target instance to the active state, and then route the user request to the first target instance that has been restored to the active state; wherein, the dormant instance is an instance that has stored the model state through a memory snapshot, released GPU resources and is not providing services for the time being.

[0010] Secondly, this application provides a service scheduling apparatus, the apparatus comprising:

[0011] The detection module is used to detect whether there is an idle active instance when a user request is received; wherein, the active instance is an instance that is bound to GPU resources and provides the target service;

[0012] The scheduling module is used to route the user request to an idle active instance when an idle active instance exists; when no idle active instance exists, it selects a first target instance from dormant instances, loads a memory snapshot of the first target instance to restore the first target instance to an active state, and then routes the user request to the first target instance that has been restored to an active state; wherein, the dormant instance is an instance that has stored the model state through a memory snapshot, released GPU resources, and is not providing services for the time being.

[0013] Thirdly, this application provides an electronic device, including: at least one processor, and a memory communicatively connected to the processor;

[0014] The memory stores computer-executed instructions;

[0015] The at least one processor executes computer execution instructions stored in the memory to implement the method as described in any of the first aspects.

[0016] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any of the first aspects.

[0017] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects.

[0018] The service scheduling method provided in this application is executed by any electronic device to dynamically schedule target services that depend on GPU resources in scenarios with multiple service instances. Specifically, when an electronic device receives a user request, it checks whether there is an idle active instance. If so, the user request is routed to the idle active instance; otherwise, a first target instance is selected from the dormant instance pool, and its pre-stored GPU memory snapshot is loaded to quickly restore it to an active state. Subsequently, the user request is routed to the restored instance. In this process, since active instances can process requests immediately, and dormant instances achieve rapid state saving and restoration through GPU memory snapshot technology, the lengthy model loading, environment initialization, and warm-up processes are avoided, thereby shortening the instance state switching time and reducing service response latency. At the same time, the design of releasing GPU resources through dormant instances avoids resource idleness, achieving a balance between service real-time performance and efficient resource utilization. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 This is a schematic diagram illustrating an application scenario of a service scheduling method provided in an embodiment of this application.

[0021] Figure 2 A flowchart illustrating a service scheduling method provided in this application embodiment. Figure 1 ;

[0022] Figure 3 A flowchart illustrating a service scheduling method provided in this application embodiment. Figure 2 ;

[0023] Figure 4 This is a schematic diagram of the structure of a service scheduling device provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0025] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0027] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0028] It should be noted that the service scheduling method, apparatus, equipment, storage medium and program product provided in this application can be used in the field of artificial intelligence, or in any field other than artificial intelligence. The application field of the service scheduling method, apparatus, equipment, storage medium and program product in this application is not limited.

[0029] In the field of artificial intelligence, especially in large-scale model inference scenarios, service processing relies on massively parallel computing capabilities. Graphics Processing Units (GPUs), with their multi-core architecture and efficient parallel processing characteristics, have become the core hardware resource supporting these high-computing-power-demand services.

[0030] It should be understood that large models typically have billions or even trillions of parameters, requiring massive matrix operations in a single inference iteration. Relying solely on the CPU is insufficient to meet millisecond-level response requirements; therefore, the parallel computing capabilities of GPUs are essential for efficient processing. As business scale expands and user needs diversify, service request volume exhibits significant dynamic fluctuations, leading to increasingly higher demands for service response speed (e.g., inference latency must be controlled within hundreds of milliseconds) and resource utilization efficiency (e.g., GPU utilization must be kept within a reasonable range).

[0031] Currently, known service scheduling methods typically employ a fixed-number instance deployment model, pre-launching a fixed number of service instances bound to GPU resources within the cluster. This model is prone to causing request queuing or even service interruption when facing sudden traffic surges due to insufficient instance resources; conversely, during periods of low traffic, instances remain idle for extended periods, resulting in wasted GPU and other hardware resources (GPU utilization is less than 20% in some scenarios), making it difficult to achieve dynamic resource adaptation and efficient utilization. Therefore, related technologies have proposed elastic scheduling schemes based on serverless architectures. These schemes create and destroy instances on demand to match traffic changes, automatically scaling up new instances when request volume surges and destroying redundant instances when traffic decreases, thereby achieving dynamic resource allocation.

[0032] However, the instance state switching (startup / destruction) process in the above elastic scheduling scheme has a significant bottleneck. When a new instance starts, it needs to complete a series of operations, including container environment initialization, model weight loading (transferring and parsing model files ranging from several gigabytes to tens of gigabytes), and GPU resource binding. This entire process can take tens of seconds or even minutes. After an instance is destroyed, the model state information is completely lost, and the loading process must be repeated upon restarting. This "full reconstruction" state switching method leads to frequent response delays during traffic fluctuations, making it impossible to achieve efficient resource utilization while ensuring service real-time performance. This problem is particularly pronounced in scenarios with stringent real-time requirements, such as large model inference.

[0033] Therefore, this application provides a service scheduling method, apparatus, device, storage medium, and program product, aiming to solve the above-mentioned technical problems of the known art. Specifically, the service scheduling method of this application proposes a dual-state management mechanism of active and dormant instances. When an idle active instance exists, requests are directly routed; when no idle instance exists, a dormant instance is quickly restored to the active state by loading a video memory snapshot to handle requests, thereby achieving a balance between service response speed and resource utilization efficiency.

[0034] It should be understood that the service scheduling method of this application can be used in any AI service scenario that requires dynamic management of GPU resources to balance service latency and resource costs. For example, Figure 1 This is a schematic diagram illustrating an application scenario of a service scheduling method provided in an embodiment of this application. For example... Figure 1 As shown, the method of this application can be used in large model inference service cluster scenarios.

[0035] Specifically, in this scenario, the electronic device acts as the cluster's management node, continuously monitoring the load status of all service instances. When a new user inference request is received from a user terminal, the management node first attempts to route it to an idle, active instance. If all active instances are not idle, the management node immediately selects a dormant first target instance from the dormant instance pool, loads its memory snapshot, and binds GPU resources to restore it to an active state in a very short time. The request is then routed to this newly restored instance for processing.

[0036] In the above process, since active instances can directly respond to requests and dormant instances can be restored through memory snapshots without reloading the complete model, the instance state switching time is greatly shortened. At the same time, dormant instances release GPU resources, reducing idle waste.

[0037] In this embodiment, the aforementioned electronic device can also be a central scheduler in a cluster, or a custom scheduler or dedicated controller deployed in a cluster, and this application does not limit it to these. Furthermore, those skilled in the art will understand that the aforementioned electronic device can be an independent physical server, or a virtualization platform or cloud platform composed of multiple servers; essentially, it is any device with computing and network communication capabilities.

[0038] It should be understood that, in addition to the large model inference service scenario mentioned above, the method of this application can also be used in AI model training task scheduling, high-performance computing (HPC) job scheduling, cloud gaming resource scheduling and other scenarios. This application does not limit these to any specific scenario, as long as it is a distributed service scenario that requires GPU acceleration and has fluctuating service requests.

[0039] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings, taking any electronic device as the execution subject as an example.

[0040] This application provides a service scheduling method. Figure 2 A flowchart illustrating a service scheduling method provided in this application embodiment. Figure 1 ,like Figure 2 As shown, the method in this embodiment includes:

[0041] S201, upon receiving a user request, checks whether there is an idle active instance.

[0042] Among them, the active instance is the instance that is bound to the graphics processing unit (GPU) resources and provides the target service.

[0043] Specifically, in this embodiment, the electronic device receives and distributes user requests based on a gateway. The user request is for a target service, which can be an inference service, an AI image generation service, a natural language processing service, etc. It should be understood that in the distributed service scenario of this embodiment, multiple service instances (including active and dormant instances) are deployed in a distributed structure to achieve elastic scheduling and load balancing of the services.

[0044] Furthermore, for different types of user requests, the electronic device routes them to the corresponding target service. For example, when a user requests "generate a summary based on the input text," the electronic device routes the request to an active instance that provides natural language processing services; when a user requests "perform object detection on an uploaded image," the electronic device routes it to an active instance that runs an AI image generation service (including object detection functionality); if all active instances of the corresponding type are busy, a target instance is selected from the dormant instance pool corresponding to that type of service, its memory snapshot is loaded to restore it to an active state, and then the request is routed, ensuring that different types of service requests can be matched and processed efficiently.

[0045] In this embodiment, active instances and dormant instances are pre-created based on the minimum and maximum number of instances configured in the user-defined resource definition file. The instances with the minimum number of instances are initially marked as active, while the remaining instances are marked as dormant after the memory snapshot storage model state is processed.

[0046] Specifically, in this embodiment, during distributed service initialization, the instance count parameter configured in the user-defined resource definition is read. This instance count parameter includes a minimum number of instances and a maximum number of instances. Then, the corresponding number of service instances are created according to the maximum number of instances to complete the container environment initialization and model loading process. At this point, all instances are bound to GPU resources. Next, the instances with the minimum number of instances are marked as active, providing the corresponding target service externally. A GPU memory snapshot operation is performed on the remaining instances, persisting the model weights and context state to the shared cache or host memory, releasing GPU resources. After the snapshot is saved, these instances are marked as dormant, not providing services externally, and only available for later on-demand recovery.

[0047] In this embodiment, the memory snapshot operation can be implemented using checkpointing and recovery techniques known in the field of process or container migration, or based on relevant APIs provided by the GPU driver, which will not be elaborated here.

[0048] It should be understood that when a distributed service provides different services, the allocation ratio of the initial active instances can be dynamically adjusted based on the service type and the minimum number of instances. For example, for inference services with high-frequency requests, a higher proportion of instances in the minimum number of instances can be initially marked as active to improve response speed; for image generation services with low-frequency requests, the proportion of the initial active instances can be appropriately reduced to reduce resource consumption.

[0049] The above settings enable fine-grained pre-allocation of instances based on user configuration and service characteristics. This ensures basic service capabilities through the minimum number of active instances while reserving dormant instances within the maximum number range to meet sudden demands. It also avoids the overhead of duplicate instance creation, achieving precise matching of resources and demands from the initialization stage, laying an efficient foundation for subsequent dynamic scheduling.

[0050] In this embodiment, the electronic device determines whether there are idle active instances capable of handling new requests by monitoring at least one of the following indicators among the currently active instances: instance load, queue length, and request concurrency. Specifically, in this embodiment, when the instance load of an active instance is lower than a preset load threshold (e.g., load rate lower than 70%), the queue length is zero or less than a preset queue length (e.g., fewer than 5 queued requests), and the request concurrency has not reached the maximum processing capacity of that instance, it is determined that there are idle active instances capable of handling new requests; conversely, if the load of all active instances exceeds the threshold, the queue length reaches the upper limit, or the concurrency is saturated, it is determined that there are no idle active instances.

[0051] In the above process, by monitoring multiple dimensions or a single load metric to determine whether an active instance is idle, the real-time processing capacity of the instance can be accurately grasped, ensuring that user requests are reasonably allocated to active instances with remaining capacity, avoiding resource waste or request congestion. At the same time, the flexible idle confirmation method can adapt to the scheduling needs of different scenarios, further improving the adaptability and efficiency of service scheduling, and providing a reliable basis for quickly responding to requests or activating dormant instances.

[0052] In practical applications, electronic devices can also determine whether there are idle active instances simply by monitoring the instance load of each currently active instance. For example, if the CPU utilization of an active instance is below 60% and the GPU utilization is below 50%, the instance is considered idle; if the CPU utilization of all active instances exceeds 80% and the GPU utilization exceeds 75%, then there are no idle instances. This embodiment does not limit the method of confirming idleness, as long as it reflects whether the active instance has the ability to handle new requests.

[0053] S202, if it exists, then route the user request to an idle active instance.

[0054] Specifically, in this embodiment, the electronic device routes user requests to idle active instances according to a load balancing strategy (such as round-robin, weighted round-robin, or least connections). When multiple idle active instances exist, the instance with the lowest load is selected first; if the loads are the same, requests are allocated according to the instance registration time order or a preset weight to ensure load balancing among active instances and avoid overload of a single instance due to concentrated requests.

[0055] S203, if not, select the first target instance from the dormant instances, load the video memory snapshot of the first target instance to restore the first target instance to the active state, and then route the user request to the first target instance that has been restored to the active state.

[0056] Among them, dormant instances are instances whose model state has been stored through a memory snapshot, whose GPU resources have been released, and which are not currently providing services.

[0057] In this embodiment, the electronic device selects a corresponding number of instances as the first target instances from the dormant instances according to a preset expansion ratio or expansion step size strategy. Specifically, if the expansion ratio strategy is adopted, the electronic device determines the number of instances to be expanded based on the ratio of the total load of the currently active instances to the maximum load threshold (e.g., when the load reaches 80% of the threshold, the expansion is carried out at 50% of the current number of active instances); if the expansion step size strategy is adopted, dormant instances are selected in a fixed number (e.g., 2 instances are expanded each time) until the active instances can meet the request processing requirements. When selecting, the instance that last entered the dormant state is given priority because its memory snapshot storage time is more recent, and its state is easier to recover.

[0058] It should be understood that in practical applications, the first target instance can also be selected based on factors such as the historical recovery speed of the dormant instance and its storage location (such as local cache or remote storage). For example, the instance with the fastest recovery speed or the storage location closest to the current scheduling node can be selected first. This embodiment does not limit this.

[0059] Furthermore, in practical applications, instances corresponding to user requests can be selected from dormant instances as candidate instances. These candidate instances provide the target service corresponding to the user request. Then, based on a preset scaling ratio or scaling step size strategy, a corresponding number of instances are selected as the first target instances from these candidate instances. For example, when scaling up by 30% of the current number of active instances, dormant candidate instances that can provide the corresponding target service are first screened out. Then, a number that meets the ratio is selected from them (e.g., if there are 10 active instances, 3 are selected from the candidate instances), and instances that have most recently entered dormancy are given priority to ensure that the selected first target instances not only match the service type but can also quickly return to the active state.

[0060] In the above process, by selecting the first target instance through a preset expansion ratio or expansion step size strategy, the number of instances to be expanded can be precisely controlled according to the actual load demand, avoiding resource waste caused by excessive expansion or service delays caused by insufficient expansion. Prioritizing instances that have recently entered the dormant state can reduce the data verification and adaptation overhead during the recovery process by leveraging their more recent memory snapshot status, further shortening the switching time from dormant to active state, ensuring that user requests can be responded to quickly, and improving the flexibility and practicality of the scheduling strategy.

[0061] It should be understood that after identifying the first target instance, the electronic device restores the first target instance to an active state by loading a snapshot of its GPU memory. Specifically, the electronic device rebinds GPU resources to the first target instance and loads the model state stored in the GPU memory snapshot onto the GPU, completing the rapid reconstruction of the instance's runtime environment and enabling it to handle user requests. Simultaneously, the electronic device updates the service registration information in the service registry so that the first target instance can be used for subsequent request processing.

[0062] In this embodiment, the model state stored in the GPU memory snapshot includes model weights, context state, cache, and intermediate variables. Model weights refer to the trained model parameters, which are the core data for model inference. Context state includes environmental information such as the computation graph structure and inter-layer connections during model runtime. The cache contains temporary data generated during recent model processing requests, which can be reused to reduce redundant computation. Intermediate variables refer to intermediate results generated during model inference; retaining these variables avoids recalculation and accelerates the first inference after recovery.

[0063] The service scheduling method provided in this embodiment distinguishes between active and dormant instances. When there are idle active instances, requests are directly routed. When there are no idle instances, the first target instance selected from the dormant instances is quickly loaded based on the video memory snapshot to quickly restore to the active state and process the requests. This achieves dynamic management and efficient utilization of GPU resources.

[0064] The method described in this embodiment solves the problem of excessive time consumption for GPU resource binding and model loading during instance state switching in known technologies. It improves resource utilization efficiency while ensuring service real-time performance, effectively balancing service response speed and resource cost.

[0065] This application also provides an embodiment of a service scheduling method, illustrating that the method of this application can also trigger a scaling-down operation when necessary to further save resources. Figure 3 A flowchart illustrating a service scheduling method provided in this application embodiment. Figure 2 The following is combined with Figure 3 The implementation process and principles of this method are explained in detail. For example... Figure 3 As shown, the method in this embodiment further includes:

[0066] S301, when determining to trigger a scaling-down operation based on at least one of the following metrics among the current active instances: instance load, queue length, and request concurrency, select a second target instance from the active instances.

[0067] Specifically, in this embodiment, the electronic device still determines whether to trigger a scaling-down operation based on several indicators, including the instance load, queue length, and request concurrency of each currently active instance. More specifically, when the average load of all active instances remains below a preset scaling-down threshold (e.g., load rate below 30%), the queue length remains zero for an extended period, and the request concurrency is significantly lower than the total processing capacity of the instances, the electronic device determines to trigger a scaling-down operation. It should be understood that the preset scaling-down threshold can be configured according to requirements, and this embodiment does not limit its value.

[0068] In practical applications, the decision to trigger a scaling-down operation can also be made by selecting only one or two of these methods; this implementation does not limit this approach. For example, scaling-down can be triggered by monitoring the average load of active instances and when the load is below 20% for 5 consecutive minutes; or by combining queue length and request concurrency and triggering a scaling-down operation when the queue length is zero and the concurrency is below 10% of the instance's maximum capacity.

[0069] In this embodiment, when determining to trigger a scaling down operation, the electronic device first obtains the instance load and / or idle time of each active instance; and determines the active instances whose instance load is lower than a preset load threshold and / or whose idle time is longer than a preset duration as the second target instance.

[0070] Specifically, the electronic device first filters out active instances whose instance load is below a preset load threshold (e.g., load rate below 25%). Then, it further filters out instances whose idle time exceeds a preset duration (e.g., no request processing for 10 consecutive minutes). Instances that meet both conditions are identified as the second target instances. If only the load condition or only the idle time condition is met, the instance will not be included in the second target instance range to avoid accidentally deleting instances that may still handle sudden requests.

[0071] In practical applications, the second target instance can also be determined based solely on instance load (e.g., directly selecting the N instances with the lowest load) or solely on idle time (e.g., selecting several instances with the longest idle time). This embodiment does not limit this. Furthermore, the preset load threshold and preset time can also be configured according to requirements, and this embodiment does not limit their values.

[0072] In the above process, by combining the dual conditions of instance load and idle time to determine the second target instance, it can accurately identify instances that have been under low load for a long time and are truly idle, while avoiding excessive scaling down due to misjudgment of a single indicator. This ensures that the remaining active instances after scaling down can still stably handle current and short-term requests, reducing resource waste while ensuring service stability and redundancy.

[0073] S302, after performing a video memory snapshot operation on the second target instance, release the corresponding GPU resources and mark the second target instance as hibernating.

[0074] Specifically, after confirming that the second target instance has no outstanding user request processing tasks, the electronic device performs a memory snapshot operation on its currently running model state (including model weights, context state, cache, and intermediate variables), storing the snapshot data in shared storage or local cache. After the snapshot storage is complete, the binding relationship between the instance and GPU resources is released, the GPU memory and computing resources it occupies are released, and the instance state is updated to sleep state. It should be understood that the released GPU resources can be occupied by other instances.

[0075] In this embodiment, after the electronic device performs a video memory snapshot operation on the second target instance and releases the corresponding GPU resources, it retains the container runtime environment corresponding to the second target instance, thus avoiding the time consumption of re-initializing the container when it is restored to the active state, and further shortening the state switching cycle.

[0076] In addition, after marking the second target instance as dormant, the service registration information of the second target instance is deregistered from the service registry to prevent new user requests from being routed to that instance and ensure the accuracy of service scheduling.

[0077] In the method of this embodiment, since a video memory snapshot operation is performed on the second target instance during the scaling down process to save the model state, the GPU resources it occupies are released and it is marked as dormant, while the container runtime environment is preserved, the dormant instance can be quickly restored to the active state by loading the video memory snapshot when needed later, without re-initializing the container and loading the complete model. Thus, while realizing the dynamic release of resources to save GPU resources, the efficiency of instance state switching is guaranteed, providing reliable support for the elastic scaling of services.

[0078] As described above, this application, by introducing a mechanism for switching between active and dormant states of model service instances, combined with memory snapshots and traffic-aware scheduling strategies, achieves fine-grained management of GPU resources and rapid instance recovery capabilities. This effectively improves the response efficiency and resource utilization of model services, ensuring service stability and cost-effectiveness under varying loads. Specifically:

[0079] 1. Effectively reduce cold start time: With the help of memory snapshot technology, when recovering from dormant state to active state, there is no need to reload model files or rebuild the context environment, which greatly shortens the time required for instance cold start, especially suitable for high-frequency start and stop scenarios of large models;

[0080] 2. Improve GPU resource utilization: Dormant instances release their GPU resources without destroying the model state, allowing resources to be redistributed to other high-priority tasks or users, significantly improving GPU utilization efficiency and reducing computing resource waste;

[0081] 3. Supports rapid elastic scaling of services: The platform can quickly wake up dormant instances or release some active instances based on real-time traffic load, achieving second-level elastic scaling capabilities and improving the system's resilience and agility in the face of sudden traffic surges.

[0082] 4. Improve service availability and scheduling accuracy: By scheduling requests only to active instances and combining this with the gateway status registration mechanism, we ensure that all instances participating in the service are in a ready state, effectively avoiding service interruptions and delays, and improving overall service availability and request processing success rate.

[0083] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0084] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0085] The above embodiments introduce a method for training a recommendation model and a method for recommending products from the perspective of process flow. The following embodiments introduce a training device for a recommendation model and a device for recommending products from the perspective of virtual modules or virtual units. For details, please refer to the following embodiments.

[0086] This application provides a service scheduling device. Figure 4 This is a schematic diagram of the structure of a service scheduling device provided in an embodiment of this application, as shown below. Figure 4 As shown, the device includes:

[0087] Detection module 41 is used to detect whether there is an idle active instance when a user request is received; wherein, the active instance is an instance that is bound to GPU resources and provides the target service;

[0088] The scheduling module 42 is used to route user requests to idle active instances when there are idle active instances; when there are no idle active instances, it selects a first target instance from dormant instances, loads the video memory snapshot of the first target instance to restore the first target instance to the active state, and then routes user requests to the first target instance that has been restored to the active state; wherein, the dormant instance is an instance that has stored the model state through a video memory snapshot, released GPU resources, and is not providing services for the time being.

[0089] In another possible implementation of this application, the active instance and the dormant instance are pre-created based on the minimum number of instances and the maximum number of instances configured in the user-defined resource definition file. The instance with the minimum number of instances is initially marked as active, and the remaining instances are marked as dormant after the memory snapshot storage model state is processed.

[0090] In another possible implementation of this application embodiment, the scheduling module 42 is specifically used for:

[0091] Based on the preset expansion ratio or expansion step size strategy, select a corresponding number of instances from dormant instances as the first target instances.

[0092] Another possible implementation of this application embodiment is that the model state stored in the video memory snapshot includes model weights, context state, cache, and intermediate variables.

[0093] In another possible implementation of this application embodiment, the scheduling module 42 is further used for:

[0094] When a scaling-down operation is triggered based on at least one of the following metrics: instance load, queue length, and request concurrency of each currently active instance, a second target instance is selected from the active instances.

[0095] After performing a memory snapshot operation on the second target instance, the corresponding GPU resources are released, and the second target instance is marked as hibernating.

[0096] In another possible implementation of this application embodiment, the scheduling module 42 is specifically used for:

[0097] Get the instance load and / or idle time of each active instance;

[0098] Active instances whose instance load is below a preset load threshold and / or whose idle time is longer than a preset duration are identified as the second target instances.

[0099] Another possible implementation of this application embodiment is to retain the container runtime environment corresponding to the second target instance.

[0100] In another possible implementation of this application embodiment, the scheduling module 42 is further used for:

[0101] After marking the target active instance as dormant, deregister the target active instance's service registration information from the service registry.

[0102] In another possible implementation of this application embodiment, the detection module 41 is specifically used for:

[0103] By monitoring at least one of the following metrics for each currently active instance: instance load, queue length, and request concurrency, it can be determined whether there are any idle active instances that can handle new requests.

[0104] The service scheduling apparatus provided in this application is applicable to the above-described service scheduling method embodiments, and will not be described in detail here.

[0105] This application provides an electronic device. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown, Figure 5 The illustrated electronic device includes at least one processor 51 and a memory 52. ​​The processor 51 and the memory 52 are connected, for example, via a bus 53. Optionally, the electronic device may also include a transceiver 54. It should be noted that in practical applications, the transceiver 54 is not limited to one, and the structure of this electronic device does not constitute a limitation on the embodiments of this application.

[0106] Processor 51 may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 51 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0107] Bus 53 may include a pathway for transmitting information between the aforementioned components. Bus 53 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 53 may be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus 53 is represented by a single thick line, but this does not mean that there is only one bus 53 or only one type of bus 53.

[0108] The memory 52 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0109] The memory 52 is used to store computer execution instructions for implementing the scheme of this application, and the execution is controlled by the processor 51. The processor 51 is used to execute the computer execution instructions stored in the memory 52 to implement the content shown in the foregoing method embodiments.

[0110] This application also provides a computer-readable storage medium, which may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Specifically, the computer-readable storage medium stores computer-executable instructions, which are used to implement the methods in the above embodiments.

[0111] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the technical solution of the above method embodiments. Its implementation principle and technical effects are similar, and will not be repeated here.

[0112] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0113] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A service scheduling method, characterized in that, The method includes: Upon receiving a user request, the system checks whether there are any idle active instances; wherein, the active instance is an instance that is bound to a graphics processing unit (GPU) resource and provides the target service. If it exists, the user request will be routed to an idle active instance; If it does not exist, select the first target instance from the dormant instances, load the memory snapshot of the first target instance to restore the first target instance to the active state, and then route the user request to the first target instance that has been restored to the active state; wherein, the dormant instance is an instance that has stored the model state through a memory snapshot, released GPU resources and is not providing services for the time being.

2. The method according to claim 1, characterized in that, The active instance and the dormant instance are pre-created based on the minimum and maximum number of instances configured in the user-defined resource definition file. The instance with the minimum number of instances is initially marked as active, and the remaining instances are marked as dormant after the memory snapshot storage model state is processed.

3. The method according to claim 1 or 2, characterized in that, The step of selecting the first target instance from the dormant instances includes: Based on the preset expansion ratio or expansion step size strategy, a corresponding number of instances are selected from the dormant instances as the first target instances.

4. The method according to claim 3, characterized in that, The model state stored in the video memory snapshot includes model weights, context state, cache, and intermediate variables.

5. The method according to claim 1 or 2, characterized in that, The method further includes: When a scaling-down operation is triggered based on at least one of the following metrics: instance load, queue length, and request concurrency of each currently active instance, a second target instance is selected from the active instances. After performing a memory snapshot operation on the second target instance, the corresponding GPU resources are released, and the second target instance is marked as hibernating.

6. The method according to claim 5, characterized in that, The step of selecting a second target instance from the active instances includes: Obtain the instance load and / or idle time of each active instance; The active instances whose instance load is lower than a preset load threshold and / or whose idle time is longer than a preset duration are identified as the second target instances.

7. The method according to claim 5, characterized in that, The container runtime environment corresponding to the second target instance is retained.

8. The method according to claim 5, characterized in that, The method further includes: After marking the second target instance as dormant, the service registration information of the second target instance is deregistered from the service registry.

9. The method according to claim 1 or 2, characterized in that, The detection of whether there is an idle active instance includes: By monitoring at least one of the following metrics for each currently active instance: instance load, queue length, and request concurrency, it can be determined whether there are any idle active instances that can handle new requests.

10. A service scheduling device, characterized in that, The device includes: The detection module is used to detect whether there is an idle active instance when a user request is received; wherein, the active instance is an instance that is bound to GPU resources and provides the target service; The scheduling module is used to route the user request to an idle active instance when an idle active instance exists; when no idle active instance exists, it selects a first target instance from dormant instances, loads a memory snapshot of the first target instance to restore the first target instance to an active state, and then routes the user request to the first target instance that has been restored to an active state; wherein, the dormant instance is an instance that has stored the model state through a memory snapshot, released GPU resources, and is not providing services for the time being.

11. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-9.

Images