Model service management system, method, and electronic device

Abstract

The application discloses a model service management system and method and electronic equipment, and relates to the technical field of model management. The system comprises a service management module for managing model services, a deployment group located in a plurality of physical clusters is set in the model services, and different types of instances are set in the deployment group. An index monitoring module is used to obtain index values of the instances in the deployment group to obtain current index values of the model services. A service adjustment module is used to generate a service adjustment request according to the current index values and expected index values to adjust the model services. The system can solve the problems that it is difficult to manage various model services across physical clusters and it is impossible to perform adjustment operations such as expansion and contraction of the model services in combination with the overall resource requirements of the model services. The system uniformly manages different types of model services through the deployment group and supports deployment of the model services across physical clusters. The model services are adjusted by using the index values of the instances in the deployment group and the expected index values, so that the resource requirements of the model services are met.

Description

Technical Field

[0001] This invention relates to the field of model management technology, specifically to model service management systems, methods, and electronic devices. Background Technology

[0002] Currently, the inference process of large models can be divided into two stages: the prefill (P) stage and the decoding (D) stage. When deploying large model services, a coupled deployment approach is typically used, where P instances and D instances run within the same container pod. For scenarios requiring high performance and high load, a separate deployment approach is adopted, where P instances and D instances run on different pods. Currently, for coupled services, a single deployment controller (Deployment) can manage multiple instances of the service. For P / D separated services, two Deployments are used to manage multiple P instances and D instances respectively, combined with HPA (Horizontal Pod Autoscaler) to automatically adjust the number of instances under each Deployment.

[0003] However, current clusters often contain both large model services with separate P / D configurations and coupled configurations. HPA uses CPU and memory usage as hardware metrics, which cannot accurately capture the dynamic resource requirements of P / D separated model services. Furthermore, the current method of managing model services using only Deployments is insufficient because Deployments are typically limited to a single physical cluster, making it difficult to directly manage and schedule model services across different physical clusters in a unified manner. Summary of the Invention

[0004] This invention provides a model service management system, method, and electronic device to solve the problems of difficulty in managing various model services across physical clusters and the inability to perform scaling up or down adjustments to model services based on the overall resource requirements of the model services.

[0005] In the first aspect, this application provides a model service management system, which includes: a service management module, an indicator monitoring module, and a service adjustment module; The service management module is used to manage a first preset number of model services, wherein the model services include a second preset number of deployment groups, and the deployment groups include a first type of instance and a second type of instance. The first type of instance and the second type of instance are either in a separate state or in a coupled state, and the deployment groups are located in one or more physical clusters. The metrics monitoring module is used to obtain the first metric value of the first instance and the second metric value of the second instance in the deployment group, and to obtain the current metric value of the model service based on the service type, the first metric value, and the second metric value of the model service. The service adjustment module is used to obtain the current value of the indicator from the indicator monitoring module and generate a service adjustment request based on the current value and the expected value of the indicator. The service management module is also used to adjust the model service according to the service adjustment request received.

[0006] Secondly, this application provides a model service management method, which includes: Manage a first preset number of model services, wherein the model services include a second preset number of deployment groups, and the deployment groups include first type instances and second type instances. The first type instances and the second type instances are either in a separate state or in a coupled state, and the deployment groups are located in one or more physical clusters. Obtain the first metric value of the first instance and the second metric value of the second instance in the deployment group. Based on the service type, the first metric value, and the second metric value of the model service, obtain the current metric value of the model service. Generate a service adjustment request based on the current value and expected value of the indicator, and adjust the model service according to the service adjustment request.

[0007] Thirdly, this application provides a model service management device, which includes: The model service management module is used to manage a first preset number of model services, wherein the model service includes a second preset number of deployment groups, and the deployment group includes a first type of instance and a second type of instance. The first type of instance and the second type of instance are in a separate state or a coupled state, and the deployment group is located in one or more physical clusters. The indicator value determination module is used to obtain the first indicator value of the first instance and the second indicator value of the second instance in the deployment group, and to obtain the current indicator value of the model service based on the service type, the first indicator value and the second indicator value of the model service. The model service adjustment module is used to generate service adjustment requests based on the current value and expected value of the indicator, and to adjust the model service according to the service adjustment requests.

[0008] Fourthly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the model service management method of the second aspect or any corresponding embodiment described above.

[0009] Fifthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the model service management method of the second aspect or any corresponding embodiment described above.

[0010] Sixthly, this application provides a computer program product, including computer instructions for causing a computer to execute the model service management method of the second aspect or any corresponding embodiment described above.

[0011] This application utilizes a service management module to manage model services. Deployment groups are set up within each model service, and these groups reside in one or more physical clusters. Furthermore, first-type and second-type instances are configured within each deployment group, either in a separate or coupled state. A metric monitoring module retrieves the metric values ​​of the instances within the deployment group, obtaining the current metric values ​​for the model service. A service adjustment module generates service adjustment requests based on the current and expected metric values ​​to adjust the model service. This addresses the challenges of managing various model services across physical clusters and the inability to scale up or down the model service based on its overall resource requirements. The system manages different types of model services uniformly through deployment groups, supports cross-physical cluster deployment, coordinates resources across multiple clusters, and improves resource utilization. The metric monitoring module collects metric values ​​calculated from instances to determine the current metric values ​​of the model service, and then adjusts the model service's scale based on these current and expected values ​​to meet its resource requirements. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this application, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of a model service management system according to an embodiment of this application; Figure 2 This is an architecture diagram of the management and scaling process of LLM services according to embodiments of this application; Figure 3 This is a schematic diagram of a cluster resource topology tree according to an embodiment of this application; Figure 4 This is a flowchart illustrating the model service management method according to an embodiment of this application; Figure 5 This is a structural block diagram of a model service management device according to an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0015] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0016] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] Large Language Models (LLMs) have demonstrated powerful generalization capabilities and practical effectiveness in numerous tasks such as text generation and language understanding. The inference process of a large language model can typically be divided into two phases: the Prefill phase and the Decode phase. The Prefill phase processes all input tokens, generates the first output token, and creates a KVCache (key-value pair cache). The Decode phase utilizes the KVCache for multiple iterations, generating a token and its corresponding KVCache in each iteration. The Prefill phase is usually computationally intensive, while the Decode phase is typically limited by GPU memory bandwidth bottlenecks. Currently, large language model services are typically deployed using a coupled Prefill (P) and Decode (D) phase deployment, meaning the P and D phases run in the same Pod. However, this approach has relatively simple resource scheduling and is difficult to optimize for the computational characteristics of different phases, making it more suitable for scenarios with small models, simple tasks, and few users. For scenarios requiring high performance and high load, the development of separate deployment of the P and D phases is underway. That is, the Prefill and Decode phases run on different Pods or different nodes, which can optimize the resource usage of each phase individually, but places higher demands on network bandwidth, cache transmission and scheduling strategies.

[0018] Currently, within a physical cluster, Deployments can be used to manage service instances. For P / D coupled model services, one Deployment can manage multiple instances of the model service. For P / D separated model services, two Deployments are used to manage multiple P instances and multiple D instances respectively. In the physical cluster, the Metrics Server can be used to obtain metrics such as processor utilization and memory utilization, which, combined with HPA, automatically adjusts the number of Deployment instances. HPA can be configured with scaling metric names, scaling thresholds, periodic intervals, and the service's Deployment name. After configuration, HPA will periodically read the scaling metric values ​​from the Metrics Server, determine if the current metric value is greater than the scaling threshold, and if so, scale up; otherwise, scale down. For coupled services, since there is only one Deployment, one HPA can be set up for that Deployment to control its scaling. For P / D separated services, two HPAs are set up to scale independently for P and D respectively.

[0019] However, current physical clusters often contain both P / D separated model services and P / D coupled model services. HPA cannot distinguish between P instances and D instances in a P / D separated architecture. Independent scaling of P and D instances' Deployments can lead to situations where one P / D service has idle resources while the other has insufficient resources, ultimately dragging down overall throughput. Furthermore, HPA uses metrics such as processor utilization and memory utilization, which cannot accurately capture the dynamic resource requirements of P / D separated services. Moreover, the current design, which uses Deployments to manage load, cannot deploy model services across physical clusters.

[0020] Based on the above, this application provides a model service management system with three modules: a service management module, a service scaling module, and a metric monitoring module. The service management module incorporates deployment groups to uniformly manage both P / D coupled and P / D separated services. It provides entry points for deployment group discovery and invocation, and includes service scheduling and P / D ratio calculation functions. The service scheduling function considers network affinity, homogeneous resources, and heterogeneous resources. The scaling module integrates scaling decision-making functions for both P / D separated and coupled services, and designs scaling thresholds and cooldown time mechanisms, as well as a method for scaling P / D type services proportionally. The service monitoring module integrates monitoring metrics for both P / D separated and coupled services. For P / D type services, a weighted average of Prefill and Decode type metrics is designed as a measure of the overall service performance. This system utilizes deployment groups to unify P / D separated or coupled model services and proposes service scaling methods and monitoring metrics corresponding to each deployment group, thereby achieving lifecycle management and scaling of various cross-cluster services.

[0021] The specific application environment architecture or specific hardware architecture on which the execution of the model service management method depends is described here.

[0022] According to an embodiment of this application, a model service management system embodiment is provided. It should be noted that the modules in this system can be set in a computer system such as a set of executable instructions. Although the connection relationship between modules is shown in the structural diagram, in some cases, different modules can be connected with a different connection relationship than shown here.

[0023] This embodiment provides a model service management system. Figure 1 This is a schematic diagram of the structure of the model service management system according to an embodiment of this application, such as... Figure 1 As shown, the system includes: a service management module, a metric monitoring module, and a service adjustment module; The service management module is used to manage a first preset number of model services, wherein the model services include a second preset number of deployment groups, and the deployment groups include a first type of instance and a second type of instance. The first type of instance and the second type of instance are either in a separate state or in a coupled state, and the deployment groups are located in one or more physical clusters. The metrics monitoring module is used to obtain the first metric value of the first instance and the second metric value of the second instance in the deployment group, and to obtain the current metric value of the model service based on the service type, the first metric value, and the second metric value of the model service. The service adjustment module is used to obtain the current value of the indicator from the indicator monitoring module and generate a service adjustment request based on the current value and the expected value of the indicator. The service management module is also used to adjust the model service according to the service adjustment request received.

[0024] Specifically, in this embodiment, for coupled model services, a one-cluster-one-Deployment approach is used to deploy the model service, and this Deployment is defined as a deployment group. All deployment groups in the physical clusters constitute a single model service, providing users with a unified service call entry point. Each deployment group can manage multiple replicas. For P / D separated model services, P instances and D instances are bound together, with each instance implemented using a Deployment. The structure formed by these two Deployments is also defined as a deployment group. A model service can have multiple deployment groups, and a deployment group can have multiple service replicas. P instances and D instances of the same service replica are bound to the same scheduling step, ensuring network affinity between the two roles. During scheduling, replicas are created first in the low-priority resource pool to reserve high-priority resources for more critical workloads.

[0025] The service management module manages a first preset number of model services, where the first preset number and the second preset number represent one or more. Each model service contains a second preset number of deployment groups. Each deployment group contains first-type instances and second-type instances, which can be either separate or coupled. For P / D separated model services, the first-type instance is a P instance, and the second-type instance is a D instance; they reside in different Pods or nodes and are in a separate state. For coupled model services, the P phase and D phase reside in the same Pod; in this case, the first-type instance can be a Pod in the P phase, and the second-type instance can be a Pod in the D phase; they are in a coupled state. Deployment groups reside in one or more physical clusters, enabling cross-physical cluster deployment of model services.

[0026] The service management module manages the entire lifecycle of large model services, including creation, deletion, scheduling, and providing service call entry points. The service management module manages two types of services: P / D separated model services, where the Prefill and Decode phases run in different inference Pods, and coupled services, where the Prefill and Decode phases run together in the same Pod. Service providers control the service lifecycle, including creation and deletion, by interacting with the service management module.

[0027] The metrics monitoring module integrates monitoring metrics for both P / D separated and coupled model services. This module collects metrics from the model services, such as TPS (Tokens Per Second), TTFT (Time to First Token Second), and RLS (Request Latency Sensations). When there are multiple instances in a deployment group, the metric values ​​are recorded as averages. The metric monitoring module connects to the service management module, obtaining the first metric value from the first instance and the second metric value from the second instance in the deployment group. Based on the model service's service type, the first metric value, and the second metric value, the current metric value for the model service is obtained. For example, for P / D separated model services, the weighted sum of the metric values ​​from the P and D instances is used as the overall metric, i.e., the current metric value; for coupled model services, the sum of the metric values ​​from multiple instances is used as the current metric value.

[0028] The service adjustment module integrates the scaling decision-making functions for both P / D separated and coupled model services. It also incorporates scaling thresholds and cooldown time mechanisms, as well as a method for proportional scaling of P / D type services. The module obtains current metric values ​​from the metric monitoring module and generates service adjustment requests based on these values ​​and expected values. For example, it can use the relationship between the current and expected metric values ​​to scale the model service up or down, generating a service scaling-down request; or it can adjust the number of service replicas in the model service based on the ratio of the current to the expected metric value, generating a service adjustment request. The expected metric value can be a single value or a range of values. Service adjustment requests include service scaling-down and service scaling-up requests. Upon receiving a service adjustment request, the service management module adjusts the model service accordingly.

[0029] The metrics monitoring module includes a metrics statistics submodule and a metrics processing submodule. The inference engine of the currently launched model generally has metric calculation capabilities, capable of calculating metrics such as TPS, TTFT, and RLS. The metrics monitoring module periodically collects metric values ​​calculated from the first and second type instances in the deployment group. The first metric value includes, for example, the TPS, TTFT, and RLS values ​​for the first type of instance; the second metric value includes, for example, the TPS, TTFT, and RLS values ​​for the second type of instance. When there are multiple instances in the deployment group, the metric values ​​are recorded as averages.

[0030] The first preset type is a P / D separation model service. The second preset type is a coupled model service.

[0031] When the service type is the first preset type, the weighted average of the Prefill and Decode types is used as the overall metric. This is based on preset weights (…). , The first and second indicator values ​​are weighted and summed, and the weighted sum is used as the current value of the indicator, as shown in formula (1).

[0032] (1) in, This is an adjustable parameter, ranging from 0 to 1; different indicators have different values. For example: The specific values ​​are set based on empirical values. For throughput metrics, the Decode type takes precedence. Set it to a smaller value, for example, TPS of 0.3; for the first token delay metric, Prefill dominates. Set it to a larger value, for example, TTFT is set to 0.8; for end-to-end delay metrics, PD equalization, Take the median value, for example, RLS is 0.5.

[0033] When the service type is the second preset type, the sum of the first and second indicator values ​​is used as the current indicator value. After the indicators are collected, they can provide the service adjustment module with the basis for scaling up and down the model service. The current indicator values ​​of the model service are collected according to the above method, as shown in Table 1.

[0034] Table 1 Current Value Record Table of Indicators

[0035] By combining the model service management system of this embodiment with the large language model, an architecture for the management and scaling processes of LLM services can be obtained, such as... Figure 2As shown, the service scaling module is set as the service adjustment module. The service scaling module contains scaling decisions for both P / D separation and P / D coupling types. The service management module has functions for service creation, service scheduling, P / D ratio calculation, service deletion, and call entry points. The service management module contains model service 1, model service 2, ..., model service N. Model service 1 is a model service of the P / D separation type, and model service 2 is a model service of the P / D coupling type. In deployment group 1 of model service 1, the Prefill phase contains multiple P instances, and the Decode phase contains multiple D instances. If there are 2 replicas, the P / D ratio is 2:1, meaning two P instances and one D instance together constitute one service replica. Additionally, the metrics monitoring module is used to collect metrics from the service management module, and also for metric storage and maintenance. The service adjustment module reads metrics from the metrics monitoring module and executes corresponding strategies based on the scaling decisions in the P / D separation or P / D coupling types. It can also read service information from the service management module.

[0036] The model service management system provided in this embodiment manages model services using a service management module. Deployment groups are set up within each model service, and these groups reside in one or more physical clusters. Furthermore, first-type instances and second-type instances are set up within each deployment group, either in a separate or coupled state. A metric monitoring module obtains the metric values ​​of the instances within the deployment group to determine the current metric values ​​of the model service. A service adjustment module generates service adjustment requests based on the current and expected metric values ​​to adjust the model service. This system manages different types of model services uniformly through deployment groups, supports cross-physical cluster deployment of model services, coordinates resources across multiple clusters, and improves resource utilization. The metric monitoring module collects metric values ​​calculated by instances to determine the current metric values ​​of the model service. Based on the current and expected metric values, the system scales up or down the model service to meet its resource requirements. This solves the problems of difficulty in managing various model services across physical clusters and the inability to perform scaling up or down adjustments based on the overall resource requirements of the model service.

[0037] As an optional embodiment, the system also includes a front-end interface module; the service management module includes a service creation submodule and a service scheduling submodule; The front-end interface module is used to obtain the first service parameters of the service to be adjusted input through the front-end interface and transmit the first service parameters to the service creation sub-module; The service creation submodule is used to obtain the instance ratio between the first type of instance and the second type of instance, create other service adjustment requests based on the instance ratio and the first service parameters, and transmit the other service adjustment requests to the service scheduling submodule so that the service scheduling submodule can adjust the model service according to the other service adjustment requests.

[0038] Specifically, the front-end interface module can provide the service provider with a UI (User Interface).

[0039] The service provider fills in the first service parameters in the front-end UI interface. These parameters include: service name, model name to be used by the service, the number of GPUs that can be used per service replica, service priority (set to the lowest priority for coupled services), network affinity constraints (set to cluster level for coupled services), scaling metrics and expected values, and service type (P / D separation or coupling). The front-end UI module packages these first service parameters into a request and sends it to the service creation submodule in the back-end service management module.

[0040] The service creation submodule obtains the instance ratio between the first type of instances and the second type of instances, such as the optimal ratio of P instances to D instances. After receiving the first service parameter, the service creation submodule creates other service adjustment requests based on the instance ratio and the first service parameter. For example, for coupled model services, it generates a request to initialize a replica as an other service adjustment request and sends it to the service scheduling submodule. For P / D separated model services, it first calls the P / D ratio calculation submodule to calculate the optimal ratio of P instances to D instances, and then creates a request to initialize a replica based on the instance ratio and the first service parameter, sending it as an other service adjustment request to the scheduling submodule. The other service adjustment requests are then transmitted to the service scheduling submodule so that it can adjust the model service accordingly.

[0041] Additionally, service providers can generate service deletion requests through the front-end interface module. The service deletion submodule in the service management module executes the service deletion operation based on the service deletion request. For example, it iterates through each physical machine in the physical machine hierarchy to see if there are any instances of the model service; if so, it deletes those instances.

[0042] In this embodiment, parameters are conveniently input through the front-end UI, adapting to service types with coupling and P / D separation. Adjustment requests are accurately generated based on the instance ratio, supporting rapid service initialization, adjustment, and deletion, simplifying service management processes, and improving the efficiency of model service configuration and operation and maintenance.

[0043] As an optional embodiment, the service management module includes an entry point submodule; The entry submodule is invoked to find deployment groups belonging to the same model service and record the first type of instance of the deployment group in a preset file; The entry point submodule is used to forward the model call request to the first type instance according to the preset file when a model call request is received.

[0044] Specifically, the call entry submodule in the service management module periodically identifies deployment groups belonging to the same model service and records these deployment groups in a preset file, such as Table 2, the instance record table, with examples shown in Table 2. In particular, for P / D separation type model services, the call entry submodule only records Prefill type instances (i.e., the first type instances), because the internal logic of the P / D separation type model service automatically calls the Decode instance simply by forwarding user requests to the Prefill instance.

[0045] When a user sends a model call request, the call entry submodule uses a polling method to forward the model call request to the first type instance based on information such as the instance IP and replica name in the preset file, and then forwards it to each instance in the deployment group.

[0046] Table 2 Instance Record Table

[0047] As an optional embodiment, the service adjustment module includes a data processing submodule, a judgment submodule, and a request generation submodule; The data processing submodule is used to obtain the current number of replicas of the model service, as well as the target value of the indicator, the current time, and the historical adjustment time. The data processing submodule is also used to obtain the target number of replicas based on the ratio between the target value and the current value of the indicator and the current number of replicas, determine the time difference between the current time and the historical adjustment time, and determine the ratio between the target number of replicas and the current number of replicas. The judgment submodule is used to compare the quantity ratio with a first preset threshold to obtain a first comparison result, and to compare the time difference with a second preset threshold to obtain a second comparison result. The request generation submodule is used to generate a service adjustment request based on the second service parameters of the model service, the first comparison result, and the second comparison result.

[0048] Specifically, the service adjustment module is a service scaling module used to adjust the scaling of two types of model services. The core of the service scaling module is the scaling decision function, which includes scaling decision functions for P / D separation type and coupled type model services. The number of instances in the replicas is calculated according to the optimal ratio in the P / D separation type service (for example, if the number of replicas is 1 and the P / D ratio is 2:1, then 1*(2+1) instances are needed), while in the coupled type service, it is one instance.

[0049] The data processing submodule obtains the current number of replicas of the model service. and obtain the target value of the indicator. Current time and historical adjustment time .

[0050] The data processing submodule determines the ratio between the target value and the current value of the indicator, as well as the current number of replicas, to obtain the target number of replicas. For example: based on current observation indicators The ratio between the target and expected metrics is used to calculate the number of replicas that should be available under the expected conditions. For situations where a higher value of the metric is better (such as TPS, Decode TPS, etc.), the target number of replicas is calculated using formula (2). For situations where a smaller indicator value is better (such as request latency), the target number of replicas can be calculated using formula (3). .

[0051] (2) (3) Determine the time difference between the current time and the historical adjustment time. , Utilizing time difference A cooling mechanism is introduced to prevent the system from shaking or oscillating due to frequent expansion and contraction.

[0052] Determine the ratio of the target number of replicas to the current number of replicas. , Quantity ratio This indicates the degree of deviation between the current load and the target load.

[0053] For example, the first preset threshold is: , , It is typically set to 0.1 to 0.5, indicating that the current number of replicas exceeds the current number. Double the capacity to avoid minor fluctuations triggering expansion. It is generally set to 0.1 to 0.5 to avoid slight fluctuations triggering shrinkage.

[0054] For example, the second preset threshold is: , , It is typically set to 0.1 to 0.5, indicating that the current number of replicas exceeds the current number. Double the capacity to avoid minor fluctuations triggering expansion. The typical setting is 60 to 120 seconds. Allow a "cooling-off period" for capacity expansion to avoid disruptions caused by expansion followed by contraction. It is generally set to 0.1 to 0.5 to avoid slight fluctuations triggering shrinkage. It is generally set to 60s to 120s. Allow a certain "cooling-off period" for volume reduction to avoid shocks caused by expanding or reducing volume immediately after reduction.

[0055] The submodule will determine the quantity ratio The first comparison result is obtained by comparing the time difference with a first preset threshold; the second comparison result is obtained by comparing the time difference with a second preset threshold. The request generation submodule generates a service adjustment request based on the second service parameters of the model service, the first comparison result, and the second comparison result. For example, if the following conditions are met simultaneously: ,and This triggers an expansion operation, increasing the number of replicas. If both conditions are met: ,and This will trigger a shrinkage operation, reducing the number of replicas to [a smaller number]. .

[0056] Generate service adjustment requests, such as based on service name, service type, service priority, service network affinity constraints, scaling up / down, and the number of replicas to be adjusted. Based on parameters such as these, a service adjustment request is generated and sent to the scheduling submodule of the service management module.

[0057] In this embodiment, the scaling decision-making functions for P / D separation and coupled model services are integrated using a service adjustment module. A threshold is employed to prevent scaling from being triggered by slight fluctuations in metrics, and a cooling mechanism is used to prevent system jitter or oscillation due to frequent scaling. Furthermore, P / D separation model services are scaled up or down proportionally to avoid a decrease in service throughput caused by scaling.

[0058] As an optional embodiment, the service management module includes a service scheduling submodule; The service scheduling submodule is used to determine the service priority of the model service corresponding to the service adjustment request, and to obtain the pending requests from the service adjustment request based on the service priority. The service scheduling submodule is also used to obtain the request type of the pending request, the network affinity constraint information corresponding to the pending request, the instance ratio and cluster resource information, and to determine the number of service replicas corresponding to the pending request. The instance ratio is the ratio between the number of first type instances and the number of second type instances in the model service corresponding to the pending request. The service scheduling submodule is also used to process pending requests based on request type, network affinity constraints, instance ratio, cluster resource information, and the number of service replicas.

[0059] Specifically, when the service scheduling submodule performs actual scheduling, it adopts the following steps for P / D separation or coupling types. The scheduling rules are as follows: when scaling up, under the network affinity constraint, homogeneous resources are used first, and scaling up is carried out proportionally in existing deployment groups first; when scaling down, high network affinity levels are released first, and heterogeneous resources are released first.

[0060] The service scheduling submodule determines the service priority of the model service corresponding to the service adjustment request. It sorts all scaling requests for services in the data center according to priority (each service is assigned a priority upon creation; the priority can be a number, such as 1, 2, 3). This ensures that high-priority services are processed first, preventing low-priority services from preempting resources from high-priority services. Based on service priority, it retrieves pending requests from the service adjustment requests. For example: sorting requests according to service priority to generate a request queue. In generating the request queue Get pending requests In addition, it needs to be processed in a loop. Each request in the process, until it is completed. All requests in the.

[0061] The service scheduling submodule obtains the request type, network affinity constraints, instance ratio, and cluster resource information of the pending requests. Request types include, for example, service scaling-up and scaling-down requests. Network affinity constraints for the pending requests include cluster-level, aggregation-level, rack-level, and physical machine-level affinity requirements. Depending on the task of the large model service, different network affinity constraints will be assigned when the service is created. The instance ratio is the ratio between the number of first-type instances and the number of second-type instances in the model service corresponding to the pending request. For example, if the first-type instances are P instances and the second-type instances are D instances, the ratio of P instances to D instances is 2:1, 3:1, or other ratios. Cluster resource information includes, for example, the number of GPU cards in the physical cluster.

[0062] Determine the number of service replicas corresponding to the request to be processed. For example, obtain the number of replicas S that need to be adjusted from the request. (For services with P / D separation, when calculating the required number of instances, it is necessary to calculate according to the P / D ratio of the service. For example, if the number of replicas is 2 and the P / D ratio is 2:1, then 2 replicas are needed.) There are (2+1) instances, each requiring one GPU card. For coupled services, one replica corresponds to one instance. For example, if there are 2 replicas, then 2 instances are needed, each requiring one GPU card.

[0063] The service scheduling submodule processes pending requests based on request type, network affinity constraints, instance ratio, cluster resource information, and the number of service replicas.

[0064] In this embodiment, the service scheduling submodule stratifies resources based on the network topology of the entire data center, groups GPU devices into homogeneous and heterogeneous groups, and binds the P / D of the same replica to the same scheduling step. This ensures network affinity between the two roles and prioritizes creating replicas in the low-priority resource pool to reserve high-priority resources for more critical workloads. This intelligent scheduling strategy avoids services being limited by local resource exhaustion and optimizes network and hardware priorities globally.

[0065] As an optional embodiment, the service management module includes an instance ratio calculation submodule; The instance ratio calculation submodule is used to determine the processor number threshold and test metrics of the model service, and generate a third preset number of candidate instance ratios based on the processor number threshold. The instance ratio calculation submodule is also used to configure the model service according to the candidate instance ratio, test the configured model service according to the test data and test configuration information, and obtain the test values ​​of the test indicators corresponding to the candidate instance ratio during the test process. The instance ratio calculation submodule is used to determine the difference between the test value corresponding to the candidate instance ratio and the target value of the test metric, take the candidate instance ratio corresponding to the test value with the smallest difference as the instance ratio, and transmit the instance ratio to the service scheduling submodule.

[0066] Specifically, due to the performance bottleneck differences between the P and D phases, achieving maximum performance typically requires asymmetric allocation of P and D instances. To determine the optimal ratio of P to D instances, an instance ratio calculation submodule performs the P / D ratio calculation function. This submodule uses different open-source data as test data depending on the service model. It utilizes the service provider's configured service single-replica limit on the number of GPUs used and scaling metrics, employing stress testing to determine the optimal P / D ratio.

[0067] The instance ratio calculation submodule determines the processor number threshold and test metrics for the model service. The processor number threshold, for example, is the limit on the number of GPUs Z that can be used per service replica, where Z is 4, 5, or another value. Test metrics include, for example, TPS and TTFT.

[0068] A third preset number of candidate instance ratios are generated based on a processor count threshold. This third preset number represents multiple ratios; for example, the candidate ratio set is initialized based on the number of GPUs (Z) limited to a single service replica. , Includes the proportion of multiple candidate instances.

[0069] The instance ratio calculation submodule configures the model service based on the candidate instance ratios. It then tests the configured model service using test data and test configuration information. Test data may include, for example, the open-source ShareGPT dataset, a multi-turn dialogue dataset. Test configuration information includes stress test settings such as the number of concurrent requests and duration. The module then tests the configured model service based on the test data and configuration information, for example, by constructing the required test data from the open-source dataset, setting the stress test configuration, and retrieving the ratios from the ratio set. For each i, perform a stress test.

[0070] During testing, test values ​​of the corresponding test metrics for the candidate instance ratio are obtained. For example, during stress testing, service performance metrics are collected to evaluate the performance of each service at that ratio. Here, we use the scaling metrics configured by the service provider as the metrics for measuring service performance. Test values ​​of test metrics include, for example, TPS values ​​and TTFT values.

[0071] Based on the performance of each ratio, the optimal ratio obtained from this P / D ratio calculation is determined, and this ratio is the P / D ratio in each replica of the service.

[0072] The instance ratio calculation submodule determines the difference between the test value corresponding to the candidate instance ratio and the target value of the test metric. The target value is determined based on the type of test metric. For example, if TPS is set as the test metric, a larger value is better, so a larger value, such as 100 or 200, is used as the target value. In this case, the closer the value is to the target value, the larger the test value. If TTFT is set as the test metric, a smaller value is better, so a smaller value, such as 0 or 0.01, is used as the target value. The candidate instance ratio corresponding to the test value with the smallest difference is taken as the instance ratio and transmitted to the service scheduling submodule.

[0073] In this embodiment, the most suitable ratio is determined in advance based on the type of task to avoid performance degradation of the entire model due to the performance bottleneck difference between the P stage and the D stage.

[0074] As an optional embodiment, the service scheduling submodule includes a cluster resource information determination unit; The cluster resource information determination unit is used to obtain the data center's device level, device identifier, and device type. The cluster resource information determination unit is also used to create a cluster resource topology tree based on the device level, device identifier, and device type, wherein the cluster resource topology tree contains cluster resource information.

[0075] Specifically, cluster resource information is used to determine the initialization of the cluster resource topology tree.

[0076] The cluster resource information determination unit traverses the resources of the entire data center to obtain the device level, device identifier, and device type. Among them, the device level includes, for example: cluster level, aggregation level, rack level, physical machine level, and device level; the device identifier includes, for example: aggregation 1, aggregation 2, rack 1, rack 2, machine 1, etc.; and the device type includes, for example: homogeneous node, heterogeneous node, GPU type 1, GPU type 2, etc.

[0077] The cluster resource information determination unit creates a cluster resource topology tree based on device level, device identifier, and device type. For example, it constructs a cluster resource topology tree from the various physical cluster resources of the data center. Figure 3 As shown, the root node of the topology tree of a data center composed of one or more physical clusters is the data center in a logical sense. The intermediate layers include cluster level, aggregation level, rack level, and physical machine level. The leaf nodes are the GPU devices mounted on each physical machine. We call nodes whose last-level leaf nodes all belong to the same GPU type homogeneous nodes, and those that do not belong to the same GPU type heterogeneous nodes. The cluster resource topology tree is as follows: Figure 3 As shown, it will not be elaborated further here.

[0078] In this embodiment, a cluster resource topology tree is constructed by obtaining device level, identifier and type, which structurally integrates data center resources, clearly distinguishes homogeneous and heterogeneous nodes, provides an accurate resource hierarchy view, lays the foundation for service scheduling, and improves scheduling decision efficiency.

[0079] As an optional embodiment, the service scheduling submodule includes a first request processing unit; The first request processing unit is used to determine the target device level corresponding to the request to be processed based on network affinity constraint information and cluster resource topology tree when the request type is the first request type. The first request processing unit is also used to determine whether there is an idle deployment group at the target device level based on the cluster resource topology tree. If there is an idle deployment group, it creates service replicas in the idle deployment group according to the number of service replicas and the instance ratio. The ratio between the number of first-type instances and second-type instances in the service replicas is the instance ratio. The first request processing unit is also used to create a new deployment group at the target device level if no idle deployment group exists, and to create service replicas in the new deployment group according to the number of service replicas and the instance ratio.

[0080] Specifically, the first request type is a service scaling request. If the service adjustment request is a service scaling request, the first request processing unit determines the target device level corresponding to the request to be processed based on network affinity constraint information and the cluster resource topology tree. For example, it searches for the level that meets its network affinity constraints in the cluster resource topology tree from top to bottom, such as the cluster level, aggregation level, rack level, physical machine level, etc. Starting from the target device level, it searches for nodes where instances can be placed.

[0081] The first request processing unit determines whether there is an idle deployment group at the target device level based on the cluster resource topology tree. For example, it traverses all nodes at the target device level from left to right to determine whether there are any tree nodes at the target device level that belong to the deployment group of the model service to which the request belongs. If so, it sorts these tree nodes in order from homogeneous to heterogeneous. After sorting, it determines whether there are any available GPU cards in the leaf nodes of the tree nodes, and the number of available GPUs is greater than or equal to the number of instances of one replica. If such a leaf node exists, it is determined that there is an idle deployment group.

[0082] If an idle deployment group exists, a service replica is created in the idle deployment group based on the number of service replicas and the instance ratio. S = S-1, and the service replica is placed in the corresponding tree node. The specific child node to which it is placed is determined by starting from the current tree node and following a hierarchical order, from homogeneous to heterogeneous, to find the child node and then place the service replica there. The ratio of the number of first-type instances and second-type instances in the service replica is the instance ratio; for example, the P / D instance ratio is 2:1.

[0083] Repeat the above steps until no tree node at that level can be found to belong to the deployment group of the service to which the request belongs, and the two conditions of having enough GPU cards are met; or S equals 0, then the scheduling ends.

[0084] If no idle deployment group exists, the first request processing unit creates a new deployment group at the target device level. For example, it traverses all tree nodes at the target device level from left to right. If there is no deployment group at the target device level, or if an existing deployment group cannot accommodate the remaining S replicas, a new deployment group is created, i.e., a new Deployment is created. Based on the number of service replicas and the instance ratio, service replicas are created in the new deployment group. For example, the tree nodes at this level are sorted from homogeneous to heterogeneous. After sorting, it is determined whether any leaf node of a tree node has an available GPU card, and if the number of available GPUs is greater than or equal to the number of instances of one replica. If so, S = S-1, and a replica is placed in that tree node. The specific child node to which the replica is placed is determined by starting from that tree node and searching for the child node in hierarchical and homogeneous-to-heterogeneous order, and then placing the replica there.

[0085] Then repeat the method of this embodiment until no tree nodes can be found to hold replicas at this level. If S is still not equal to 0, it means that the current cluster resources are very tight and there are no free resources for expansion, then the process ends. Request, process the next request If S equals 0, then the current request scheduling ends, and the next request is processed. .

[0086] In this embodiment, for a service expansion request, the first request processing unit determines the target device level based on network affinity constraints and the cluster resource topology tree, prioritizes utilizing idle deployment groups, and creates new ones if none exist, creating service replicas according to instance ratio and homogeneity priority. This efficiently responds to expansion needs, optimizes resource allocation and utilization, adapts to different service types, ensures service stability, and improves scheduling decision-making and expansion execution efficiency.

[0087] As an optional embodiment, the service scheduling submodule includes a second request processing unit; The second request processing unit is used to determine the deployment group of the model service corresponding to the request to be processed based on network affinity constraint information and cluster resource information when the request type is the second request type. The second request processing unit is also used to release the deployment group of the model service corresponding to the pending request based on the instance ratio, preset order and number of service replicas.

[0088] Specifically, the second request type is a service scaling-down request. When the request type is the second request type, the second request processing unit determines the deployment group of the model service corresponding to the request to be processed, based on network affinity constraint information and cluster resource information. For example, all deployment groups of the service. Resources are released in the order that deployment groups that use heterogeneous resources are prioritized for release.

[0089] The second request processing unit releases the deployment groups of the model services corresponding to the pending requests based on the instance ratio, preset order, and number of service replicas. For example, according to the distribution of deployment groups in the topology tree, it traverses each level from bottom to top. For a tree node at a certain level, it checks whether there is a deployment group under the tree node. If so, it sorts these tree nodes in the order of heterogeneous -> homogeneous, and then traverses the deployment groups belonging to that tree node in order. Based on the instance ratio, it releases the first type of instance and the second type of instance to reduce one replica, and updates S = S - 1. The above operation is repeated until S equals 0, at which point it stops.

[0090] In this embodiment, for scaling down requests, deployment groups are determined based on network affinity and cluster resources. Resources are released in a heterogeneous to homogeneous, bottom-up order, combined with instance ratio and replica count, to accurately reclaim redundant resources, optimize resource allocation, and ensure service stability.

[0091] According to an embodiment of this application, a model service management method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of executable instructions, as described above in the model service management system. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0092] This embodiment provides a model service management method. Figure 4 This is a flowchart of a model service management method according to an embodiment of this application, such as... Figure 4 As shown, the process includes the following steps: Step S401: Manage a first preset number of model services, wherein the model service includes a second preset number of deployment groups, the deployment group includes a first type instance and a second type instance, the first type instance and the second type instance are in a separate state or a coupled state, and the deployment group is located in one or more physical clusters.

[0093] Specifically, the service management module manages a first preset number of model services, where the first preset number and the second preset number represent one or more. Each model service contains a second preset number of deployment groups. Each deployment group contains first-type instances and second-type instances, which can be either separate or coupled. For P / D separated model services, the first-type instance is a P instance, and the second-type instance is a D instance; they reside in different Pods or nodes and are in a separate state. For coupled model services, where the P and D phases reside in the same Pod, the first-type instance can be a Pod in the P phase, and the second-type instance can be a Pod in the D phase; they are in a coupled state. Deployment groups reside in one or more physical clusters, enabling cross-physical cluster deployment of model services.

[0094] Step S402: Obtain the first indicator value of the first instance and the second indicator value of the second instance in the deployment group. Based on the service type of the model service, the first indicator value, and the second indicator value, obtain the current indicator value of the model service.

[0095] Specifically, the metrics monitoring module integrates monitoring metrics for both P / D separated and coupled model services. This module collects metrics statistically derived from the model services, such as TPS, TTFT, and RLS. When there are multiple instances in a deployment group, the metric values ​​are recorded as averages. The metric monitoring module connects to the service management module, obtaining the first metric value from the first instance and the second metric value from the second instance in the deployment group. For P / D separated model services, the weighted sum of the metric values ​​from the P and D instances is used as the overall metric, i.e., the current metric value. For coupled model services, the sum of the metric values ​​from multiple instances is used as the current metric value for the model service.

[0096] Step S403: Generate a service adjustment request based on the current value and expected value of the indicator, and adjust the model service according to the service adjustment request.

[0097] Specifically, the service adjustment module integrates the scaling decision-making functions for both P / D separated and coupled model services. It also incorporates scaling thresholds and cooldown time mechanisms, as well as a method for proportional scaling of P / D type services. For example, it can use the relationship between the current value and the expected value of an indicator to perform scaling up or down processing on the model service, generating service scaling down requests. It can also use the ratio between the current value and the expected value of an indicator to adjust the number of service replicas in the model service, generating service adjustment requests. The expected value can be a single value or a range of values. Service adjustment requests include service scaling down requests and service scaling up requests.

[0098] The model service management method provided in this embodiment manages different types of model services uniformly through deployment groups, supports cross-physical cluster deployment of model services, and can coordinate resources from multiple clusters to improve resource utilization. It uses an indicator monitoring module to collect indicator values ​​calculated by instances to determine the current indicator values ​​of the model services. Based on the current indicator values ​​and expected indicator values, it scales up or down the model services to meet their resource requirements. This solves the problems of difficulty in managing various model services across physical clusters and the inability to scale up or down the model services in accordance with their overall resource requirements.

[0099] As an optional embodiment, the method further includes: Upon receiving the third service parameter for the service to be created, obtain the instance ratio between the first type of instance and the second type of instance; Based on the third service parameters and instance ratio, create a service creation request; Upon receiving the fourth service parameter of the service to be deleted, a service deletion request is created based on the fourth service parameter. Create a service to be created based on a service creation request, or delete a service to be deleted based on a service deletion request.

[0100] Specifically, service providers can create or delete services. When creating a service, the service provider fills in the third service parameters of the service to be created through the front-end UI interface, such as: service name, model name to be used by the service, number of GPUs limited to a single copy of the service, service priority, network affinity constraints of the service, scaling metrics and expected values ​​of the metrics, and service type, etc.

[0101] Upon receiving the third service parameters for the service to be created, the optimal instance ratio between P instances and D instances is obtained. Based on the third service parameters and the instance ratio, a service creation request is generated. The specific process is described in the above embodiment and will not be repeated here.

[0102] When a service needs to be deleted, the service provider fills in the fourth service parameter of the service to be deleted through the front-end UI interface. This parameter includes information such as the service name, the model name used by the service, the network affinity constraints of the service, and the service type. A service deletion request containing the fourth service parameter is then created.

[0103] The process of creating a service to be created based on a service creation request is described in the above embodiment and will not be repeated here. Based on a service deletion request, the service to be deleted is deleted. For example, the service deletion submodule in the service management module performs the service deletion operation, iterating through each physical machine in the physical machine hierarchy to check if there are instances of the model service. If so, these instances are deleted.

[0104] As an optional embodiment, generating a service adjustment request based on the current value and expected value of the indicator includes: Get the current number of replicas of the model service, and get the target value of the metric, the current time, and the historical adjustment time; If the current value of the indicator is of the first indicator type, determine the first ratio between the target value of the indicator and the current value of the indicator, and obtain the target number of replicas based on the first ratio and the current number of replicas. If the current value of the indicator is of the second indicator type, determine the second ratio between the current value of the indicator and the target value of the indicator, and obtain the target number of replicas based on the second ratio and the current number of replicas, where the second ratio is the reciprocal of the first ratio; Determine the time difference between the current time and the historical adjustment time, and determine the ratio of the target number of replicas to the current number of replicas; The quantity ratio is compared with the first preset threshold to obtain the first comparison result, and the time difference is compared with the second preset threshold to obtain the second comparison result. A service adjustment request is generated based on the second service parameter of the model service, the first comparison result, and the second comparison result.

[0105] Specifically, obtain the current number of replicas of the model service. and obtain the target value of the indicator. Current time and historical adjustment time .

[0106] The first type of metric is such as TPS and Decode TPS, where higher values ​​are better. The second type of metric is such as request latency, where lower values ​​are better.

[0107] The target number of copies is calculated using formula (2) or formula (3) above; the calculation process will not be repeated here. The second ratio is... The first ratio is The second ratio is the reciprocal of the first ratio.

[0108] Determine the time difference between the current time and the historical adjustment time. , Utilizing time difference A cooling mechanism is introduced to prevent the system from shaking or oscillating due to frequent expansion and contraction.

[0109] Determine the ratio of the target number of replicas to the current number of replicas. , Quantity ratio This indicates the degree of deviation between the current load and the target load.

[0110] For example, the first preset threshold is: , The second preset threshold is, for example: , The quantity ratio is compared with a first preset threshold to obtain a first comparison result, and the time difference is compared with a second preset threshold to obtain a second comparison result. Based on the above, a service adjustment request is generated.

[0111] As an optional embodiment, the first preset threshold includes a first sub-threshold and a second sub-threshold, and the second preset threshold includes a third sub-threshold and a fourth sub-threshold. A service adjustment request is generated based on the second service parameters of the model service, the first comparison result, and the second comparison result, including: Determine the number of service replicas to be adjusted based on the current number of replicas and the target number of replicas; If the quantity ratio is determined to be greater than the first sub-threshold based on the first comparison result, and the time difference is determined to be greater than the third sub-threshold based on the second comparison result, a service expansion request is created based on the second service parameters and the number of service replicas to be adjusted, wherein the service expansion request is included in the service adjustment request; If the quantity ratio is determined to be less than the second sub-threshold based on the first comparison result, and the time difference is determined to be greater than the fourth sub-threshold based on the second comparison result, a service scaling-down request is created based on the second service parameters and the number of service replicas to be adjusted, wherein the service scaling-up request is included in the service adjustment request.

[0112] Specifically, the first sub-threshold is The second sub-threshold is The third sub-threshold is The fourth sub-threshold is .

[0113] If both conditions are met: ,and If this is not the case, a scaling-up operation is triggered. Based on the second service parameter and the number of service replicas to be adjusted, a service scaling-up request is created. The service scaling-down request is then used to expand the number of replicas to... .

[0114] If both conditions are met: ,and If this is not the case, a scaling-down operation is triggered. Based on the second service parameter and the number of service replicas to be adjusted, a service scaling-down request is created. This service scaling-down request reduces the number of replicas to [a specified value]. .

[0115] In this embodiment, two types of indicators are distinguished, and the target number of replicas is calculated according to the corresponding ratio. Combined with the proportion of load deviation and the cooldown time, a scaling-up or scaling-down request is generated based on a preset threshold. This effectively adapts to different indicator characteristics, avoids system oscillations caused by frequent scaling, accurately matches service load requirements, and ensures service stability.

[0116] As an optional embodiment, adjusting the model service according to a service adjustment request includes: When the service adjustment request is of type 1, determine the target device level corresponding to the service adjustment request and the number of first replicas of the service replicas that need to be adjusted for the service adjustment request, and determine the instance ratio between type 1 instances and type 2 instances. Determine if an idle deployment group corresponding to the service adjustment request exists at the target device level; If an idle deployment group exists at the target device level, create new service replicas in the idle deployment group according to the instance ratio; If no idle deployment group exists at the target device level, create a new deployment group at the target device level, and create new service replicas in the new deployment group according to the instance ratio; Subtract the preset step size from the number of first copies; The subsequent steps begin by determining whether there is an idle deployment group corresponding to the service adjustment request at the target device level, and continue until the number of the first replica is zero, at which point the process ends.

[0117] Specifically, the first request type is a service scaling request. If the service adjustment request is a service scaling request, determine the target device level corresponding to the service adjustment request and the first replica number S of the service replicas that need to be adjusted for the service adjustment request. S is, for example, 2, 3 or other values. Determine the instance ratio, for example, the P / D instance ratio is 2:1.

[0118] Determine if an idle deployment group corresponding to the service adjustment request exists at the target device level. For example, traverse all nodes at the target device level from left to right, and determine if any tree nodes at the target device level belong to the deployment group of the model service to which the request belongs. If so, sort these tree nodes in order from homogeneous to heterogeneous. After sorting, determine if any leaf node of the tree node has an available GPU card, and if the number of available GPUs is greater than or equal to the number of instances of one replica. If such a leaf node exists, it is determined that an idle deployment group exists.

[0119] If an idle deployment group exists, create new service replicas in the idle deployment group according to the instance ratio; if no idle deployment group exists, create a new deployment group at the target device level, and create new service replicas in the new deployment group according to the instance ratio. The process of creating service replicas is described in the above embodiments and will not be repeated here.

[0120] The preset step size is 1. Subtract the preset step size from the number of replicas in the first cluster, i.e., S = S - 1. Repeat the above process until S = 0. Alternatively, if S is still not equal to 0 after multiple repetitions, it indicates that the current cluster resources are very tight and there are no available resources for expansion, then the process ends. Request, process the next request .

[0121] As an optional embodiment, adjusting the model service according to a service adjustment request includes: If the service adjustment request is of type 2, determine the number of second replicas of the service replicas that need to be adjusted for the created deployment group and the service adjustment request. Determine the resource type corresponding to the created deployment group; Based on resource type, preset order, and number of second copies, identify the deployment groups to be released from the created deployment groups and release them.

[0122] Specifically, the second request type is a service scaling-down request. When the request type is the second request type, the deployment group of the model service corresponding to the request to be processed is determined based on network affinity constraint information and cluster resource information. For example, all deployment groups of this service. Resources are released in the order that deployment groups using heterogeneous resources are prioritized for release. The number of second replicas of the service that need to be adjusted in the service adjustment request is S.

[0123] The second request processing unit releases the deployment groups of the model services corresponding to the pending requests based on the instance ratio, preset order, and number of service replicas. For example, according to the distribution of deployment groups in the topology tree, it traverses each level from bottom to top. For a tree node at a certain level, it checks whether there is a deployment group under the tree node. If there is, it determines whether the resource type of the deployment group is heterogeneous or homogeneous. The preset order is from heterogeneous to homogeneous, and these tree nodes are sorted according to the preset order. Then, it traverses the deployment groups belonging to that tree node in order, releasing the first type of instance and the second type of instance according to the instance ratio to reduce one replica, and updates S=S-1. The above operation is repeated until S equals 0, at which point it stops.

[0124] This embodiment also provides a model service management device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0125] This embodiment provides a model service management device, such as Figure 5 As shown, it includes: The model service management module 501 is used to manage a first preset number of model services, wherein the model service includes a second preset number of deployment groups, the deployment group includes a first type instance and a second type instance, the first type instance and the second type instance are in a separate state or a coupled state, and the deployment group is located in one or more physical clusters. The indicator value determination module 502 is used to obtain the first indicator value of the first instance and the second indicator value of the second instance in the deployment group, and to obtain the current indicator value of the model service based on the service type, the first indicator value and the second indicator value of the model service. The model service adjustment module 503 is used to generate a service adjustment request based on the current value and expected value of the indicator, and to adjust the model service according to the service adjustment request.

[0126] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0127] In this embodiment, the model service management device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0128] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0129] The following is a detailed reference. Figure 6 This diagram illustrates a suitable structural design for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 601, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0130] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0131] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a memory 608, or installed from a ROM 602. When the computer program is executed by the processor 601, it performs the functions defined in the model service management method of the embodiments of the present invention.

[0132] Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0133] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described model service management method embodiments at runtime.

[0134] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0135] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described model service management method embodiments.

[0136] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model service management method embodiments.

[0137] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0138] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0139] The model service management system, method, and electronic device provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A model service management system, characterized in that, The system includes: a service management module, a metric monitoring module, and a service adjustment module; The service management module is used to manage a first preset number of model services, wherein the model service includes a second preset number of deployment groups, the deployment group includes a first type of instance and a second type of instance, the first type of instance and the second type of instance are in a separate state or a coupled state, and the deployment group is located in one or more physical clusters; The indicator monitoring module is used to obtain the first indicator value of the first type of instance and the second indicator value of the second type of instance in the deployment group, and to obtain the current indicator value of the model service based on the service type of the model service, the first indicator value and the second indicator value; The service adjustment module is used to obtain the current value of the indicator from the indicator monitoring module, and generate a service adjustment request based on the current value of the indicator and the expected value of the indicator. The service management module is also used to adjust the model service according to the service adjustment request when the service adjustment request is received; The system also includes a front-end interface module; the service management module includes a service creation submodule and a service scheduling submodule. The front-end interface module is used to obtain the first service parameter of the service to be adjusted input through the front-end interface, and transmit the first service parameter to the service creation submodule; the service creation submodule is used to obtain the instance ratio between the first type instance and the second type instance, create other service adjustment requests according to the instance ratio and the first service parameter, and transmit the other service adjustment requests to the service scheduling submodule, so that the service scheduling submodule adjusts the model service according to the other service adjustment requests; The service management module includes a service scheduling submodule; The service scheduling submodule is used to determine the service priority of the model service corresponding to the service adjustment request, and to obtain pending requests from the service adjustment requests according to the service priority; the service scheduling submodule is also used to obtain the request type of the pending request, the network affinity constraint information, instance ratio, and cluster resource information corresponding to the pending request, and to determine the number of service replicas corresponding to the pending request, wherein the instance ratio is the ratio between the number of first type instances and the number of second type instances in the model service corresponding to the pending request; the service scheduling submodule is also used to process the pending request according to the request type, the network affinity constraint information, the instance ratio, the cluster resource information, and the number of service replicas; The service management module includes an instance ratio calculation submodule; The instance ratio calculation submodule is used to determine the processor number threshold and test indicators of the model service, and generate a third preset number of candidate instance ratios based on the processor number threshold; the instance ratio calculation submodule is also used to configure the model service according to the candidate instance ratios, test the configured model service according to test data and test configuration information, and obtain the test values ​​of the test indicators corresponding to the candidate instance ratios during the test; the instance ratio calculation submodule is used to determine the difference between the test value corresponding to the candidate instance ratio and the target value of the test indicator, take the candidate instance ratio corresponding to the test value with the smallest difference as the instance ratio, and transmit the instance ratio to the service scheduling submodule.

2. The system according to claim 1, characterized in that, The service adjustment module includes a data processing submodule, a judgment submodule, and a request generation submodule; The data processing submodule is used to obtain the current number of replicas of the model service, and to obtain the target value of the indicator, the current time, and the historical adjustment time. The data processing submodule is further configured to obtain the target number of replicas based on the ratio between the target value of the indicator and the current value of the indicator and the current number of replicas, determine the time difference between the current time and the historical adjustment time, and determine the ratio between the target number of replicas and the current number of replicas. The judgment submodule is used to compare the quantity ratio with a first preset threshold to obtain a first comparison result, and to compare the time difference with a second preset threshold to obtain a second comparison result. The request generation submodule is used to generate the service adjustment request based on the second service parameters of the model service, the first comparison result, and the second comparison result.

3. The system according to claim 1, characterized in that, The service scheduling submodule includes a cluster resource information determination unit; The cluster resource information determination unit is used to obtain the data center's device level, device identifier, and device type. The cluster resource information determination unit is further configured to create a cluster resource topology tree based on the device level, the device identifier, and the device type, wherein the cluster resource topology tree contains the cluster resource information.

4. The system according to claim 3, characterized in that, The service scheduling submodule includes a first request processing unit; The first request processing unit is configured to, when the request type is a first request type, determine the target device level corresponding to the request to be processed based on the network affinity constraint information and the cluster resource topology tree; The first request processing unit is further configured to determine whether there is an idle deployment group at the target device level based on the cluster resource topology tree; if the idle deployment group exists, to create service replicas in the idle deployment group based on the number of service replicas and the instance ratio, wherein the ratio between the number of first type instances and the number of second type instances in the service replicas is the instance ratio; The first request processing unit is further configured to, if the idle deployment group does not exist, create a new deployment group at the target device level, and create the service replicas in the new deployment group according to the number of service replicas and the instance ratio.

5. The system according to claim 3, characterized in that, The service scheduling submodule includes a second request processing unit; The second request processing unit is configured to, when the request type is the second request type, determine the deployment group of the model service corresponding to the request to be processed based on the network affinity constraint information and the cluster resource information; The second request processing unit is further configured to release the deployment group of the model service corresponding to the pending request based on the instance ratio, preset order and the number of service replicas.

6. A model service management method, characterized in that, The method is applied to the model service management system according to any one of claims 1 to 5, and the method includes: Manage a first preset number of model services, wherein the model services include a second preset number of deployment groups, the deployment groups include a first type of instance and a second type of instance, the first type of instance and the second type of instance are in a separate state or a coupled state, and the deployment groups are located in one or more physical clusters; Obtain the first indicator value of the first type of instance and the second indicator value of the second type of instance in the deployment group, and obtain the current indicator value of the model service based on the service type of the model service, the first indicator value and the second indicator value; A service adjustment request is generated based on the current value and expected value of the indicator, and the model service is adjusted according to the service adjustment request.

7. The method according to claim 6, characterized in that, The method further includes: Upon receiving the third service parameter for the service to be created, obtain the instance ratio between the first type of instance and the second type of instance; Based on the third service parameter and the instance ratio, a service creation request is created. Upon receiving the fourth service parameter of the service to be deleted, a service deletion request is created based on the fourth service parameter. The service to be created is created according to the service creation request, or the service to be deleted is deleted according to the service deletion request.

8. The method according to claim 6, characterized in that, The step of generating a service adjustment request based on the current value and expected value of the indicator includes: Obtain the current number of replicas of the model service, and obtain the target value of the metric, the current time, and the historical adjustment time; If the current value of the indicator is of the first indicator type, determine the first ratio between the target value of the indicator and the current value of the indicator, and obtain the target number of replicas based on the first ratio and the current number of replicas. If the current value of the indicator is of the second indicator type, determine the second ratio between the current value of the indicator and the target value of the indicator, and obtain the target number of replicas based on the second ratio and the current number of replicas, wherein the second ratio is the reciprocal of the first ratio; Determine the time difference between the current time and the historical adjustment time, and determine the ratio of the target number of replicas to the current number of replicas; The quantity ratio is compared with a first preset threshold to obtain a first comparison result, and the time difference is compared with a second preset threshold to obtain a second comparison result. The service adjustment request is generated based on the second service parameters of the model service, the first comparison result, and the second comparison result.

9. The method according to claim 8, characterized in that, The first preset threshold includes a first sub-threshold and a second sub-threshold, the second preset threshold includes a third sub-threshold and a fourth sub-threshold, and generating the service adjustment request based on the second service parameters of the model service, the first comparison result, and the second comparison result includes: Based on the current number of replicas and the target number of replicas, determine the number of service replicas to be adjusted; If the quantity ratio is determined to be greater than the first sub-threshold based on the first comparison result, and the time difference is determined to be greater than the third sub-threshold based on the second comparison result, a service expansion request is created based on the second service parameters and the number of service replicas to be adjusted, wherein the service expansion request is included in the service adjustment request; If, based on the first comparison result, the quantity ratio is determined to be less than the second sub-threshold, and based on the second comparison result, the time difference is determined to be greater than the fourth sub-threshold, a service scaling-down request is created based on the second service parameters and the number of service replicas to be adjusted, wherein the service scaling-down request is included in the service adjustment request.

10. The method according to claim 6, characterized in that, The step of adjusting the model service according to the service adjustment request includes: When the service adjustment request is of the first request type, the target device level corresponding to the service adjustment request and the first number of service replicas that need to be adjusted by the service adjustment request are determined, and the instance ratio between the first type instance and the second type instance is determined. Determine whether an idle deployment group corresponding to the service adjustment request exists at the target device level; If an idle deployment group exists at the target device level, create a new service replica in the idle deployment group according to the instance ratio; If no idle deployment group exists at the target device level, a new deployment group is created at the target device level, and new service replicas are created in the new deployment group according to the instance ratio. Subtract the preset step size from the number of the first replicas; The subsequent steps begin from determining whether there is an idle deployment group corresponding to the service adjustment request at the target device level, and continue until the number of the first replicas is zero, at which point the process ends.

11. The method according to claim 6, characterized in that, The step of adjusting the model service according to the service adjustment request includes: If the service adjustment request is of the second request type, determine the created deployment group corresponding to the service adjustment request and the number of second replicas of the service replicas that need to be adjusted by the service adjustment request; Determine the resource type corresponding to the created deployment group; Based on the resource type, preset order, and the number of second replicas, determine the deployment group to be released from the created deployment group, and release the deployment group to be released.

12. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the model service management method according to any one of claims 6 to 11 by executing the computer instructions.

Images