High-availability model scheduling deployment method based on resource reallocation

Abstract

The invention discloses a high-availability model scheduling and deploying method based on resource redistribution, which comprises the following steps of: performing resource distribution and self-adaptive capacity expansion and contraction on a recommendation model through an online scheduling system to complete scheduling and deploying of the model; the method is a self-adaptive capacity expansion and contraction method, automatic deployment of the model can be realized, and operation and maintenance manpower is liberated; in order to improve the high availability of services in the experiment change process by shortening the resource redistribution time interval, the scheme adopts a Boolean belt strategy to dynamically adjust model service resources, and the real-time performance, the availability rate and the resource utilization rate can be considered at the same time; according to the scheme, one-time resource adjustment can be completed in dozens of seconds on average, and the daily availability rate of model services can reach 99.99%.

Description

Technical Field

[0001] The present invention relates to the technical field of recommendation systems, and in particular to a high-availability model scheduling and deployment method based on resource reallocation. Background Art

[0002] The core processes of the advertising / information flow recommendation system include recall, rough ranking, fine ranking, and mixed ranking strategies. Recall, rough ranking, and fine ranking are model-driven, and their models are deployed on the inference service. Therefore, we can conclude that the inference service is the cornerstone of the recommendation system. Generally speaking, the resource demand of the model changes dynamically, because the model request volume changes frequently with the changes in the A / B Test experiment.

[0003] The commonly used model scheduling and deployment methods in the industry are:

[0004] Pre-manual capacity expansion is a model in which the operation and maintenance department manually expands the capacity before the experimental changes. When using this method, not only is it necessary to reserve a spare resource pool, but the resource utilization rate is low, and it is also inefficient and error-prone. It is not suitable for recommendation scenarios with frequent iterations.

[0005] All-to-all hybrid deployment, that is, all models are deployed on all devices. The advantages of this method are: simple, no need to adjust the deployment when traffic changes, but the disadvantages are that the source is not isolated and the performance is poor (high scheduling cost when there are many processes, multiple models share memory resulting in small cache, low cache hit rate, etc.); the memory / video memory usage is high, and too many models cannot be deployed; especially for GPU inference, since the video memory is much smaller than the memory, this strategy is no longer effective, so a new strategy must be developed; taking the 16G video memory of a single GPU T4 as an example, it is much smaller than the memory of the CPU model. In addition, because the GPU uses multi-stream technology to improve its utilization rate, more video memory is required to load a single model.

[0006] Automatic elastic expansion and contraction, that is, deploying one model on each device, maintaining an idle resource pool, passively expanding and contracting the model according to the request volume and load, and triggering model resource contraction by low load, that is, passively expanding and contracting the model according to the request volume and load; the disadvantages of this method are: it is necessary to maintain an idle resource pool. If the resource pool is too large, the resource utilization rate is low; if the resource pool is too small, the change range of the experimental traffic will be limited; the traffic will increase while the capacity is expanded. During the experimental release process, the model with increased traffic will be unavailable due to overload. The expansion and contraction speed is slow, and the docker expansion and contraction takes 10 minutes. The expansion and contraction speed cannot keep up with the speed of experimental changes, and there is a risk of avalanche.

[0007] All of the above methods have obvious shortcomings and deficiencies, which make them difficult to use in actual production environments. Therefore, a highly available model scheduling and deployment solution is urgently needed to solve the above technical problems. Summary of the invention

[0008] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0009] Therefore, in order to solve the above technical problems, the present invention provides the following technical solutions: a highly available model scheduling and deployment method based on resource reallocation, which dynamically allocates resources according to the request volume of the model through an online scheduling system, adaptively expands and shrinks capacity, and completes the automated deployment of model services.

[0010] As a preferred solution of the high-availability model scheduling and deployment method based on resource reallocation described in the present invention, when the model traffic changes, the resources are reallocated through the scheduling system, and the model deployment plan is generated according to the model traffic size, and then the route is issued through the routing table, and the corresponding model service is automatically pulled up.

[0011] As a preferred solution of the high-availability model scheduling and deployment method based on resource reallocation described in the present invention, the method shortens the resource reallocation time interval so that the model request volume and model instance number curves basically overlap, thereby ensuring high availability during experimental changes and avoiding service overload.

[0012] As a preferred solution of the high-availability model scheduling and deployment method based on resource reallocation described in the present invention, the method of shortening the resource reallocation time interval includes: using the Bollinger Band strategy to obtain accurate statistical results of the model request volume, and then dynamically adjusting the model service resources, while taking into account real-time, availability and resource utilization.

[0013] As a preferred solution of the high-availability model scheduling and deployment method based on resource reallocation described in the present invention, the statistical results of the request volume distribution are calculated using the Bollinger Band strategy, and the specific algorithm is as follows:

[0014] Collect the model request volume in the last N seconds and calculate E and stddev

[0015] Try to estimate the upper bound of the request volume upper_bound

[0016] Default upper_bound = E + stddev

[0017] When the number of requests in the last second is greater than E+stddev, the model is scaled up and the upper bound of the estimate is enlarged (expanding quickly).

[0018] When the number of requests in the last second is less than E-stddev, the model is scaled down, and the estimated upper bound is shrunk to E (slowly).

[0019] upper_bound=E(if last_rate <E-stddev)

[0020] upper_bound=E+stddev(if E-stddev <last_rate<E+stddev)

[0021] upper_bound=E+k*stddev(if last_rate>E+stddev).

[0022] As a preferred solution of the high-availability model scheduling and deployment method based on resource reallocation described in the present invention, in which: in the above-mentioned calculation method, the method of obtaining better k and N hyperparameters includes establishing an offline data set and grid search.

[0023] As a preferred solution of the high-availability model scheduling and deployment method based on resource reallocation described in the present invention, the method prioritizes the availability of the expansion model when adjusting the deployment by overestimating the call ratio of the expansion model.

[0024] As a preferred solution of the high-availability model scheduling and deployment method based on resource reallocation described in the present invention, the average completion time of a resource adjustment is 10s-20s.

[0025] As a preferred solution of the high-availability model scheduling and deployment method based on resource reallocation described in the present invention, the daily availability rate of the model service is 99.99%.

[0026] As a preferred solution of the high-availability model scheduling and deployment method based on resource reallocation described in the present invention, the method is applicable to GPU inference services and CPU inference services.

[0027] Beneficial effects of the present invention:

[0028] The adaptive scaling method proposed in the present invention can realize automatic model deployment and liberate operation and maintenance manpower; in order to improve the high availability of services during experimental changes by shortening the resource reallocation time interval, this solution adopts the Bollinger Band strategy to dynamically adjust the model service resources, which can take into account real-time performance, availability and resource utilization at the same time; this solution only takes a dozen seconds to complete an average resource adjustment, and the daily availability of the model service can reach 99.99%. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0030] Figure 1 It is a schematic diagram of the adaptive expansion and contraction structure in resource reallocation of the present invention.

[0031] Figure 2 This is an architecture diagram of the online model scheduling system of the present invention.

[0032] Figure 3 This is a flow chart of reallocating resources according to model traffic in the present invention.

[0033] Figure 4 This is a graph showing the relationship between the number of model instances and the amount of requests during experimental changes in the prior art of the present invention.

[0034] Figure 5 This is a graph showing the relationship between the number of model instances and the amount of requests during the experimental changes of the present invention.

[0035] Figure 6 This is a schematic diagram of the statistical results of a certain model request amount of the present invention. DETAILED DESCRIPTION

[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0037] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0038] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0039] Reference Figures 1 to 3 , 5 to 6, are embodiments of the present invention, which provide a highly available model scheduling and deployment method based on resource reallocation, and recommend that the model be used to allocate resources through an online scheduling system, adaptively scale up and down, and complete the scheduling and deployment of the model; that is, adaptively scale up and down, liberating operation and maintenance manpower (such as Figure 1 shown);

[0040] The core challenges of high-availability model scheduling and deployment methods are:

[0041] (1) High real-time performance: Machine deployment needs to be adjusted in real time based on traffic changes;

[0042] (2) High availability: When the traffic is required to be stable or changing, the system availability remains stable;

[0043] (3) High resource utilization: The additional resource overhead required to balance real-time performance and availability needs to be minimized.

[0044] In this solution, when the model traffic changes, resources are reallocated through the scheduling system, a model deployment plan is generated based on the model traffic size, and then the route is issued through the routing table to automatically start the corresponding model service.

[0045] For experimental changes, model instances often lag behind the increase in request volume, which can easily cause service overload. In this case, this solution shortens the resource reallocation time interval so that the model request volume and model instance number curves basically overlap (such as Figure 5 In order to shorten the time interval for resource reallocation, we need to obtain accurate statistical results of model request volume (such as Figure 6 shown).

[0046] The statistical results of the request volume distribution are distorted, as follows:

[0047] (1) Small window statistics (second level): There is jitter and the statistics are inaccurate;

[0048] (2) Large window statistics (minute level): As the number of requests increases, the statistical mean lags behind the latest value, which will underestimate the number of requests and lead to insufficient capacity and overload.

[0049] In view of the problems existing in the above statistical results, this solution adopts the Bollinger Band strategy to solve them. The specific algorithm is as follows:

[0050] Collect the model request volume in the last N seconds and calculate E and stddev

[0051] Try to estimate the upper bound of the request volume upper_bound

[0052] Default upper_bound = E + stddev

[0053] When the number of requests in the last second is greater than E+stddev, the model is scaled up and the upper bound of the estimate is enlarged (expanding quickly).

[0054] When the number of requests in the last second is less than E-stddev, the model is scaled down, and the estimated upper bound is shrunk to E (slowly).

[0055] upper_bound=E(if last_rate <E-stddev)

[0056] upper_bound=E+stddev(if E-stddev <last_rate<E+stddev)

[0057] upper_bound=E+k*stddev(if last_rate>E+stddev)

[0058] Better k and N hyper-parameters can be obtained by establishing an offline data set and performing grid search.

[0059] In summary, this solution implements a highly available model automation deployment solution during experimental changes, namely, an adaptive scaling method, which can realize automatic model deployment and liberate operation and maintenance manpower; in order to improve the high availability of services during experimental changes by shortening the resource reallocation time interval, this solution adopts the Bollinger Band strategy to dynamically adjust model service resources, which can take into account real-time performance, availability and resource utilization at the same time; this solution only takes a dozen seconds to complete an average resource adjustment, and the daily availability of model services can reach 99.99%.

[0060] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A highly available model scheduling and deployment method based on resource reallocation, characterized in that: Through the online scheduling system, resources are dynamically allocated according to the number of model requests, and capacity is adaptively expanded and reduced to complete the automated deployment of model services.

2. The high-availability model scheduling and deployment method based on resource reallocation according to claim 1, characterized in that: When the model traffic changes, resources are reallocated through the scheduling system, and a model deployment plan is generated based on the model traffic size. The route is then issued through the routing table, and the corresponding model service is automatically started.

3. The high-availability model scheduling and deployment method based on resource reallocation according to claim 1, characterized in that: This method shortens the resource reallocation time interval so that the model request volume and model instance number curves basically overlap, ensuring high availability during experimental changes and avoiding service overload.

4. The high-availability model scheduling and deployment method based on resource reallocation according to claim 1, characterized in that: Ways to shorten the resource reallocation time interval include: using the Bollinger Band strategy to obtain accurate statistical results of model request volume, and then dynamically adjusting model service resources while taking into account real-time performance, availability, and resource utilization.

5. The high-availability model scheduling and deployment method based on resource reallocation according to claim 1, characterized in that: The statistical results of the request volume distribution are calculated using the Bollinger Band strategy. The specific algorithm is as follows: Collect the model request volume in the last N seconds and calculate E and stddev Try to estimate the upper bound of the request volume upper_bound Default upper_bound = E + stddev When the number of requests in the last second is greater than E+stddev, the model is expanded and the upper bound of the estimate is enlarged. When the number of requests in the last second is less than E-stddev, the model is scaled down and the upper limit of the estimate is reduced to E. upper_bound=E(if last_rate <E-stddev) upper_bound=E+stddev(if E-stddev <last_rate<E+stddev) upper_bound=E+k*stddev(if last_rate>E+stddev).

6. The high-availability model scheduling and deployment method based on resource reallocation according to claim 1, characterized in that: In the above calculation method, ways to obtain better k and N hyperparameters include establishing an offline data set and grid search.

7. The high-availability model scheduling and deployment method based on resource reallocation according to claim 1, characterized in that: This method prioritizes the availability of the expansion model when adjusting the deployment by overestimating the call ratio of the expansion model.

8. The high-availability model scheduling and deployment method based on resource reallocation according to claim 1, characterized in that: The average time it takes to complete a resource adjustment is 10s-20s.

9. The high-availability model scheduling and deployment method based on resource reallocation according to claim 1, characterized in that: The daily availability of the model service is 99.99%.

10. The high-availability model scheduling and deployment method based on resource reallocation according to claim 1, characterized in that: This method is applicable to GPU inference services and CPU inference services.

Images