Model service system and method, server, computing device, and storage medium

By introducing memory pool and prefetcher mechanisms in the model service system, multiple servers allow sharing memory pool to store model parameters, solving the problem of memory resource waste and performance bottlenecks during service of large-size models, and achieving more efficient memory usage and cost reduction.

WO2024235170A1PCT designated stage expired Publication Date: 2024-11-21ALIBABA CLOUD COMPUTING CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/092721
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-16
Filing Date
2024-05-11
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

When existing model service systems deal with large-size machine learning models, they lead to waste of memory resources and performance bottlenecks, especially the problem of device memory waste caused by the replication of model parameters has not been effectively solved.

Method used

Multiple servers share a memory pool device to store model parameters, and use a prefetcher to obtain the required parameters from the memory pool device before or during the execution of the model, avoiding each server from storing model parameters independently, thereby saving memory resources.

Benefits of technology

It realizes the rapid acquisition of model parameters without increasing the memory of a separate server, which reduces memory usage, reduces the cost of model service, and improves the scalability and efficiency of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024092721_21112024_PF_FP_ABST
    Figure CN2024092721_21112024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to a model service system and method, a server, a computing device, and a storage medium. A model service system comprises: a plurality of servers, which are configured to respectively deploy at least one machine learning model so as to provide a service; and a memory pool device, which is configured to store at least part of the parameters of the machine learning model deployed on the plurality of servers; wherein the plurality of servers are configured, prior to respectively executing the machine learning model deployed thereon or while respectively executing the machine learning model deployed thereon, to obtain from the memory pool device at least part of the parameters of the executed machine learning model. Using the solution of the present disclosure allows for a plurality of servers to share a single memory pool to store model parameters, thereby allowing for rapidly obtaining the model parameters from the memory pool on demand, and in certain situations, allowing for avoiding memory capacity limitations in the servers or allowing for saving memory overhead. Moreover, in certain specific embodiments, the present disclosure can likewise reduce the amount of memory used by model servitization.
Need to check novelty before this filing date? Find Prior Art

Description

Model service system and method, server, computing device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 16, 2023, with application number 202310561418.9 and application name “Model service system and method, server, computing device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of machine learning, and in particular to model serving. Background Art

[0003] Model servitization generally refers to a method of deploying machine learning models (such as deep learning models) and providing prediction services (inference services) to users or applications. In some cases, when using models for services, accelerators such as graphics processing units (GPUs) are usually used to speed up the calculation process. Current accelerators generally use high bandwidth memory (HBM), which is an expensive high-performance memory. During the calculation process, this memory is generally used to store parameters and intermediate results. However, as the model size increases, the number of model parameters increases dramatically, resulting in huge memory consumption.

[0004] In addition, in some cases, to allow for more requests per second, multiple identical copies of a single model can be deployed on multiple machines in the model serving system to provide services simultaneously. However, currently, each machine must download and store an identical copy of the model parameters, which wastes a significant amount of device memory resources. This memory waste is particularly prominent in the case of large-scale models.

[0005] Therefore, an improved model service solution is needed.

[0006] Summary of the Invention

[0007] A technical problem to be solved by the present disclosure is to provide an improved model service solution.

[0008] According to a first aspect of the present disclosure, a model service system is provided, comprising: a plurality of servers, configured to respectively deploy at least one machine learning model to provide services; and a memory pool device, configured to store at least part of the parameters of the machine learning models deployed on the plurality of servers; wherein the plurality of servers are configured to obtain at least part of the parameters of the executed machine learning models from the memory pool device before or during the respective execution of the machine learning models deployed thereon.

[0009] Optionally, at least one of the multiple servers is configured to execute the machine learning model deployed thereon in stages, and the at least one server includes a prefetcher; wherein, the prefetcher is configured to pre-acquire parameters required for each stage from the memory pool device before the at least one server starts executing each stage of the machine learning model.

[0010] Optionally, the prefetcher is configured to: determine the execution time of each stage to be executed by the at least one server on the machine learning model, and the transmission time of the parameters required for each stage from the memory pool device to the at least one server, and determine the starting time point for pre-acquiring the parameters required for each stage from the memory pool device based on the execution time and the transmission time.

[0011] Optionally, the prefetcher obtains the execution time and / or the transmission time for each stage by causing the at least one server to test-run the machine learning model or by evaluating the amount of parameters and / or computing power required for each stage of the machine learning model.

[0012] Optionally, the prefetcher is further configured to: insert prefetch instructions for prefetching the parameters required for each stage from the memory pool device into the machine learning model according to the determined starting time point for prefetching the parameters required for each stage.

[0013] Optionally, at least two of the multiple servers are configured to deploy the same machine learning model and share parameters of the same machine learning model stored in the memory pool device.

[0014] Optionally, at least one of the multiple servers includes a controller and an accelerator controlled by the controller; and the model service system also includes an interconnection device, which connects the memory pool device with the controller and / or the accelerator in the at least one server, and is used to transmit parameters from the memory pool device to the controller and / or the accelerator.

[0015] Optionally, the memory pool device includes a memory pool that is read-only for the multiple servers; wherein at least part of the parameters of the machine learning models deployed on the multiple servers are stored in the read-only memory pool.

[0016] According to a second aspect of the present disclosure, a server for providing model services is provided, which is configured to deploy and execute at least one machine learning model in stages to provide services. The server includes: a memory module, configured to store partial parameters of the at least one machine learning model; and a prefetcher, configured to pre-acquire parameters required for each stage from a memory pool device located outside the server into the memory module before the server starts each stage of executing the at least one machine learning model.

[0017] According to the third aspect of the present disclosure, a method for providing model services using a server is provided, wherein the server is a server in the model service system according to the first aspect of the present disclosure or a server according to the second aspect of the present disclosure, and the method includes: receiving a model service request; and executing the machine learning model required by the model service request to provide corresponding services; wherein, before executing the machine learning model or during executing the machine learning model, the server obtains at least part of the parameters of the executed machine learning model from the memory pool device.

[0018] According to a fourth aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method described in the third aspect above.

[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising an executable code. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described in the third aspect above.

[0020] According to a sixth aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the third aspect above.

[0021] Thus, using the solution of this disclosure, multiple servers can share a memory pool to store model parameters, allowing them to quickly retrieve model parameters from the memory pool on demand. In some cases, this can avoid internal server memory capacity limitations or save memory overhead. In addition, in some specific embodiments, this disclosure can also reduce memory usage of the entire system and / or each server when the model is served. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0023] FIG1 and FIG2 respectively show schematic block diagrams of a model service system in the prior art.

[0024] FIG3 shows a schematic block diagram of a model serving system according to an embodiment of the present disclosure.

[0025] FIG4 shows a schematic flowchart of a method for providing model services according to an embodiment of the present disclosure.

[0026] FIG5 shows a schematic diagram of model modification according to an embodiment of the present disclosure.

[0027] FIG6 shows a schematic structural diagram of a computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0029] As mentioned earlier, with the development of technology, the scale of machine learning models (hereinafter referred to as "models") is getting larger and larger, and the model parameters are also increasing; and with the rapid growth of model size, the demand for memory of accelerators when models are served also increases significantly, which leads to the need for more expensive hardware. For example, many artificial intelligence (AI) applications require larger AI models (such as large language models (LLM) such as generative pre-trained transformers (GPT)). This is because larger models can more accurately capture the complexity of the underlying data, thereby obtaining better predictions. For example, existing ultra-large-scale neural network models can have hundreds of millions or even hundreds of billions of parameters. The latest research shows that the model size is increasing exponentially.

[0030] Currently, there are several main strategies for providing large-scale model services:

[0031] 1. Increase hardware resources:

[0032] Increasing hardware resources allows for the deployment of larger and more complex models within a single instance. However, this strategy has very limited scalability and resilience. First, the cost of adding resources can be prohibitive. Second, system resources have a ceiling. As more and more resources are added, the system will reach its ceiling. Furthermore, sometimes, even increasing existing infrastructure is not enough to prevent bottlenecks when the load (e.g., user requests) suddenly increases.

[0033] 2. Distributed services:

[0034] The model can be deployed on multiple servers to form a cluster. For example, as shown in Figure 1, at least one identical copy of a single model can be deployed on each of N servers 111-11N, and the parameters of the single model are stored in the device memory of the accelerator in each server (such as memory 1411 and / or 1412 as shown in Figure 1). For the sake of simplicity and clarity, Figure 1 only shows two accelerators in each server. However, those skilled in the art will understand that the number of accelerators in an actual server may be greater than two.

[0035] As shown in FIG1 , the load balancing device 150 can distribute the currently received multiple model service requests (e.g., calls from users or other devices) as evenly as possible among the N servers for processing based on various strategies, so as to allow more requests to be processed per second. This strategy allows resources to be increased (e.g., the number of servers used) as demand (e.g., user requests) increases. However, this strategy requires all servers to store an identical copy of the model parameters in their memory, which wastes a significant amount of device memory resources. This problem is particularly prominent in the case of large-scale models.

[0036] 3. Model split:

[0037] The model can be split into multiple parts and distributed on multiple different servers (which can be processed in parallel). For example, a model can be split into two parts and deployed on servers 111-1 and 111-2 as shown in Figure 2. Each server instance can run its specific model fragment and then aggregate the results when needed.

[0038] Compared to the second strategy (distributed service strategy) above, this strategy allows the deployment of very large models that cannot be placed on a single server. In addition, it can use more temporary buffers (scratch buffers), thus providing greater throughput, but at the cost of increased latency.

[0039] In addition, as shown in Figure 2, this model splitting strategy can also be combined with the second distributed service strategy described above to handle high request volumes. For example, identical replicas of a single model can be deployed on each of N rows of servers, with each replica being deployed on a separate server in each row (e.g., servers 111-1 and 111-2 in the first row, or servers 112-1 and 112-2 in the second row, etc.) using the model splitting strategy. However, like the second strategy described above, this strategy still suffers from issues such as parameter redundancy leading to wasted memory resources.

[0040] Specifically, existing model serving systems simply replicate model instances across multiple servers. Even with sharded models, each model fragment still needs to be replicated to achieve high throughput. This simplistic replication results in significant waste of device memory. With the exponential growth of AI model parameters, this problem becomes increasingly severe and urgently requires a solution.

[0041] This disclosure proposes a novel model servitization solution that allows multiple servers to share a single memory pool device to store model parameters. This allows for rapid, on-demand retrieval of model parameters from the memory pool, and in some cases, avoids internal server memory limitations or reduces memory overhead. Furthermore, in certain specific embodiments, this disclosure can reduce memory usage across the entire system and / or on individual servers during model servitization.

[0042] Those skilled in the art should understand that although the above discussion is about the problems that arise in the case of large-scale model servitization, the solution of the present disclosure is not limited to the servitization of large-scale models, but is applicable to the servitization of various models, but it can bring more significant effects in the case of large-scale model servitization. For example, in some embodiments, the present disclosure proposes a memory-efficient large-scale model servitization solution for data centers, which aims to solve the problems of resource waste and performance bottlenecks caused by the presence of a large amount of redundant data in the system when the data center provides services for large-scale models. Through the solution of the present disclosure as described in detail below, model parameter sharing can be achieved, thereby effectively solving this problem and reducing the deployment cost of large-scale models.

[0043] In the present disclosure, machine learning models may include deep learning models, which typically use multi-layer neural networks to perform various tasks (such as target monitoring, speech recognition, natural language processing, image recognition, etc.), and thus may include various neural network models, such as various CNN models, etc.

[0044] Model parameters, also known as model weights, are parameters used in machine learning to help the model accurately predict or infer. In some cases, model parameters can be obtained through training or other methods.

[0045] A memory pool generally refers to a collection of memory that can be shared by multiple computers / servers.

[0046] Accelerators can include various general-purpose hardware suitable for model execution (such as training or inference), or hardware specifically designed to accelerate model execution, such as GPUs, deep learning processors (DPUs), tensor processing units (TPUs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc.

[0047] The controller for controlling the accelerator in the server of the present disclosure may include various general-purpose processors, such as a CPU (Central Processing Unit).

[0048] The large-scale models mentioned in this disclosure may include models with a parameter count of more than one million, or even tens of millions, or hundreds of millions.

[0049] The technical solution of the present invention will be described in detail below by taking the deployment of identical copies of a single model on multiple servers as an example, but those skilled in the art should understand that the present invention is not limited thereto.

[0050] For example, in some embodiments, each server in the model service system can deploy different models to provide various corresponding services. Even in the case where the servers cannot share the same model parameters, the solution according to the present disclosure can still bring beneficial effects. For example, it can avoid the inability to deploy the model due to the memory capacity limit inside the server exceeding the number of model parameters, or it can reduce the memory capacity set inside the server to save memory overhead. Each server can quickly obtain model parameters from the memory pool on demand before or during the execution of the model, without having to download all parameters to the memory when deploying the model.

[0051] For example, in some other embodiments, at least two of the multiple servers in the model serving system can deploy the same machine learning model and share the parameters of the same machine learning model stored in a memory pool device. This allows the model parameters to be stored in the memory pool rather than in the memory of each server when deployed, and retrieved from the memory pool into the server's memory only when needed, thereby reducing memory usage.

[0052] For example, in some embodiments, the server can more efficiently utilize its internal memory space. For example, when executing a model, only the parameters required for the current calculation and the parameters required for the next optional stage (e.g., a manually or machine-defined execution stage or a predetermined time period) need to be stored in memory. These model parameters can be readily retrieved from the memory pool on demand. The server's internal memory usage and model execution speed can be weighed against actual needs to determine the method for retrieving parameters from the memory pool, such as when and which parameters to retrieve.

[0053] Those skilled in the art will appreciate that, in the context of the present disclosure, deploying a model refers to loading the model's program instructions and other execution files, and does not necessarily require loading the model's parameters. For example, in some embodiments, the server may not load any model parameters or only load a portion of the parameters when deploying the model.

[0054] At least some embodiments of the solution of the present disclosure will be described in detail below with reference to FIG. 3 to FIG. 5 .

[0055] FIG3 shows a schematic block diagram of a model serving system according to an embodiment of the present disclosure.

[0056] As shown in FIG3 , the model service system may include multiple (eg, N) servers 311 - 31N, which may be configured to respectively deploy at least one machine learning model to provide services.

[0057] In the embodiment shown in FIG3 , the N servers may be deployed with the same model, thereby simultaneously processing multiple service requests for the model. The multiple service requests may be distributed to the servers using a load balancing device 350. However, as previously mentioned, the solution of the present disclosure is not limited thereto. The multiple servers may each deploy one or more models, and may deploy the same or different models, for example, including models that are completely or partially identical to those of other servers.

[0058] In addition, although not shown in FIG3 , in some embodiments, a model can be deployed on multiple servers using a model partitioning scheme similar to FIG2 , so that the model is jointly executed by the multiple servers. In this case, all or some of the multiple servers can be connected to the memory pool device 360 ​​to obtain model parameters therefrom.

[0059] In the embodiment shown in Figure 3, servers 311-31N have similar structures, that is, they each include a controller (such as a CPU) and a plurality of accelerators (such as GPUs) controlled by the controller. Although Figure 3 shows that each server includes two accelerators, those skilled in the art will understand that the number of accelerators in each server is not limited and may be one or more than two. In addition, those skilled in the art will understand that servers 311-31N may have structures that are completely identical, partially identical, or completely different from each other, and the structure of each server is not limited to the structure shown in Figure 3, but as long as it can perform the model service task. In addition, although not shown in Figure 3, in some embodiments, the accelerators within the server may also be directly interconnected to each other, for example, through various known interconnection links.

[0060] In some embodiments, an accelerator in the present disclosure (e.g., accelerator 3311) may include a computational core (not shown) and high-bandwidth memory (e.g., memory 3411). In some cases, accelerators 3311 and 3312 in a server may support a single address space, i.e., all accelerators use the same address space as controller 321 (i.e., host). For example, each accelerator may have a unified memory function.

[0061] In addition, as shown in Figure 3, the model service system may also include a memory pool device 360, which can be configured to store at least part of the parameters of the machine learning model deployed on the aforementioned N servers 311-31N.

[0062] Thus, the aforementioned N servers 311-31N can obtain at least part of the parameters of the executed machine learning models from the memory pool device 360 ​​before or during the execution of the machine learning models deployed thereon.

[0063] In some embodiments, the memory pool device 360 ​​can be a centralized memory pool system that stores some or all parameters of all models deployed by the server. This memory pool system can provide data to each server on an on-demand basis.

[0064] In some implementations, each server may address required parameter data by sending information such as an offset value and a length to the memory pool device 360 ​​.

[0065] Since the parameters of the model are all constants, the memory pool device according to the present disclosure does not need to adopt various similar memory pool technologies such as a distributed shared memory mechanism, thereby supporting the downloading of model parameters from the memory pool to each server with lower system complexity and less latency.

[0066] Traditional memory pool technology is usually more complicated, because it needs to support simultaneous reading and writing of multiple clients and needs to resolve conflicts and other issues. However, since the memory pool is used to provide model parameters to the server in the solution disclosed herein, and the model parameters will basically not change after the model is deployed, there is basically no need to consider the situation where the server writes data to the memory pool. Therefore, in some embodiments, the memory pool device 360 ​​can be implemented using a simplified memory pool, for example, it can include a memory pool 380 that is read-only for multiple servers to store at least part of the parameters of each deployed model. For example, the read-only memory pool can be a memory pool similar to constant memory, which can be read-only for direct access to the server, but is both readable and writable to the host (controller) of the memory pool device. That is to say, in the solution disclosed herein, "a memory pool that is read-only for multiple servers" means that multiple servers cannot directly write to the memory pool, but in some cases can indirectly write to the memory pool through the host of the memory pool device. In some cases, when a model is deployed on a server, the server or other device may write model parameters into the memory pool via the host of the memory pool device 360 ​​, and during subsequent execution of the model, the server directly reads the parameters stored in the memory pool device 360 ​​.

[0067] In some embodiments, as shown in FIG3 , in order to achieve rapid acquisition of parameters, the model service system may further include an interconnection device (e.g., interconnection links 391-39N in the figure), which connects the memory pool device 360 ​​with the controllers 321-32N in each server and is used to transmit parameters from the memory pool device 360 ​​to the controllers 321-32N. The connection method of the interconnection device disclosed in the present invention is not limited to the method shown in FIG3 . In some embodiments, the memory pool device 360 ​​may be connected to the controllers 321-32N in each server and each accelerator 3311, 3312, etc., or only to the memory pool device 360 ​​and each accelerator 3311, 3312, etc. in each server without connecting the controller 321-32N. Depending on the connection method of the interconnection device, the parameters can be transmitted directly to the memory in each accelerator or transmitted to the memory in each accelerator via the controller.

[0068] In some embodiments, the interconnection device of the present disclosure may include a CXL switch having multiple CXL (Compute Express Link) memory expanders, or multiple CXL memory expanders in each server. In some embodiments using a CXL switch, the interconnection device can provide a direct interconnection with each accelerator, which has a bandwidth of PCIe Gen5 x 16. In addition, in this case, data movement conforms to memory semantics, and therefore can support the aforementioned simplified memory pool implementation. This implementation is ideal for data-intensive workloads. Those skilled in the art understand that the interconnection device of the present disclosure is not limited to this, and for example, other possible high-speed interconnection links (including Infiniband (Infinite Bandwidth Technology, abbreviated as IB), NVLink, etc.) can also be used to implement it.

[0069] How to provide model services according to some exemplary solutions of the present disclosure will be described in detail below in conjunction with FIG. 4 and FIG. 3 .

[0070] FIG4 shows a schematic flowchart of a method for providing model services according to an embodiment of the present disclosure.

[0071] As shown in FIG4 , in step S410 , a model service request is received.

[0072] In some embodiments, each server, such as shown in FIG3 , may directly receive the corresponding model service request, or each server may receive the model service request distributed from the load balancing device 350, such as shown in FIG3 . The load balancing device 350 may use various existing distribution schemes to distribute the model service request so as to maximize the balance of workloads among the servers.

[0073] Then, in step S420, the machine learning model required by the model service request is executed to provide the corresponding service, wherein, before or during the execution of the machine learning model, the server obtains at least part of the parameters of the executed machine learning model from a memory pool device located outside the server.

[0074] In some embodiments, the model may be executed by a server that receives the request, such as in FIG3 . Since the server does not store all parameters of the model deployed thereon according to the solution of the present disclosure, the server may obtain the required parameters from the external memory pool device 360 ​​as needed before or during the execution of the model.

[0075] For example, in some embodiments, when the server initially downloads each model, it does not download parameters. During model execution, when the server attempts to read a specific parameter (e.g., a subset of the entire model parameters), a page fault may occur. In this case, the server can handle the interrupt and transfer the required data from the memory pool device 360 ​​to the corresponding device memory. However, this process is time-consuming and can slow down execution. Therefore, to avoid page faults, the present disclosure further proposes a prefetching scheme.

[0076] As shown in the embodiment of Figure 3, a prefetcher can be added to the controller of each server so that before the specific parameters need to be read during the execution of the model, they can be retrieved from the memory pool device 360 ​​in advance to the memory module of the server (for example, the memory including the accelerator or controller). Those skilled in the art will understand that in other embodiments of the present disclosure, some or all of the multiple servers included in the model service system may not have a prefetcher, and the prefetcher can be added to any one or more servers according to actual needs. In addition, although Figure 3 shows that a prefetcher is added to the controller of each server, in other embodiments of the present disclosure, a prefetcher can be added to one or more accelerators. In some embodiments, the prefetcher can be run in the runtime system in the controller or accelerator, which can be the runtime system of any suitable software. The prefetcher can be run by calling or other methods.

[0077] In some embodiments, the model to be executed by the server generally includes at least some operations that require multi-step sequential calculations, wherein the parameters used in each step of the calculation may be different, and thus the server must execute the model in stages. For example, a deep learning model generally includes several layers that perform various processes in sequence, and thus the server generally executes the model in layers in sequence. Therefore, various suitable methods can be used to manually or automatically divide the model into several stages of sequential execution, and the parameters required for each execution stage can be grouped into a subset and acquired together. For example, one or more layers in the deep learning model can be divided into an execution stage, and the parameters required for the one or more layers can be pre-fetched together from the memory pool device 360 ​​as a subset.

[0078] That is, in some embodiments, the server can execute the machine learning model deployed thereon in stages, and the prefetcher in the server can pre-fetch the parameters required for each stage from the memory pool device 360 ​​before the server starts executing each stage of the model into the memory module of the server. The memory module of the server (including various memories in the controllers and accelerators within the server) can only store part of the parameters of the model to be executed, for example, the parameters required for the current execution stage and the parameters required for one or more subsequent stages, without having to store all the parameters of the model.

[0079] In some embodiments, a prefetcher can determine when and which parameters to download to the current server. The prefetcher can retrieve the parameters needed for one or more layers to be executed next so that the data movement time overlaps with the current layer calculation time, thereby not affecting the execution speed or minimizing the impact on the execution speed.

[0080] As mentioned above, the model can be divided into several stages for execution. Therefore, in some embodiments, the prefetcher in each server can perform the following operations:

[0081] Determine the execution time of each stage that the server will perform on the model, and the time it takes to transfer the parameters required for each stage from the memory pool device to the server, and

[0082] A start time point for pre-acquiring parameters required for each stage from the memory pool device is determined according to the determined execution time and transmission time.

[0083] The aforementioned execution time of each stage may include various information such as the execution start time, execution end time, and / or execution duration of each stage. The aforementioned parameter transmission time may include various information such as the time of a single transmission, the average time of multiple transmissions, etc. In some embodiments, based on the determined execution time and transmission time, the time point obtained by subtracting the transmission time of the parameters required for an execution stage from the execution start time of that stage may be used as the latest start time point for pre-acquiring the parameters required for each stage from the memory pool device, thereby ensuring that all parameters required for the execution of that stage are already in the server's memory.

[0084] The following describes how to perform the aforementioned prefetching operation by taking the example of dividing a layer in a deep learning model into a stage.

[0085] For most AI models, the order in which the layers are executed is fixed, and thus the corresponding parameters are used in the same order. For the first execution of the model and subsequent executions, the code being run is essentially unchanged. Therefore, in some embodiments, the prefetcher can obtain the execution time for each stage of model execution and / or the transmission time of the parameters required for each stage from the memory pool device to the server by having the server perform a trial run (warm-up) of the model (without prefetching parameters at this time). In some cases, execution information such as the order and time intervals of access to the model parameters can also be collected through the trial run. Those skilled in the art should understand that "trial run" generally refers to test run operations and the like before正式 providing services. For example, execute the model once in the ordinary manner without prefetching operations, analyze its behavior during execution, so as to determine where to preload which parameter(s) in advance.

[0086] In addition, in other embodiments, instead of using the trial run method, an evaluation method can also be adopted to predict the aforementioned execution time and / or transmission time for each stage. The execution time for each stage and / or the transmission time of the parameters for each stage can be predicted by manual or machine analysis of the model. For example, each layer in the model can be examined and its dimensions, number of channels, and computational complexity, etc. can be considered to predict the time required for each layer to execute. In addition, for example, the number and data size of the parameters used by each layer can be examined to obtain the amount of data to be transmitted for each layer, and the transmission time required for the parameters of each layer can be calculated based on the interconnection transmission speed obtained by means such as statistics.

[0087] After determining the start time point for prefetching the parameters required for each stage, the prefetcher can perform the parameter fetching operation in various ways.

[0088] For example, an active prefetch method can be adopted, such as inserting prefetch instructions for prefetching the parameters required for each stage from the memory pool device into the model according to the determined start time point. Then the modified model is deployed on the server. The model can be modified manually or automatically by machine.

[0089] FIG. 5 shows a schematic diagram of model modification according to an embodiment of the present disclosure. For example, given the original model 500 shown in FIG. 5, the execution start time t(i) for each layer and the parameter set C(i) required for each layer can be first collected through the aforementioned various methods. Then, calculate the prefetch insertion time point I(i) = (t(i) - c(i) / S), where S is the interconnection transmission speed and c(i) is the total data volume of the parameter set C(i). Then, determine the final insertion position of the prefetch instruction as max({j|t(j) < I(i)}), where the obtained j value indicates that the prefetch instruction should be inserted at least before the j-th layer.

[0090] As shown in Figure 5 , the original model 500 on the left is modified based on the insertion positions determined above to produce the model 500' on the right. For example, a statement calling a prefetcher is inserted before the first layer L1 to prefetch the parameter set C2 required by the second layer L2. Since the communication operations involved in prefetching C2 and the computational operations performed by the first layer L1 do not interfere with each other, the two operations can be performed in parallel as shown in Figure 5 . This ensures that the parameter set C2 required by the second layer L2 has been retrieved into the corresponding memory of the server before the execution start time t2 of the second layer L2, thus avoiding delays in the execution of the second layer L2. In the embodiment shown in Figure 5 , the parameter prefetching operation does not affect the model execution speed. Therefore, the execution start time t(i) of each layer in the modified model 500' remains the same as in the original model 500. In addition, although not shown in Figure 5 , the operation of prefetching the parameter set C1 required by the first layer L1 can also be performed in parallel with one or more operations before the execution of the first layer L1, thereby also avoiding delays in the execution of the first layer L1. The advantages of the aforementioned active prefetching include the ability to accurately calculate the insertion point of the prefetch.

[0091] In other implementations, the model may not be modified, but other timing or triggering methods may be used to initiate the operation of retrieving parameters from the memory pool device at or before the determined start time. For example, a prefetch program may be called / started simultaneously when executing a certain operation at each layer, so that parameter prefetching operations for one or more subsequent layers can be performed in parallel.

[0092] Several exemplary solutions for model servitization according to the present disclosure have been described in detail above with reference to the accompanying drawings. These solutions allow multiple servers to share a single memory pool device to store model parameters, enabling rapid on-demand retrieval of model parameters from the memory pool. In some cases, this can also avoid server-side memory capacity limitations or reduce memory overhead.

[0093] In addition, in some specific embodiments, the present disclosure can also reduce the memory usage of the entire system and / or each server when the model is served. For example, the present disclosure introduces a memory pool, which allows all servers to share a single copy of the model parameters. In addition, in some specific embodiments, in order to reduce the impact on performance, the present disclosure further proposes a prefetching scheme. By combining the memory pool and the prefetching scheme, the present disclosure can achieve model parameter sharing and minimize the use of device memory in the server without compromising performance. As a result, especially in the case of providing large-scale model services, the use of device memory in the server can be significantly reduced, and therefore more models can be deployed or more cost-efficient accelerators can be used, thereby reducing the deployment cost of large-scale models.

[0094] FIG6 shows a schematic diagram of the structure of a computing device that can be used to implement the above-mentioned method for providing model services according to an embodiment of the present disclosure.

[0095] 6 , a computing device 600 includes a memory 610 and a processor 620 .

[0096] The processor 620 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 620 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU) or a digital signal processor (DSP). In some embodiments, the processor 620 may be implemented using customized circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0097] The memory 610 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 620 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that retains stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 610 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 610 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0098] The memory 610 stores executable codes. When the executable codes are processed by the processor 620 , the processor 620 can execute the method for providing model services mentioned above.

[0099] The method for providing a model service and the corresponding system or device according to the present disclosure have been described above in detail with reference to the accompanying drawings.

[0100] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0101] In addition, the method according to the present disclosure may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0102] Alternatively, the present disclosure may also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor is caused to perform the various steps of the above-mentioned method according to the present disclosure.

[0103] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

[0104] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems and methods according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A model service system, comprising: A plurality of servers are configured to respectively deploy at least one machine learning model to provide services; as well as a memory pool device configured to store at least part of the parameters of the machine learning model deployed on the plurality of servers; In which, the multiple servers are configured to obtain at least part of the parameters of the executed machine learning models from the memory pool device before or during the execution of the machine learning models deployed thereon respectively.

2. The model service system according to claim 1, wherein: At least one server among the plurality of servers is configured to execute the machine learning model deployed thereon in stages, and the at least one server includes a prefetcher; The pre-fetcher is configured to pre-acquire parameters required for each stage from the memory pool device before the at least one server starts executing each stage of the machine learning model.

3. The model service system according to claim 2, wherein: The prefetcher is configured to: determining the execution time of each stage to be performed by the at least one server on the machine learning model, and the transmission time of the parameters required for each stage from the memory pool device to the at least one server, and A start time point for pre-acquiring parameters required for each stage from the memory pool device is determined according to the execution time and the transmission time.

4. The model service system according to claim 3, wherein: The prefetcher obtains the execution time and / or the transmission time for each stage by causing the at least one server to test-run the machine learning model or by evaluating the amount of parameters and / or the amount of calculation required for each stage of the machine learning model.

5. The model service system according to claim 3, wherein: The prefetcher is also configured to: insert prefetch instructions for prefetching the parameters required for each stage from the memory pool device in the machine learning model according to the determined starting time point for prefetching the parameters required for each stage.

6. The model service system according to claim 1, wherein: At least two of the multiple servers are configured to deploy the same machine learning model and share parameters of the same machine learning model stored in the memory pool device.

7. The model service system according to claim 1, wherein: At least one server of the plurality of servers includes a controller and an accelerator controlled by the controller; and The model service system also includes an interconnection device, which connects the memory pool device with the controller and / or the accelerator in the at least one server and is used to transmit parameters from the memory pool device to the controller and / or the accelerator.

8. The model service system according to claim 1, wherein: The memory pool device includes a memory pool that is read-only for the plurality of servers; Wherein at least part of the parameters of the machine learning models deployed on the multiple servers are stored in the read-only memory pool.

9. A server for providing model services, configured to deploy and execute at least one machine learning model in stages to provide services, the server comprising: A memory module configured to store partial parameters of the at least one machine learning model; as well as A prefetcher is configured to pre-acquire parameters required for each stage from a memory pool device located outside the server into the memory module before the server starts each stage to execute the at least one machine learning model.

10. A method for providing a model service using a server, wherein the server is a server in a model service system according to any one of claims 1 to 8 or a server according to claim 9, the method comprising: Receive model service requests; as well as Execute the machine learning model required by the model service request to provide corresponding services; Wherein, before executing the machine learning model or during executing the machine learning model, the server obtains at least part of the parameters of the executed machine learning model from the memory pool device.

11. A computing device comprising: processor; as well as A memory having executable codes stored therein, which, when executed by the processor, causes the processor to execute the method according to claim 10. 12 . A computer program product, comprising executable codes, which, when executed by a processor of an electronic device, cause the processor to perform the method according to claim 10 . 13 . A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to claim 10 .

Citation Information

Patent Citations

  • Model parameter storage method and system and parameter server cluster

    CN112306682A

  • Model parameter calling method, device and equipment and readable storage medium

    CN115222031A

  • Memory management method and device for deep learning model, equipment and storage medium

    CN115794400A

  • Model service system and method, server, computing device and storage medium

    CN116860469A

  • Model parameter sharing between inference application instances in processing unit of information processing system

    US20220318656A1