A cloud multi-pre-training language model management and inference method and electronic equipment
By managing and scheduling pipeline processing using virtualized pre-trained neural network models (vBERT), the resource constraints of cloud platforms in multi-model deployments are solved, achieving efficient model management and inference, and improving the performance and throughput of cloud services.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-11
- Publication Date
- 2026-04-07
AI Technical Summary
When deploying and processing multiple pre-trained language models, cloud platforms face challenges such as limited GPU memory, insufficient transmission bandwidth, high computational overhead, and reduced model accuracy. Traditional compression methods cannot effectively address these challenges.
A virtualized pre-trained neural network model (vBert) is used. Requests are received through a configuration dispatcher, a shallow feature lookup table is built, and requests are scheduled in a pipeline manner to achieve model management and inference. Model instances are stored on the hard disk using an adapter to reduce GPU memory usage and perform efficient computation on the GPU.
It increases throughput on a single graphics card, reduces memory usage and CPU/IO overhead, improves inference performance, and provides customized, high-efficiency services.
Smart Images
Figure CN116737864B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a model virtualization method in the field of artificial intelligence, and more particularly to a cloud-based method for managing and inferring multiple pre-trained language models, and an electronic device. Background Technology
[0002] Deep learning is increasingly being integrated into cloud services. Recently, a series of neural network models known as pre-trained language models have become popular in the field of natural language processing. These models are pre-trained on large amounts of data, enabling them to learn language features and achieve good results on various downstream tasks, especially those with data distributions similar to the pre-training data. This pre-training paradigm is sometimes referred to as "homogenization." The sharing of pre-trained language models across a range of downstream tasks is driving the trend in building natural language processing and other artificial intelligence systems. This sharing of pre-trained language models is particularly well-suited to the characteristics of cloud computing. Cloud platforms can offer their pre-trained language models as cloud resources to tenants. Tenants can then perform inference processes at a lower cost by sharing the models and hardware resources.
[0003] However, pre-trained language models are not a panacea, as they may not perform well for specific domains or highly customized tasks. For example, an online retail store might not achieve high accuracy with a general-purpose pre-trained language model and would need further training based on data from different retail sectors. Furthermore, each merchant on the platform might need to further train their model based on their own retail sector data to provide more precise services to customers, such as question-answering chatbots and user feedback analysis. This further training approach can ultimately result in a large number of different model copies. Although further training can be performed in the background during off-peak hours, it presents new challenges for cloud platform operators: First, GPU memory is limited, making it worthwhile for cloud service providers to deploy a large number of models on limited hardware resources, as a customized model can be hundreds of megabytes in size. This means that only a few models can be deployed on a single GPU at any given time, while other models must be stored in memory or SSD and loaded onto the GPU for computation through model replacement. This approach, in turn, introduces other challenges:
[0004] (1) Limited transmission bandwidth. Since the pre-trained language model itself occupies a large amount of storage space, the overhead of model replacement becomes a performance bottleneck, which seriously affects the efficiency of the system.
[0005] (2) GPU computational overhead. Pre-trained language models are computationally intensive. Running multiple models simultaneously on a single GPU results in lower throughput and higher latency.
[0006] (3) Reduced model accuracy. Traditional lossy model compression methods, including model distillation, quantization, and pruning, cannot solve the problem of model accuracy because these methods lose a lot of accuracy and cannot meet the requirements of cloud services. More importantly, although these methods reduce the size of the model compared to the original model, they still cannot solve the problem of processing a large number of different models on the same GPU. Summary of the Invention
[0007] To address the shortcomings of existing technologies, this invention proposes a cloud-based multi-pre-trained language model management and inference method, as well as an electronic device.
[0008] According to a first aspect of the present invention, a cloud-based multi-pretrained language model management and inference method is provided, characterized in that the method includes:
[0009] Step S1: Receive model management requests and inference requests from tenants through the configuration dispatcher;
[0010] Specifically, a model management request is a request initiated by a tenant to change the content and structure of the vBert model instance tree.
[0011] Step S2: Build and maintain a shallow feature lookup table through the configuration manager, and update the vBert model instance tree;
[0012] Step S3: Configure the scheduler to schedule and process model management requests and inference requests in a pipeline manner.
[0013] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described cloud-based multi-pre-trained language model management and inference method.
[0014] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described cloud-based multi-pre-trained language model management and inference method.
[0015] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a cloud-based multi-pre-trained language model management and inference method, which provides customized services to a large number of tenants through virtualized pre-trained neural networks on a single graphics card; This invention stores the calculation results of shallow transformers in the shallow feature lookup table of the pre-trained neural network, thus achieving higher throughput; At the same time, this invention stores a large number of tenant model adapters in host memory to reduce video memory usage; and Pipeline processing can reduce additional CPU and I / O overhead, thereby improving inference performance. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 This is a flowchart of the method of the present invention;
[0018] Figure 2 This is a flowchart illustrating the process of processing inference requests using the method of the present invention;
[0019] Figure 3 This is a flowchart illustrating the process of handling model management requests using the method of this invention;
[0020] Figure 4 This is an example diagram of a virtualized pre-trained language model instance tree;
[0021] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0023] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.
[0024] like Figure 1 As shown, this invention proposes a cloud-based multi-pretrained language model management and inference method, which specifically includes the following steps:
[0025] Step S1: The dispatcher receives model management requests and inference requests from tenants.
[0026] Among them, a model management request refers to a tenant initiating a request to change the content and structure of the vBert model instance tree.
[0027] The vBert model instance tree is a three-layer tree structure. The first layer contains a general pre-trained neural network model with several transformer layers; the second layer contains domain model instances for several domains, including but not limited to news, biology, and finance; the third layer contains dedicated tenant model instances. Model instances in the vBert model instance tree are stored on disk as adapters, each consisting of two forward fully connected layers and one skip connection layer. Multiple versions of model instances can exist for the same domain or the same tenant.
[0028] Specifically, the second layer of the vBert model instance tree contains model instances from different domains, as well as different versions of model instances within the same domain. It should be noted that each domain model instance consists of several versions; the initial version is obtained by further training based on the general pre-trained neural network in the first layer. Other versions within the same domain are obtained by further training vBert based on the initial version for that domain.
[0029] The third level of the vBert model instance tree contains model instances from different tenants, as well as different versions of model instances within the same tenant. Each model instance contains several versions. The initial version is obtained by further training based on the domain model instance, while the remaining versions are obtained by further training based on the initial version.
[0030] The model instance is an adapter containing two fully connected layers and one skip connected layer. The second and third layers are actually stored on disk as the adapter.
[0031] The adapter is a neural network module containing two forward fully connected layers and one skip connection layer, mathematically expressed as: Linear(·) is a fully connected layer. It uses the ReLU activation function. An adapter module injects task-specific features into the transformer, and each transformer layer contains two adapter modules. The two adapters are located after the multi-head attention layer and the fully connected layer inside the transformer layer, respectively. The neural network model composed of the transformer and adapters can then be considered a pre-trained language model after further training. vBert further trains the adapters as follows: vBert starts from the tenant-specified domain branch, selects a model instance in the chosen domain, and randomly initializes the parameters of the tenant-specific adapter. During further training, vBert keeps the parameters of the pre-trained neural network backbone layers unchanged, only updating the parameters of the adapters and the output layer. After training, vBert inserts the adapter into the third layer of the model instance tree and saves it to disk. vBert uses adapters to train models efficiently. Cloud service providers can replace only a few megabytes of adapters without replacing the large pre-trained language model itself, thus providing accurate and efficient services to each tenant.
[0032] The model management requests issued by the tenants include the following four types:
[0033] (1) Create a new domain model instance.
[0034] The virtualized Bert (vBert) pre-trained language model takes the text data uploaded by the tenant as input into the general pre-trained language model, further trains the general pre-trained language model, and uses the training result as a brand new domain model instance, inserting it into the second layer of the vBert model instance tree.
[0035] (2) Update existing domain model instances.
[0036] The tenant selects an existing domain model instance for a specific domain from the second level of the vBert model instance tree and uploads new text data. vBert inputs the text data into the domain model instance and trains it to obtain a new domain model instance. vBert inserts the trained new domain model instance into the second level of the vBert model instance tree, making it a completely new version for a specific domain.
[0037] (3) Create a new tenant model instance.
[0038] The tenant uploads text data and selects a domain model instance from the second level of the vBert model instance tree. vBert then trains the corresponding domain model instance by inputting the tenant's text data into it and inserts the final result into the third level of the vBert model instance tree.
[0039] (4) Update the existing tenant model instance.
[0040] Tenants upload new text data and select their own exclusive model instance from the third level of the vBert model instance tree. vBert uses the text data uploaded by the tenant as a training dataset to further train the tenant's model and inserts the final result as a brand new version of the tenant's model instance into the third level of the model instance tree.
[0041] The inference request includes the tenant ID, domain ID, and text to be processed.
[0042] Furthermore, vBert combines the tenant ID and domain ID to locate the tenant model instance. The dispatcher sends inference requests issued by tenants to the scheduler for processing in batches.
[0043] It's important to note that vBert uses a request queue to receive inference requests from tenants. The dispatcher retrieves multiple inference requests from the same tenant from the request queue each time, packages them, and sends them to the scheduler for processing. The request queue is a container used by vBert to temporarily store all tenants' inference requests. After the scheduler finishes its calculations, it sends the results to the results queue, and then the request dispatcher returns the contents of the results queue to the respective tenants. The results queue is a container used by vBert to temporarily store the scheduler's calculation results.
[0044] Step S2: Build and maintain a shallow feature lookup table through the manager, and update the vBert model instance tree;
[0045] The shallow feature lookup table is a three-layer tree structure. The basic entries in the shallow feature lookup table are key-value pairs. The key is the encoded triple, and the value is the intermediate computation result obtained by the shallow transformer in calculating the triple. The intermediate computation result refers to the intermediate state obtained after the input text has passed through the first few layers of the transformer model.
[0046] The first layer of the tree stores the intermediate computation results of the shallow transformers of the general pre-trained neural network model. The second layer stores the intermediate computation results of the shallow transformers for all domain model instances, where the domains refer to biology, finance, news, etc. The third layer stores the intermediate computation results of the shallow transformers for tenant model instances. The shallow transformer intermediate computation results refer to the intermediate results obtained after the input text has been processed by the first few transformer layers of the model.
[0047] Shallow feature lookup tables are stored incrementally, layer by layer, from the first to the third level of the tree. For example, a shallow feature lookup table exclusive to a third-level tenant only stores key-value pairs not present in the second-level, domain-specific shallow feature lookup table. The most basic entries in a vBert shallow feature lookup table are key-value pairs. Similarly, entries in a virtualized pre-trained neural network lookup table are also key-value pairs, where the key is an encoded triple and the value is an intermediate computation result obtained by the shallow transformer from calculating the triple.
[0048] In step S3, the scheduler uses a pipelined approach to schedule and process inference requests and model management requests.
[0049] For inference requests, the manager searches for the corresponding shallow features and adapters in the shallow feature lookup table and vBERT model instance tree based on the tenant ID, domain ID, and statement in the inference request. The scheduler then transmits the lookup results to the GPU for computation.
[0050] The scheduler uses a pipelined approach to schedule and process inference requests, including:
[0051] Step S301: Search the shallow feature lookup table of the virtualized pre-trained neural network;
[0052] Step S302: Locate tenant model instances;
[0053] Step S303: GPU performs computation to obtain the inference request result, and then feeds the inference request result back to the tenant.
[0054] It should be noted that vBert does not need to wait for all three steps mentioned above to be completed before processing the next tenant's inference request. It can process tenant inference requests in parallel. For example, after completing the shallow feature lookup table search, it can immediately proceed to the first step of processing the next tenant's inference request, thus improving the utilization of hardware resources.
[0055] Furthermore, GPU computation includes the scheduler pipelined replacement of tenant model instances from the GPU. The computation time of the Transformer layers of a general-purpose pre-trained neural network model on the GPU is longer than the transfer time of tenant model instances between the CPU and GPU. Therefore, vBert can send the adapters needed for the next batch of tenants to the GPU while the GPU computation requests the allocator to forward the previous batch of data.
[0056] The adapter transmission latency is non-negligible, and its pipelined transmission method can be divided into two types:
[0057] (1) Coarse-grained adapter pipeline transmission.
[0058] vBert transfers the next batch of required adapters to the GPU while the GPU is computing the previous batch of requests, thus masking the adapter transfer time by allowing the GPU to compute the previous batch of tenant requests.
[0059] (2) Fine-grained adapter pipeline transmission.
[0060] The computation time of a single transformer layer in the pre-trained neural network backbone model on the GPU is greater than the transmission latency of the adapter; therefore, the transmission latency of the adapter can be hidden within the GPU's computation time. vBert transmits the next batch of required adapters to the GPU during the computation of the previous transformer layer.
[0061] The model management request consists of three parts: tenant ID, domain ID, and the text file (.txt file) to be processed. The vBert manager locates the domain adapter in the vBert model instance tree based on the domain ID. The vBert scheduler transfers the domain adapter to the GPU and inputs the text file uploaded by the tenant into the vBert model for training. After training, vBert inserts the new adapter as a completely new version of the tenant model instance into the third level of the vBert model instance tree based on the tenant ID.
[0062] Furthermore, the difference between model management requests and inference requests is that management requests insert the processing result into the vBert model instance tree after processing, while inference requests return the processing result to the tenant.
[0063] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the cloud-based multi-pre-trained language model management and inference method as described above. Figure 5 The diagram shown is a hardware structure diagram of any device with data processing capabilities where the cloud-based multi-pre-trained language model management and inference method provided in this embodiment of the invention is located. (Except for...) Figure 5 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.
[0064] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the cloud-based multi-pre-trained language model management and inference method described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.
[0065] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.
Claims
1. A cloud-based method for managing and inferring multiple pre-trained language models, characterized in that, The method includes: Step S1: Receive model management requests and inference requests from tenants through the configuration dispatcher; Specifically, the model management request is initiated by a tenant to change the content and structure of the vBert model instance tree. The vBert model instance tree is a three-layer tree structure. The first layer of the vBert model instance tree contains a general pre-trained neural network model with several transformer layers. The second layer contains several domain model instances. The third layer contains several tenant model instances. The domain model instances and tenant model instances are adapters containing two fully connected layers and one skip connection layer. Step S2: Construct and maintain a shallow feature lookup table through the configuration manager, and update the vBert model instance tree; wherein, the shallow feature lookup table is a three-layer tree structure; wherein, the basic entries of the shallow feature lookup table are key-value pairs, where the key is an encoded triple and the value is the intermediate calculation result obtained by the shallow transformer to calculate the triple, and the intermediate calculation result refers to the intermediate state obtained after the input text is calculated by the shallow transformer; the first layer of the tree stores the intermediate calculation results of the shallow transformer of the general pre-trained neural network model; the second layer of the tree stores the intermediate calculation results of the shallow transformer of all domain model instances; and the third layer of the tree stores the intermediate calculation results of the shallow transformer of all tenant model instances. Step S3: Configure the scheduler to schedule and process model management requests and inference requests in a pipeline manner; The scheduler uses a pipelined approach to schedule and process inference requests, including: The manager searches for the text to be processed in the shallow feature lookup table and the vBert model instance tree based on the tenant ID, domain ID and text to be processed in the inference request. The scheduler transmits the search results to the GPU for computation to obtain the inference request result and sends the inference request result to the tenant. The scheduler uses a pipelined approach to schedule and process model management requests, including: The manager finds the domain adapter in the vBert model instance tree based on the domain ID. The scheduler transfers the domain adapter to the GPU and inputs the text to be processed uploaded by the tenant into the vBert model for training. After training, the vBert model inserts the new adapter as a new version of the tenant model instance into the third level of the vBert model instance tree based on the tenant ID.
2. The cloud-based multi-pre-trained language model management and inference method according to claim 1, characterized in that, The second layer of the vBert model instance tree contains several domain model instances from different domains; each domain model instance consists of several versions, and the initial version of each domain model instance is obtained by training the general pre-trained neural network of the first layer, while the remaining versions are obtained by training the initial version. The third layer of the vBert model instance tree contains several tenant model instances; each tenant model instance consists of several versions, the initial version of each tenant model instance is obtained by training a domain model instance, and the remaining versions are obtained by training the initial version.
3. The cloud-based multi-pre-trained language model management and inference method according to claim 1, characterized in that, The model management requests include creating new domain model instances, updating existing domain model instances, creating new tenant model instances, and / or updating existing tenant model instances. The inference request includes the tenant ID, the domain ID, and the text to be processed.
4. The cloud-based multi-pre-trained language model management and inference method according to claim 1, characterized in that, The transmission methods of the pipeline include: vBert transfers the next batch of required adapters to the GPU while the GPU is processing the previous batch of requests. or, The computation time of a transformer layer in the pre-trained neural network backbone model on the GPU is greater than the transmission latency of the adapter. vBert transmits the next batch of required adapters to the GPU during the computation of the previous transformer layer.
5. An electronic device, comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the cloud-based multi-pre-trained language model management and inference method according to any one of claims 1-4.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the cloud-based multi-pretrained language model management and inference method as described in any one of claims 1-4.
Citation Information
Patent Citations
Distributed parallel database resource management method
CN110119405A
Edge network control system and control method
CN112333774A