Model deployment policy determination and model deployment

Abstract

Provided in the embodiments of the present disclosure are a model deployment policy determination method, a model deployment method, a computing device, a storage medium, and a computer program product. The model deployment policy determination method comprises: determining model attribute information of a task processing model, and device attribute information of a second processor associated with the task processing model; when it is determined, on the basis of the model attribute information and the device attribute information, that model occupation resources of the task processing model are more than storage resources of the second processor, determining the number of offloaded layers of the task processing model; on the basis of the number of offloaded layers and an initial number of layers included in the model attribute information, determining a parameter offloading policy of the task processing model; and on the basis of the parameter offloading policy, determining a model deployment policy of the task processing model.

Description

[0001] Model deployment strategy determination and model deployment technology field

[0002]

[0001] The present disclosure relates to the field of computer technology, and more particularly to model deployment strategy determination and model deployment.

[0003]

[0002] With the development of computer technology, deep learning models have had a significant impact in fields such as computer vision, natural language processing, and data analysis. Currently, in order to improve the accuracy of deep learning models, researchers have begun to explore increasingly large neural structures, resulting in optimized deep learning models with a large number of model parameters. Since models are usually run in the GPU's video memory, and these models with a large number of model parameters also require a large amount of storage space, resulting in a high demand for GPU video memory capacity, which in turn leads to high model inference costs. Therefore, an effective technical solution is urgently needed to solve the above problems.

[0004]

[0003] In view of this, embodiments of the present disclosure provide a method for determining a model deployment strategy. One or more embodiments of the present disclosure also relate to a model deployment strategy determination apparatus, a model deployment method, a model deployment apparatus, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in related technologies.

[0005]

[0004] According to a first aspect of an embodiment of the present disclosure, a method for determining a model deployment strategy is provided, comprising: determining model attribute information of a task processing model and device attribute information of a second processor associated with the task processing model, wherein the task processing model is deployed on a task processing device, and the task processing device comprises a first processor and a second processor; determining the number of unloading layers of the task processing model when it is determined, based on the model attribute information and the device attribute information, that the model occupied resources of the task processing model are greater than the storage resources of the second processor; determining a parameter unloading strategy of the task processing model based on the number of unloading layers and the initial number of layers contained in the model attribute information, wherein the parameter unloading strategy is a strategy for unloading a target processing layer corresponding to the number of unloading layers in the task processing model from the second processor to the first processor; and determining a model deployment strategy of the task processing model based on the parameter unloading strategy.

[0006]

[0005] According to a second aspect of an embodiment of the present disclosure, a model deployment strategy determination device is provided, comprising: a first determination module, configured to determine model attribute information of a task processing model and device attribute information of a second processor associated with the task processing model, wherein the task processing model is deployed on a task processing device, and the task processing device comprises a first processor and a second processor; a second determination module, configured to determine the number of unloaded layers of the task processing model when it is determined, based on the model attribute information and the device attribute information, that the model occupied resources of the task processing model are greater than the storage resources of the second processor; a third determination module, configured to determine a parameter unloading strategy of the task processing model based on the number of unloaded layers and the initial number of layers included in the model attribute information, wherein the parameter unloading strategy is a strategy for unloading the target processing layer corresponding to the number of unloaded layers in the task processing model from the second processor to the first processor; and a fourth determination module, configured to determine the model deployment strategy of the task processing model based on the parameter unloading strategy.

[0007]

[0006] According to a third aspect of an embodiment of the present disclosure, a model deployment method is provided, which is applied to a task processing device for deploying a task processing model, the task processing device comprising a first processor and a second processor, the method comprising: receiving a model deployment strategy of the task processing model, wherein the model deployment strategy is determined according to the model deployment strategy determination method provided in the first aspect of the embodiment of the present disclosure; determining a target processing layer in the task processing model corresponding to the number of unloaded layers according to a parameter unloading strategy in the model deployment strategy, the task processing model being initially deployed on the first processor; unloading the target processing layer from the task processing model to obtain a target task processing model, wherein the target task processing model includes model parameters of task processing layers other than the target processing layer; and transferring the target task processing model from the first processor to the second processor for deployment.

[0008]

[0007] According to a fourth aspect of an embodiment of the present disclosure, a model deployment apparatus is provided, which is applied to a task processing device for deploying a task processing model, the task processing device comprising a first processor and a second processor, and the apparatus comprising: a receiving module, configured to receive a model deployment strategy of the task processing model, wherein the model deployment strategy is determined according to the model deployment strategy determination method provided in the first aspect of an embodiment of the present disclosure; a determining module, configured to determine a target processing layer in the task processing model corresponding to the number of unloaded layers according to a parameter unloading strategy in the model deployment strategy, wherein the task processing model is initially deployed on the first processor; an unloading module, configured to unload the target processing layer from the task processing model to obtain a target task processing model, wherein the target task processing model includes model parameters of task processing layers other than the target processing layer; and a transmitting module, configured to transmit the target task processing model from the first processor to the second processor for deployment.

[0009]

[0008] According to a fifth aspect of an embodiment of the present disclosure, a computing device is provided, comprising: a memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, and when the computer programs / instructions are executed by the processor, the steps of the above-mentioned model deployment strategy determination method or model deployment method are implemented.

[0010]

[0009] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the above-mentioned model deployment strategy determination method or model deployment method are implemented.

[0011]

[0010] According to the seventh aspect of the embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned model deployment strategy determination method or model deployment method.

[0012]

[0011] An embodiment of the present disclosure provides a method for determining a model deployment strategy, which determines model attribute information of a task processing model and device attribute information of a second processor associated with the task processing model, wherein the task processing model is deployed on a task processing device, and the task processing device includes a first processor and a second processor; when it is determined, based on the model attribute information and the device attribute information, that the model occupied resources of the task processing model are greater than the storage resources of the second processor, the number of unloading layers of the task processing model is determined; based on the number of unloading layers and the initial number of layers included in the model attribute information, a parameter unloading strategy of the task processing model is determined, wherein the parameter unloading strategy is a strategy for unloading a target processing layer corresponding to the number of unloading layers in the task processing model from the second processor to the first processor; based on the parameter unloading strategy, a model deployment strategy of the task processing model is determined.

[0013]

[0012] In summary, the above method determines the model occupied resources of the task processing model and the storage resources of the second processor according to the model attribute information of the task processing model and the device attribute information of the second processor associated with the task processing model. When it is determined that the model occupied resources of the task processing model are greater than the storage resources of the second processor, it means that the task processing model cannot be completely deployed in the second processor, then the number of unloaded layers of the task processing model is determined, and the parameter unloading strategy of the task processing model is determined according to the number of unloaded layers and the initial number of layers of the task processing model. This facilitates the subsequent unloading of the target processing layer corresponding to the number of unloaded layers in the task processing model from the second processor to the first processor according to the parameter unloading strategy when deploying the task processing model, so that the task processing model is deployed separately with the first processor and the second processor, which relieves the storage pressure of the second processor, thereby reducing the model inference cost and improving the model inference performance, and ensuring the stability of the model during operation.

[0014]

[0013] FIG1 is a schematic diagram of an application scenario of a method for determining a model deployment strategy provided by one embodiment of the present disclosure;

[0015]

[0014] FIG2 is a flow chart of a method for determining a model deployment strategy provided by one embodiment of the present disclosure;

[0016]

[0015] FIG3 is a schematic diagram of a task processing layer call and transmission in a model deployment strategy determination method provided by one embodiment of the present disclosure;

[0017]

[0016] FIG4 is a schematic diagram of a parameter offloading strategy in a model deployment strategy determination method provided by one embodiment of the present disclosure;

[0018]

[0017] FIG5 is a schematic diagram of a task processing model deployment and call in a model deployment strategy determination method provided by one embodiment of the present disclosure;

[0019]

[0018] FIG6 is a flowchart of a process for determining a model deployment strategy according to an embodiment of the present disclosure;

[0020]

[0019] FIG7 is a schematic diagram of the structure of a model deployment strategy determination device provided by one embodiment of the present disclosure;

[0021] FIG8 is a flow chart of a model deployment method provided by one embodiment of the present disclosure;

[0022] FIG9 is a schematic diagram of a model deployment device according to an embodiment of the present disclosure;

[0023]

[0022] FIG10 is a block diagram of a computing device according to an embodiment of the present disclosure.

[0024] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure. However, the present disclosure can be implemented in many other ways different from those described herein, and those skilled in the art can make similar promotions without violating the connotation of the present disclosure, so the present disclosure is not limited by the specific implementation disclosed below.

[0025] It should be understood that although the terms "first," "second," and so on may be used to describe various information in one or more embodiments of the present disclosure, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, "first" may also be referred to as "second," and similarly, "second" may also be referred to as "first," without departing from the scope of one or more embodiments of the present disclosure. Depending on the context, the term "if" as used herein may be interpreted as "at the time," "when," or "in response to determining."

[0026]

[0026] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0027] In one or more embodiments of the present disclosure, a large model refers to a deep learning model with large-scale model parameters, typically including hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities. Examples include large language models (LLMs) and multimodal pre-trained models.

[0028] (multi-modal pre-training model) etc.

[0029]

[0028] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model and can be applied to different tasks. Large models can be widely used in fields such as natural language processing (NLP) and computer vision. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0030]

[0029] First, the terms involved in one or more embodiments of the present disclosure are explained.

[0031] Large Models: Large models are machine learning models with a large number of parameters and computing resources. These models typically require large amounts of data and computing power during training and have millions to billions of parameters. Large models are designed to improve the model's representational power and performance, enabling it to better capture patterns and regularities in the data when handling complex tasks.

[0032]

[0031] Transformer: Transformer is a deep learning model commonly used to process sequence data and is widely used in natural language processing tasks (such as machine translation and speech recognition). The most important component is self-attention.

[0033] The self-attention mechanism simultaneously considers all positions in the input sequence, calculates the correlation between each position and all other positions, and then takes a weighted average of these correlations. In addition to the self-attention mechanism, the Transformer model also includes two parts: an encoder and a decoder.

[0034]

[0032] KV Cache: A Transformer, primarily a Decoder, performs inference using a given input sequence as its initial input, then generates additional tokens one by one to further refine the generated sequence. During the sampling process, the Transformer performs self-attention, extracting a KV vector for each token in the current sequence. During the next generation, a KV vector must be extracted again for each token in the sequence. The KV Cache avoids repeated computations by storing the KV results of previously calculated tokens in a region, which prevents repeated computations during the next generation. This region is the KV Cache.

[0035]

[0033] Tensor Parallel: Tensor parallelism is a parallel computing method used to accelerate the computation of tensor operations. In deep learning, tensors refer to multidimensional arrays, such as images, sounds, or text data. Tensor parallelism improves computational efficiency and speed by dividing large tensors into smaller ones and using multiple processors or computing units to process them simultaneously. This method can perform parallel computing in distributed systems, thereby accelerating the training and inference processes of deep learning models. Through tensor parallelism, computation time can be reduced and system throughput and performance can be improved.

[0036]

[0034] Data Parallel: Data parallelism is a parallel computing method used to simultaneously process different copies of data on multiple processors or computing units. In deep learning, data parallelism is often used to distribute and process large amounts of training data across multiple GPUs or multiple machines. The basic idea of ​​data parallelism is to divide the training data into multiple batches, each of which is assigned to a different processor or computing unit for processing. Each processor independently calculates its assigned data batch, then aggregates the results and updates the model parameters. Data parallelism can accelerate the training of deep learning models and improve the throughput and performance of the system.

[0037]

[0035] Memory offloading technology: The offloading technology mentioned in this article refers to storing some model parameters in the CPU memory, and then moving them from the CPU memory to the GPU memory when the GPU calculation requires these parameters.

[0038] Prefetching: This technique involves moving parameters from CPU memory to GPU memory before they are actually needed, thereby overlapping computation and transmission. If these parameters have already been moved when they are needed, the computation can continue without waiting, thus maintaining computational efficiency.

[0039]

[0037] The present disclosure provides a model deployment strategy determination method. One or more embodiments of the present disclosure also relate to a model deployment strategy determination apparatus, a model deployment method, a model deployment apparatus, a computing device, a computer-readable storage medium, and a computer program product, each of which is described in detail in the following embodiments.

[0040]

[0038] Referring to FIG1 , FIG1 shows a schematic diagram of an application scenario of a model deployment strategy determination method provided according to an embodiment of the present disclosure.

[0041]

[0039] FIG. 1 includes a terminal device 102 and a cloud device 104, wherein the cloud device 104 can be used as a policy determination platform to generate a model deployment policy for a user.

[0042]

[0040] In a specific implementation, in the e-commerce field, due to project requirements, an e-commerce platform may need to provide product image recognition services for consumers who purchase products. That is, consumers can upload images of a certain product, and the e-commerce platform can display similar products to the product. Therefore, the e-commerce platform can use an image recognition model to provide consumers with product image recognition services. The e-commerce platform can deploy the image recognition model on a task processing device (such as a server). Then, the cloud instance device 104 can generate a model deployment policy for the image recognition model and send it to the end device 102. The end device 102 displays it to the user, allowing the task processing device to deploy the image recognition model according to the model deployment policy.

[0043]

[0041] As shown in FIG1 , the e-commerce platform, as a user, sends the model attribute information of the image recognition model and the device attribute information of the GPU of the task processing device deploying the image recognition model to the cloud device 104 via the end device 102. The cloud device 104 can determine whether the image recognition model can be fully deployed in the GPU's video memory based on the model attribute information and the GPU's device attribute information. Then, if it is determined based on the model attribute information and the device attribute information that the model resources occupied by the image recognition model are greater than the GPU's video memory storage resources, the number of offloaded layers of the image recognition model can be calculated as n. Based on the total number of task processing layers m included in the image recognition model and the number of offloaded layers n, a parameter offload strategy for the image recognition model is determined. This parameter offload strategy can be used to indicate which task processing layers in the image recognition model can be offloaded from the GPU's video memory to the CPU's memory, thereby obtaining a model deployment strategy for the image recognition model. The Yunjia 1 device 104 can send the model deployment strategy to the Endjia 1 device 102, so that the user can deploy the image recognition model according to the model deployment strategy.

[0044]

[0042] The terminal device 102 may include a browser, an APP (Application), or a web application such as an H5 (Hypertext Markup Languages, version 5) application, a light application (also known as a mini-program, a lightweight application), or a cloud application. The terminal device may be developed based on a software development kit (SDK) of a corresponding service provided by the server, such as a real-time communication (RTC) SDK. The terminal device may be deployed in an electronic device and may rely on the device to run or on certain APPs in the device to run. The electronic device may have a display and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, or a personal computer. Various other types of applications may also be configured in the electronic device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0045]

[0043] The cloud instance device 104 can be understood as a server that provides various services, including physical servers and cloud servers. For example, a server that provides communication services to multiple clients, a server that provides background training to support models used on clients, or a server that processes data sent by clients. It should be noted that the cloud instance device 104 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The cloud instance device 104 can also be a server in a distributed system, or a server integrated with a blockchain. The cloud instance device 104 can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0046]

[0044] It is worth noting that the model deployment strategy determination method provided in the embodiment of the present disclosure can be executed by the Yunjia 1 device 104. In other embodiments of the present disclosure, the model deployment strategy determination method provided in the embodiment of the present disclosure can be executed by the Duanjia 1 device 102; in other embodiments, the model deployment strategy determination method provided in the embodiment of the present disclosure can also be jointly executed by the Duanjia 1 device 102 and the Yunjia 1 device 104.

[0047]

[0045] Referring to FIG2, FIG2 shows a flow chart of a method for determining a model deployment strategy according to an embodiment of the present disclosure, which specifically includes the following steps.

[0048]

[0046] Step 202: Determine model attribute information of a task processing model and device attribute information of a second processor associated with the task processing model, wherein the task processing model is deployed on a task processing device, and the task processing device includes a first processor and the second processor.

[0049]

[0047] The model deployment strategy determination method provided in the embodiments of the present disclosure can be applied to deployment scenarios of various task processing models. Task processing models can include machine learning models, deep learning models, neural network models, and large models. Task processing models can be used for various tasks, such as speech recognition tasks and image and text processing tasks.

[0050]

[0048] The task processing device can be understood as a server that deploys a task processing model, the second processing device can be understood as a GPU, and the first processing device can be understood as a CPU. The model attribute information of the task processing model may include information such as the model size of the task processing model, the number of task processing layers included in the task processing model, and the output character length of the task processing model. The device attribute information of the second processor may include the storage resources of the second processor. If the second processing device is a GPU, the device attribute information of the second processor may be the video memory storage resources of the GPU.

[0051]

[0049] Based on this, the model attribute information of the task processing model, such as the model size, the number of task processing layers, the output character length, and the device attribute information of the GPU deployed by the task processing model can be determined according to the user's input.

[0052]

[0050] In one embodiment of the present disclosure, the task processing model may be a trained task processing model.

[0053]

[0051] Step 204: When it is determined based on the model attribute information and the device attribute information that the model occupied resources of the task processing model are greater than the storage resources of the second processor, the number of unloading layers of the task processing model is determined.

[0054]

[0052] The model occupied resources of the task processing model can be understood as the storage resources required for the task processing model to be deployed in the second processor. The number of offloaded layers of the task processing model can be understood as the number of task processing layers in the task processing model that need to be offloaded. For example, if the task processing model includes five task processing layers, the number of offloaded layers of the task processing model can be two or three. The specific number of offloaded layers needs to be calculated based on the model occupied resources of the task processing model and the storage resources of the second processor.

[0055]

[0053] In a specific implementation, after determining the model attribute information of the task processing model and the device attribute information of the second processor associated with the task processing model, it also includes: determining the sample processing information of the task processing model; accordingly, when it is determined that the model occupied resources of the task processing model are greater than the storage resources of the second processor based on the model attribute information and the device attribute information, determining the number of unloading layers of the task processing model includes: calculating the model occupied resources of the task processing model based on the sample processing information and the model attribute information; determining the storage resources of the second processor based on the device attribute information of the second processor; when it is determined that the model occupied resources are greater than the storage resources of the second processor, determining the number of unloading layers of the task processing model corresponding to the sample processing information.

[0056]

[0054] The sample processing information of the task processing model can be understood as the number of samples processed by the task processing model, that is, the batch size of the task processing model. It is understood that the resource usage of the task processing model varies under different batch sizes. The model resource usage of the task processing model under a certain batch size may include the model weight size, activation size, KV cache size, fixed overhead of the library used, etc. After the model resource usage is calculated based on the batch size and model attribute information of the task processing model, it can be compared with the storage resources of the second processor (i.e., the GPU) to determine whether more storage space (i.e., GPU memory space) is required to run the current task processing model.

[0057]

[0055] Based on this, the model occupied resources of the task processing model can be calculated according to the batch size and model attribute information of the task processing model, and the number of unloaded layers of the task processing model under the batch size can be determined according to the model occupied resources and the GPU memory storage resources.

[0058] In practical applications, the model resource usage during the task processing model inference process is determined by the model size, the size of the intermediate cache (i.e., the KV cache) used during the inference process, and the fixed overhead of the deep learning framework used. The model size and the deep learning framework usage are fixed, while the size of the intermediate cache used during the inference process is calculated based on the batch size. Therefore, the model resource usage required by the task processing model varies with different batch sizes.

[0059]

[0057] For example, the model resource usage m of the task processing model at this batch size can be calculated based on the batch size a of the task processing model, the model size of the task processing model, the size of the intermediate cache (i.e., the KV cache) used during the inference process, and the fixed overhead of the deep learning framework used. If it is determined that the model resource usage m is greater than the GPU storage resource n, it indicates that the GPU of the device used by the user is insufficient to infer the task processing model. Therefore, model offloading is required, and the number of offloaded layers of the task processing model at this batch size is determined.

[0060]

[0058] In summary, by introducing the batch size of the task processing model, the accurate calculation of the model occupied resources of the task processing model is achieved, which facilitates the determination of the model deployment strategy of the subsequent task processing model.

[0061]

[0059] Specifically, determining the number of unloading layers of the task processing model corresponding to the sample processing information includes: calculating the target storage resources based on the resources occupied by the model and the storage resources of the second processor; determining the number of unloading layers of the task processing model corresponding to the sample processing information based on the target storage resources and the parameter size of a single task processing layer in the task processing model.

[0062]

[0060] The target storage resource can be understood as the resource difference between the model occupied resources and the storage resources of the second processor. A single task processing layer in the task processing model can be understood as a certain task processing layer in the task processing model, that is, the smallest unit for unloading the task processing model. In practical applications, the decoder in the task processing model can be used as the

[0063] (ie Decoder) as the smallest unit of unloading.

[0064]

[0061] Based on this, the resource difference between the model occupied resources and the storage resources of the second processor can be calculated, and the resource difference can be used as the target storage resource. Then, according to the target storage resource and the parameter size of a single task processing layer in the task processing model, the number of unloaded layers of the task processing model under a certain batch size can be determined.

[0065]

[0062] Continuing with the above example, when calculating the number of unloaded layers of the task processing model under batch size a, the difference mn between the model occupied resources m and the storage resources n of the second processor can be divided by the parameter size b of a single task processing layer in the task processing model. The number of unloaded layers of the task processing model under batch size a is (mn) / b.

[0066] In summary, by calculating the number of offloaded layers of the task processing model under the batch size, it is convenient to automatically determine the parameter offloading strategy (i.e., optimize the offloading and prefetching locations) based on the number of offloaded layers. This ensures flexibility and ease of use, and the parameter offloading function can be implemented without the user having to modify the model.

[0067] Furthermore, batch size can affect the processing performance of the task processing model, and different batch sizes can also affect the model resource usage of the task processing model. Therefore, in order to maximize the processing performance of the task processing model while saving labor costs, the number of offloaded layers of the task processing model under multiple batch sizes can be calculated. The specific implementation method is as follows.

[0068]

[0065] The task processing model has multiple sample processing information; accordingly, when it is determined that the model occupied resources of the task processing model are greater than the storage resources of the second processor based on the model attribute information and the device attribute information, the number of unloading layers of the task processing model is determined, including: sequentially determining multiple sample processing information as first sample processing information; calculating the model occupied resources of the task processing model based on the first sample processing information and the model attribute information; determining the storage resources of the second processor based on the device attribute information of the second processor; and when it is determined that the model occupied resources are greater than the storage resources of the second processor, determining the number of unloading layers of the task processing model corresponding to the first sample processing information.

[0069]

[0066] The first sample processing information can be understood as each sample processing information. That is, the number of offloaded layers of the task processing model corresponding to each sample processing information can be calculated, that is, the number of offloaded layers of the task processing model at each batch size.

[0070]

[0067] Specifically, multiple batch sizes can be determined, and the number of offloaded layers of the task processing model at each batch size can be calculated according to the above calculation process. The specific calculation process is similar to that described above and will not be further described in the embodiments of this disclosure.

[0068] In actual applications, the batch size of the task processing model can be provided by the user or manually set by the developer who generates the model deployment strategy.

[0071]

[0069] For example, multiple batch sizes of 1, 2, 3, 4, and 5 may be determined, and the model resources occupied by the task processing model under the batch size of 1 may be calculated, thereby further calculating the number of unloaded layers of the task processing model under the batch size of 1. Correspondingly, the number of unloaded layers of the task processing model under the batch size of 2, the number of unloaded layers under the batch size of 3, the number of unloaded layers under the batch size of 4, and the number of unloaded layers under the batch size of 5 may be calculated.

[0072]

[0070] In summary, by calculating the number of offloaded layers of the task processing model under multiple batch sizes, it is convenient to subsequently determine the model deployment strategy of the task processing model under multiple batch sizes, so that the optimized model deployment strategy can be determined to maximize the model throughput, thereby improving the processing performance of the task processing model deployed according to the final determined model deployment strategy.

[0073]

[0071] Step 206: Determine the parameter offloading strategy of the task processing model based on the number of offloading layers and the initial number of layers included in the model attribute information, wherein the parameter offloading strategy is a strategy for offloading the target processing layer corresponding to the number of offloading layers in the task processing model from the second processor to the first processor.

[0074]

[0072] The initial number of layers included in the model attribute information can be understood as the total number of layers in the task processing model. For example, if the task processing model includes five task processing layers, then the total number of layers in the task processing model is five. The parameter offloading strategy can be a strategy of offloading one layer every X layers. The offloading interval can be indirectly expressed as offloading one layer every X layers. For example, the parameter offloading strategy can be offloading one layer every four layers. When the total number of layers in the task processing model is 96 and the number of offloaded layers is 24, the offloading interval is 3, and X is 4, meaning that one layer is offloaded every four layers, or one layer is offloaded every three layers, i.e., the fourth layer, the eighth layer, and so on. The target processing layer corresponding to the number of offloaded layers can be understood as the task processing layer to be offloaded. Offloading from the second processor to the first processor can be understood as deploying the target processing layer on the first processor (i.e., the CPU). When the task processing model is subsequently called, the target processing layer in the first processor is pre-fetched to the second processor.

[0075]

[0073] In a specific implementation, determining the parameter unloading strategy of the task processing model based on the number of unloaded layers and the initial number of layers included in the model attribute information includes: calculating the number of unloaded layers and the initial number of layers included in the model attribute information according to a preset strategy algorithm to obtain the unloading interval of the task processing model; determining the parameter unloading strategy of the task processing model based on the number of unloaded layers and the unloading interval.

[0076]

[0074] In one embodiment of the present disclosure, the number of unloading layers may include a first number of unloading layers and a second number of unloading layers, the unloading interval may include a first unloading interval and a second unloading interval, and the preset strategy algorithm may be an algorithm for calculating the first number of unloading layers, the second number of unloading layers, the first unloading interval and the second unloading interval, for example, it may be a binary linear algorithm.

[0077] Specifically, for a task processing layer that needs to be offloaded, if the computation time of a single task processing layer is less than the parameter transmission time of the single task processing layer, and the computation time of two task processing layers is greater than the transmission time of a single task processing layer, then the set offloading interval can be at least 2. To maximize the computation time within the offloading interval, the offloading interval can be as large as possible. When the number of offloading layers in the task processing model is known, the optimized parameter offloading strategy can be to specify the target processing layers to be offloaded in the task processing model as evenly as possible, to avoid the problem of too small an offloading interval, which results in the data transmission delay being unable to be concealed, and thus increases the overall network inference delay.

[0078] For example, if the total number of layers in the task processing model is 96 and the number of layers to be offloaded is 24, the optimized interval should be set to 3, meaning that one layer is offloaded for every four layers. This results in a lower model latency than any other setting method. When the number of layers to be offloaded is not divisible by the total number of layers in the model, one interval will not meet the requirements. For example, when the number of layers to be offloaded is 23, which is not divisible by 96, two strategies are required. In practical applications, the formula for calculating the parameter offloading strategy (i.e., the preset strategy algorithm) is as follows.

[0079]

[0077] Where L is the total number of layers in the task processing model, and K is the number of offloaded layers. The first strategy can be expressed as offloading one layer for every a layers, where a equals L / K. The second strategy can be expressed as offloading one layer for every b layers, where b equals L / K+1. m is the first offloaded layer number, where one layer is offloaded for every a layers, and n is the second offloaded layer number, where one layer is offloaded for every b layers. For example, when L is 96 and k is 23, it can be calculated that a is 4 and b is 5. In this case, m is 19 and n is 4. The determined parameter offloading strategy is to offload one layer for every 4 layers, thus offloading 19 layers, and to offload one layer for every 5 layers, thus offloading 4 layers.

[0080]

[0078] Referring to FIG. 3 , FIG. 3 illustrates a schematic diagram of task processing layer invocation and transmission in a model deployment strategy determination method provided according to an embodiment of the present disclosure. As shown in FIG. 3 , for a single task processing layer in a task processing model, without prefetching, the invocation time (i.e., computation time) of the single task processing layer and the transmission time (i.e., data loading time) of the model parameters of the single task processing layer from the first processor to the second processor are continuous. However, with prefetching, the invocation of the single task processing layer and the transmission of the model parameters can be performed in parallel, and the data loading time is completely hidden within the invocation time, so that the processing of the task processing model is not affected. The decoding layer (i.e., decoder) in the task processing model is used as the minimum unit for offloading. For the decoding layer, the time to transmit the parameters of one decoding layer is much longer than the time to calculate the parameters of one decoding layer. To maximize overlap between computation and data transmission, the time to calculate N decoding layers can be greater than or equal to the time to transmit the parameters of one decoding layer. Therefore, if the task processing layers to be offloaded are appropriately spaced, prefetching can completely hide the data transmission time, making the impact of offloading on computational efficiency virtually negligible while also conserving GPU memory. On the one hand, the latency without prefetching is much higher than that with prefetching. Furthermore, when prefetching is used, the data loading latency is completely hidden, and model computation efficiency is not affected. In other words, during task processing in the task processing model, the call time (i.e., computation time) of the task processing layer and the transfer time of the model parameters of the task processing layer deployed on the CPU to the GPU can be carried out simultaneously, thereby minimizing the additional time overhead caused by data transfer and ensuring normal inference of the task processing model. See Figure 4, which shows a schematic diagram of a parameter offloading strategy in a model deployment strategy determination method provided according to an embodiment of the present disclosure. As shown in Figure 4, according to this parameter offloading strategy, task processing layers 1, 3, 4, and 6 are deployed on the GPU, and task processing layers 2 and 5 are deployed on the CPU. Therefore, during the task processing model inference process, the model parameters of task processing layers 2 and 5 can be loaded sequentially from the CPU to the GPU, thereby completing the task processing model inference process on the GPU.

[0081]

[0079] It can be understood that the parameter offloading strategy also corresponds to the batch size of the task processing model. Accordingly, the parameter offloading strategy of the task processing model in each batch size can be determined in multiple batch sizes.

[0082]

[0080] Step 208: Determine the model deployment strategy of the task processing model according to the parameter offloading strategy.

[0083] In practical applications, after calculating an optimized parameter offloading strategy based on the number of offloaded layers, the number of layers to be offloaded is not fixed. As the batch size increases, the number of offloaded layers also increases. When the call time of a single task processing layer cannot mask the transmission time of that single task processing layer, model inference latency will increase significantly, and throughput will also decrease. In other words, as the batch size and the number of offloaded layers increase, the model throughput will initially increase and then decrease. To ensure the highest throughput, the number of offloaded layers of the task processing model at different batch sizes, as well as the corresponding latency and throughput, can be estimated based on a cost model. The specific implementation is as follows.

[0084]

[0082] After determining the parameter unloading strategy of the task processing model, it also includes: determining the transmission time of the model parameters of a single task processing layer in the task processing model corresponding to the sample processing information from the second processor to the first processor; determining the calling time of the single task processing layer corresponding to the sample processing information; and calculating the task processing information of the task processing model corresponding to the sample processing information based on the transmission time, the calling time, the initial number of layers and the unloaded number of layers.

[0085]

[0083] The task processing information may include the throughput and latency of the task processing model. The call time of a single task processing layer can be understood as the calculation time of the single task processing layer. The transmission time from the second processor to the first processor can also be understood as the transmission time from the first processor to the second processor.

[0086]

[0084] Based on this, for a certain batch size, the transmission time of the model parameters of a single task processing layer of the task processing model at this batch size from the GPU to the CPU can be calculated, and the calculation time of a single task processing layer of the task processing model at this batch size can be calculated. Based on the transmission time, calculation time, total number of layers and number of unloaded layers, the throughput and latency of the task processing model at this batch size can be calculated.

[0087]

[0085] Accordingly, the throughput and latency of the task processing model at each batch size (ie, each sample processing information) can be calculated.

[0088]

[0086] In a specific implementation, when calculating the transmission time for model parameters of a single task processing layer (i.e., a single Decoder layer) of a task processing model at a target batch size (a certain batch size) from a second processor to a first processor, a reference transmission time for the model parameters of a single task processing layer from the second processor to the first processor when the reference batch size is 1 can be measured. Based on this reference transmission time and the target batch size, the transmission time for the model parameters of a single task processing layer of the task processing model at the target batch size is determined. Accordingly, when calculating the call time of a single task processing layer of the task processing model at the target batch size, a reference call time of a single task processing layer when the reference batch size is 1 can also be measured. Based on this reference call time and the target batch size, the call time of the single task processing layer of the task processing model at the target batch size is calculated.

[0089]

[0087] In practical applications, the formula for calculating the throughput delay of the task processing model is as follows.

[0090]

[0088] Wherein, t is the delay, throughput is the throughput, n is the target batch size (i.e., a certain batch size), t1 is the reference transmission time for the model parameters of a single task processing layer from the CPU to the GPU when the batch size is 1, and t2 is the reference call time for a single task processing layer when the batch size is 1.

[0091]

[0089] In summary, by taking throughput as a guide, the optimized batch size and number of offloaded layers can be automatically determined, thereby maximizing model throughput. This method is automated, flexible, and easy to use, and parameter write functionality can be implemented without the user having to modify the model.

[0090] Specifically, determining the model deployment strategy for the task processing model based on the parameter offloading strategy includes: determining the model deployment strategy for the task processing model based on the parameter offloading strategy and the task processing information.

[0092]

[0091] In addition, after determining the model deployment strategy of the task processing model, it also includes: deploying the task processing model in the task processing device according to the model deployment strategy.

[0093]

[0092] Specifically, after determining the model deployment strategy of the task processing model, the task processing model can be deployed in the task processing device according to the model deployment strategy.

[0094]

[0093] In one embodiment of the present disclosure, when a task processing model includes multiple batch sizes, the model deployment strategy ultimately obtained is the model deployment strategy corresponding to each batch size. In this case, the model deployment strategy with the highest throughput can be determined from the model deployment strategies corresponding to each batch size as the target model deployment strategy, and the task processing model can be deployed in the task processing device based on the target model deployment strategy. In another embodiment of the present disclosure, the model deployment strategy corresponding to each batch size can also be sent to the user, and one of the model deployment strategies corresponding to each batch size can be selected as the target model deployment strategy based on the user's input instructions. This embodiment of the present disclosure is not limited to this.

[0095]

[0094] In addition, the model deployment strategy of the task processing model under any batch size can also be calculated, and the task processing model can be deployed according to the model deployment strategy.

[0096]

[0095] In a specific implementation, deploying the task processing model in the task processing device according to the model deployment strategy includes: determining the target processing layer corresponding to the number of unloaded layers in the task processing model according to the parameter unloading strategy in the model deployment strategy, and initially deploying the task processing model on the first processor; unloading the target processing layer from the task processing model to obtain a target task processing model, wherein the target task processing model includes model parameters of task processing layers other than the target processing layer; and transferring the target task processing model from the first processor to the second processor for deployment.

[0097] Specifically, the target processing layer to be uninstalled in the task processing model can be determined based on the parameter uninstallation strategy in the model deployment strategy. The target processing layer is uninstalled from the task processing model to obtain the target task processing model. The target task processing model is then transferred from the first processor to the second processor for deployment. After deployment is complete, the target processing layers remaining in the first processor can be pre-fetched to the second processor in sequence during subsequent calls.

[0098]

[0097] Referring to FIG. 5 , FIG. 5 shows a schematic diagram of the deployment and invocation of a task processing model in a method for determining a model deployment strategy according to an embodiment of the present disclosure. As shown in FIG. 5 , when tensor parallelism and memory offloading technology work together in the task processing model, tensor parallelism is an intra-layer operation, which can split the weight parameters of the task processing layer onto multiple GPUs, with the multiple GPUs jointly calculating the results of a single task processing layer. Memory offloading technology, on the other hand, is an inter-layer operation, where parameter transfer is performed at a layer granularity. That is, the task processing layer to be offloaded can be offloaded to the CPU memory, achieving integration with tensor parallelism. Furthermore, the task processing model deployed according to the model deployment strategy can also achieve data parallelism, allowing the model parameters of the task processing layer to be transferred from the CPU memory to the GPU memory before being used.

[0099]

[0098] In summary, the task processing model deployed by the above model deployment strategy is compatible with different parallel computing methods, such as data parallelism and tensor parallelism, and can also alleviate the pressure on the GPU memory, achieving the effect of using less to replace more, thereby saving hardware costs.

[0100]

[0099] Furthermore, after the target task processing model is transferred from the first processor to the second processor for deployment, the task processing device may, in response to a call to the task processing model based on a pending task, sequentially obtain model parameters of the target processing layer from the first processor in accordance with the processing order of the task processing layers included in the task processing model, and transmit the parameters to the second processor, so that the second processor calls the target processing layer to process the pending task. It is understood that a target storage area may be allocated in the second processor, and the storage resources of the target storage area may be equal to the resources required for the model parameters of a single task processing layer. When the model parameters of the target processing layers are sequentially pre-fetched from the first processor and transmitted to the second processor, they may be transmitted to the target storage area of ​​the second processor. Furthermore, the pre-fetched model parameters of the next target processing layer may overwrite the model parameters of the previous target processing layer in the target storage area. For example, if layers 3 and 6 of the task processing model are offloaded to the first processor (i.e., the CPU), when the task processing model is invoked, the pending tasks input into the task processing model are processed sequentially by each task processing layer of the task processing model. At this point, the third task processing layer of the task processing model can be prefetched from the first processor to the target storage area. After the prefetch is complete, the sixth task processing layer of the task processing model can be prefetched from the first processor to the target storage area, overwriting the third task processing layer in the target storage area. It is understandable that, because the target processing layer of the task processing model to be offloaded to the first processor is determined based on the aforementioned model deployment strategy, the prefetch process ensures the normal operation of the task processing layer. Furthermore, if, during the execution of the task processing model, it has reached the third task processing layer but the prefetch of the third task processing layer has not yet completed, the task processing model can wait until the prefetch of the third task processing layer is complete before invoking the third task processing layer to continue processing.

[0101]

[0100] In summary, the above method judges the model occupied resources of the task processing model and the storage resources of the second processor according to the model attribute information of the task processing model and the device attribute information of the second processor associated with the task processing model. When it is determined that the model occupied resources of the task processing model are greater than the storage resources of the second processor, it means that the task processing model cannot be completely deployed in the second processor, then the number of unloading layers of the task processing model is determined, and the parameter unloading strategy of the task processing model is determined according to the number of unloading layers and the initial number of layers of the task processing model, so that when the task processing model is subsequently deployed, the target processing layer corresponding to the number of unloading layers in the task processing model is unloaded from the second processor to the first processor according to the parameter unloading strategy, so that the task processing model is deployed separately with the first processor and the second processor, which relieves the storage pressure of the second processor, thereby reducing the model inference cost and improving the model inference performance, and ensuring the stability of the model during operation.

[0102]

[0101] The following further illustrates the model deployment strategy determination method provided by the present disclosure, using its application in task processing model deployment as an example, in conjunction with FIG6. FIG6 shows a flowchart of the processing process of a model deployment strategy determination method provided by one embodiment of the present disclosure, specifically including the following steps.

[0103]

[0102] Step 602: Determine a reference transfer time for transferring model parameters of a single task processing layer in the task processing model from the CPU to the GPU when the batch size is 1, and determine a reference call time for a single task processing layer in the task processing model when the batch size is 1.

[0104]

[0103] Step 604: Determine the first batch size from multiple batch sizes.

[0105]

[0104] Among them, the first batch size can be understood as any batch size among multiple batch sizes.

[0106]

[0105] Step 606: Calculate the model occupancy resources of the task processing model under the first batch size.

[0107]

[0106] Step 608: Calculate the number of offloaded layers of the task processing model based on the GPU memory resources of the task processing device deployed with the task processing model and the model occupancy resources of the task processing model at the first batch size.

[0107] Specifically, the number of offloaded layers of the task processing model can be calculated based on the difference between the memory resources and the model occupancy resources and the total number of task processing layers included in the task processing model.

[0108]

[0108] Step 610: Determine the task processing information of the task processing model under the first batch size according to the reference transmission time, reference call time, and number of offload layers when the batch size is 1.

[0109]

[0109] Among them, the task processing information may include the throughput and delay of the task processing model.

[0110]

[0110] Step 612: Determine the parameter offloading strategy according to the number of offloading layers and the total number of layers of the task processing model.

[0111]

[0111] Step 614: Determine the model deployment strategy of the task processing model in the first batch size according to the parameter offloading strategy and the task processing information.

[0112]

[0112] Step 616: Continue executing the above step 604 until the model deployment strategy of the task processing model at each batch size is obtained.

[0113]

[0113] Specifically, after obtaining the model deployment strategy of the task processing model under each batch size, the model throughput generated after the task processing model is deployed according to each model deployment strategy can also be determined, and the model deployment strategy with the largest model throughput is determined as the target model deployment strategy. The task processing model is deployed according to the target model deployment strategy to further ensure the processing performance of the task processing model.

[0114]

[0114] In summary, the above method judges the model occupied resources of the task processing model and the storage resources of the second processor according to the model attribute information of the task processing model and the device attribute information of the second processor associated with the task processing model. When it is determined that the model occupied resources of the task processing model are greater than the storage resources of the second processor, it means that the task processing model cannot be completely deployed in the second processor, then the number of unloading layers of the task processing model is determined, and the parameter unloading strategy of the task processing model is determined according to the number of unloading layers and the initial number of layers of the task processing model, so that when the task processing model is subsequently deployed, the target processing layer corresponding to the number of unloading layers in the task processing model is unloaded from the second processor to the first processor according to the parameter unloading strategy, so that the task processing model is deployed separately with the first processor and the second processor, which relieves the storage pressure of the second processor, thereby reducing the model inference cost and improving the model inference performance, and ensuring the stability of the model during operation.

[0115]

[0115] Corresponding to the above-mentioned method embodiment, the present disclosure also provides an embodiment of a device for determining a model deployment strategy. FIG7 shows a schematic structural diagram of a device for determining a model deployment strategy provided by an embodiment of the present disclosure. As shown in FIG7, the device includes the following modules.

[0116]

[0116] The first determining module 702 is configured to determine model attribute information of a task processing model and device attribute information of a second processor associated with the task processing model, wherein the task processing model is deployed on a task processing device, and the task processing device includes a first processor and the second processor.

[0117]

[0117] The second determining module 704 is configured to determine the number of unloading layers of the task processing model when it is determined that the model occupied resources of the task processing model are greater than the storage resources of the second processor based on the model attribute information and the device attribute information.

[0118]

[0118] The third determination module 706 is configured to determine the parameter unloading strategy of the task processing model based on the number of unloading layers and the initial number of layers contained in the model attribute information, wherein the parameter unloading strategy is a strategy for unloading the target processing layer corresponding to the number of unloading layers in the task processing model from the second processor to the first processor.

[0119]

[0119] The fourth determining module 708 is configured to determine the model deployment strategy of the task processing model according to the parameter offloading strategy.

[0120]

[0120] In an optional embodiment, the second determination module 704 is further configured to: determine the sample processing information of the task processing model; accordingly, when it is determined that the model occupied resources of the task processing model are greater than the storage resources of the second processor based on the model attribute information and the device attribute information, determine the number of unloading layers of the task processing model, including: calculating the model occupied resources of the task processing model based on the sample processing information and the model attribute information; determining the storage resources of the second processor based on the device attribute information of the second processor; when it is determined that the model occupied resources are greater than the storage resources of the second processor, determine the number of unloading layers of the task processing model corresponding to the sample processing information.

[0121]

[0121] In an optional embodiment, the second determination module 704 is further configured to: calculate the target storage resources based on the resources occupied by the model and the storage resources of the second processor; and determine the number of unloaded layers of the task processing model corresponding to the sample processing information based on the target storage resources and the parameter size of a single task processing layer in the task processing model.

[0122]

[0122] In an optional embodiment, there are multiple sample processing information of the task processing model; the second determination module 704 is further configured to: determine the multiple sample processing information as the first sample processing information in sequence; calculate the model occupied resources of the task processing model according to the first sample processing information and the model attribute information; determine the storage resources of the second processor according to the device attribute information of the second processor; and when it is determined that the model occupied resources are greater than the storage resources of the second processor, determine the number of unloading layers of the task processing model corresponding to the first sample processing information.

[0123]

[0123] In an optional embodiment, the third determination module 706 is further configured to: calculate the number of unloading layers and the initial number of layers included in the model attribute information according to a preset strategy algorithm to obtain the unloading interval of the task processing model; and determine the parameter unloading strategy of the task processing model based on the number of unloading layers and the unloading interval.

[0124]

[0124] In an optional embodiment, the third determination module 706 is further configured to: determine the transmission time of the model parameters of a single task processing layer in the task processing model from the second processor to the first processor; determine the call time of the single task processing layer corresponding to the sample processing information; and calculate the task processing information of the task processing model corresponding to the sample processing information based on the transmission time, the call time, the initial number of layers and the unloaded number of layers.

[0125]

[0125] In an optional embodiment, the fourth determining module 708 is further configured to: determine the model deployment strategy of the task processing model according to the parameter offloading strategy and the task processing information.

[0126]

[0126] In an optional embodiment, the apparatus further includes a deployment module configured to: deploy the task processing model in the task processing device according to the model deployment strategy.

[0127]

[0127] In an optional embodiment, the deployment module is further configured to: determine the target processing layer corresponding to the number of unloaded layers in the task processing model according to the parameter unloading strategy in the model deployment strategy, and the task processing model is initially deployed on the first processor; unload the target processing layer from the task processing model to obtain a target task processing model, wherein the target task processing model includes model parameters of task processing layers other than the target processing layer; and transfer the target task processing model from the first processor to the second processor for deployment.

[0128]

[0128] In summary, the above-mentioned device judges the model occupied resources of the task processing model and the storage resources of the second processor based on the model attribute information of the task processing model and the device attribute information of the second processor associated with the task processing model. When it is determined that the model occupied resources of the task processing model are greater than the storage resources of the second processor, it means that the task processing model cannot be completely deployed in the second processor, then the number of unloading layers of the task processing model is determined, and the parameter unloading strategy of the task processing model is determined according to the number of unloading layers and the initial number of layers of the task processing model, so that when the task processing model is subsequently deployed, the target processing layer corresponding to the number of unloading layers in the task processing model is unloaded from the second processor to the first processor according to the parameter unloading strategy, so that the task processing model is deployed separately with the first processor and the second processor, which alleviates the storage pressure of the second processor, thereby reducing the model inference cost and improving the model inference performance, and ensuring the stability of the model during operation.

[0129]

[0129] The above is a schematic diagram of a model deployment strategy determination device according to this embodiment. It should be noted that the technical solution of the model deployment strategy determination device and the technical solution of the model deployment strategy determination method described above are based on the same concept. For details not described in detail in the technical solution of the model deployment strategy determination device, please refer to the description of the technical solution of the model deployment strategy determination method described above.

[0130]

[0130] Referring to FIG. 8 , FIG. 8 shows a flowchart of a model deployment method provided according to an embodiment of the present disclosure, which is applied to a task processing device for deploying a task processing model, wherein the task processing device includes a first processor and a second processor, and specifically includes the following steps.

[0131]

[0131] Step 802: Receive the model deployment strategy of the task processing model, wherein the model deployment strategy is determined according to the model deployment strategy determination method provided by the embodiment of the present disclosure.

[0132]

[0132] Step 804: According to the parameter offloading strategy in the model deployment strategy, determine the target processing layer in the task processing model corresponding to the number of offloading layers, and the task processing model is initially deployed on the first processor.

[0133]

[0133] Step 806: Unload the target processing layer from the task processing model to obtain a target task processing model, wherein the target task processing model includes model parameters of task processing layers other than the target processing layer.

[0134]

[0134] Step 808: Transfer the target task processing model from the first processor to the second processor for deployment.

[0135]

[0135] In an optional embodiment, after the target task processing model is transferred from the first processor to the second processor for deployment, it also includes: in response to a call to the task processing model based on the task to be processed, the model parameters of the target processing layer are obtained in sequence from the first processor in accordance with the processing order of the task processing layers included in the task processing model, and transmitted to the second processor, so that the second processor calls the target processing layer to process the task to be processed.

[0136]

[0136] In summary, the above method judges the model occupied resources of the task processing model and the storage resources of the second processor based on the model attribute information of the task processing model and the device attribute information of the second processor associated with the task processing model. When it is determined that the model occupied resources of the task processing model are greater than the storage resources of the second processor, it means that the task processing model cannot be completely deployed in the second processor, then the number of unloading layers of the task processing model is determined, and the parameter unloading strategy of the task processing model is determined according to the number of unloading layers and the initial number of layers of the task processing model, so that when the task processing model is subsequently deployed, the target processing layer corresponding to the number of unloading layers in the task processing model is unloaded from the second processor to the first processor according to the parameter unloading strategy, so that the task processing model is deployed separately with the first processor and the second processor, which alleviates the storage pressure of the second processor, thereby reducing the model inference cost and improving the model inference performance, and ensuring the stability of the model during operation.

[0137]

[0137] The above is a schematic scheme of a model deployment method of this embodiment. It should be noted that the technical scheme of this model deployment method and the technical scheme of the above-mentioned model deployment strategy determination method are based on the same concept. For details not described in detail in the technical scheme of the model deployment method, please refer to the description of the technical scheme of the above-mentioned model deployment strategy determination method.

[0138]

[0138] Corresponding to the above-mentioned method embodiment, the present disclosure also provides a model deployment apparatus embodiment, which is a task processing device used to deploy a task processing model. The task processing device includes a first processor and a second processor. FIG9 shows a schematic structural diagram of a model deployment apparatus provided by one embodiment of the present disclosure. As shown in FIG9, the apparatus includes the following modules.

[0139]

[0139] The receiving module 902 is configured to receive the model deployment strategy of the task processing model, wherein the model deployment strategy is determined according to the model deployment strategy determination method provided in the first aspect of the embodiment of the present disclosure.

[0140]

[0140] The determination module 904 is configured to determine the target processing layer corresponding to the number of unloaded layers in the task processing model according to the parameter unloading strategy in the model deployment strategy, and the task processing model is initially deployed on the first processor.

[0141]

[0141] The unloading module 906 is configured to unload the target processing layer from the task processing model to obtain a target task processing model, wherein the target task processing model includes model parameters of task processing layers other than the target processing layer.

[0142]

[0142] The transmission module 908 is configured to transmit the target task processing model from the first processor to the second processor for deployment.

[0143]

[0143] In an optional embodiment, the transmission module 906 is further configured to: in response to a call to the task processing model based on the task to be processed, obtain the model parameters of the target processing layer from the first processor in sequence according to the processing order of the task processing layers included in the task processing model, and transmit them to the second processor, so that the second processor calls the target processing layer to process the task to be processed.

[0144]

[0144] In summary, the above-mentioned device judges the model occupied resources of the task processing model and the storage resources of the second processor based on the model attribute information of the task processing model and the device attribute information of the second processor associated with the task processing model. When it is determined that the model occupied resources of the task processing model are greater than the storage resources of the second processor, it means that the task processing model cannot be completely deployed in the second processor, then the number of unloading layers of the task processing model is determined, and the parameter unloading strategy of the task processing model is determined according to the number of unloading layers and the initial number of layers of the task processing model, so that when the task processing model is subsequently deployed, the target processing layer corresponding to the number of unloading layers in the task processing model is unloaded from the second processor to the first processor according to the parameter unloading strategy, so that the task processing model is deployed separately with the first processor and the second processor, which alleviates the storage pressure of the second processor, thereby reducing the model inference cost and improving the model inference performance, and ensuring the stability of the model during operation.

[0145] The above is a schematic diagram of a model deployment device according to this embodiment. It should be noted that the technical solution of this model deployment device and the technical solution of the aforementioned model deployment method are based on the same concept. For details not described in detail in the technical solution of the model deployment device, please refer to the description of the technical solution of the aforementioned model deployment method.

[0145]

[0146] FIG10 shows a block diagram of a computing device 1000 according to an embodiment of the present disclosure. Components of the computing device 1000 include, but are not limited to, a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and a database 1050 is used to store data.

[0146]

[0147] The computing device 1000 also includes an access device 1040 that enables the computing device 1000 to communicate via one or more networks 1060. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1040 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0147]

[0148] In one embodiment of the present application, the aforementioned components of computing device 1000 and other components not shown in FIG. 10 may also be connected to one another, for example, via a bus. It should be understood that the computing device structure block diagram shown in FIG. 10 is for illustrative purposes only and does not limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0148]

[0149] Computing device 1000 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1000 may also be a mobile or stationary server.

[0149]

[0150] The processor 1020 is configured to execute the following computer program / instruction, which, when executed by the processor, implements the steps of the above-mentioned model deployment strategy determination method or model deployment method.

[0150]

[0151] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device is based on the same concept as the technical solution of the aforementioned model deployment strategy determination method or model deployment method. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the aforementioned model deployment strategy determination method or model deployment method.

[0151]

[0152] An embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned model deployment strategy determination method or model deployment method when executed by a processor.

[0152]

[0153] The above is an illustrative embodiment of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium shares the same concept as the technical solution of the aforementioned model deployment strategy determination method or model deployment method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the aforementioned model deployment strategy determination method or model deployment method.

[0153]

[0154] An embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned model deployment strategy determination method or model deployment method.

[0154]

[0155] The above is an illustrative embodiment of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product shares the same concept as the technical solution of the aforementioned model deployment strategy determination method or model deployment method. For details not described in detail in the technical solution of the computer program product, reference can be made to the description of the technical solution of the aforementioned model deployment strategy determination method or model deployment method.

[0155]

[0156] The foregoing description describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0156]

[0157] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunication signals, and software distribution media. It should be noted that the content of the computer-readable medium may be appropriately increased or decreased based on the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0157]

[0158] It should be noted that, for ease of description, the aforementioned method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present disclosure are not limited by the order of the actions described. Depending on the embodiments of the present disclosure, certain steps may be performed in a different order or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules involved are not necessarily required for the embodiments of the present disclosure.

[0158]

[0159] In the above embodiments, the description of each embodiment is given with emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0159]

[0160] The preferred embodiments disclosed above are intended only to illustrate the present disclosure. The alternative embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of the embodiments disclosed. These embodiments are selected and described in detail to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize the present disclosure. The present disclosure is limited only by the claims and their full scope and equivalents.

Claims

Claims 1. A method for determining a model deployment strategy, comprising: Determine model attribute information of a task processing model and device attribute information of a second processor associated with the task processing model, wherein the task processing model is deployed on a task processing device, and the task processing device includes a first processor and a second processor; when it is determined, based on the model attribute information and the device attribute information, that the model occupied resources of the task processing model are greater than the storage resources of the second processor, determine the number of unloading layers of the task processing model; determine a parameter unloading strategy of the task processing model based on the number of unloading layers and the initial number of layers included in the model attribute information, wherein the parameter unloading strategy is a strategy for unloading a target processing layer corresponding to the number of unloading layers in the task processing model from the second processor to the first processor; determine a model deployment strategy of the task processing model based on the parameter unloading strategy.

2. The model deployment strategy determination method according to claim 1, further comprising: after determining the model attribute information of the task processing model and the device attribute information of the second processor associated with the task processing model; determining sample processing information of the task processing model; Accordingly, when it is determined, based on the model attribute information and the device attribute information, that the model occupied resources of the task processing model are greater than the storage resources of the second processor, determining the number of offload layers of the task processing model includes: calculating the model occupied resources of the task processing model based on the sample processing information and the model attribute information; Determine the storage resources of the second processor according to the device attribute information of the second processor; and if it is determined that the resources occupied by the model are greater than the storage resources of the second processor, determine the number of offload layers of the task processing model corresponding to the sample processing information.

3. The model deployment strategy determination method according to claim 2, wherein determining the number of offload layers of the task processing model corresponding to the sample processing information comprises: Calculating target storage resources according to the resources occupied by the model and the storage resources of the second processor; The number of offloaded layers of the task processing model corresponding to the sample processing information is determined according to the target storage resource and the parameter size of a single task processing layer in the task processing model.

4. The model deployment strategy determination method according to claim 2, wherein the task processing model has multiple sample processing information; accordingly, when it is determined based on the model attribute information and the device attribute information that the model occupied resources of the task processing model are greater than the memory resources of the second processor, determining the number of offload layers of the task processing model comprises: sequentially determining the plurality of sample processing information as first sample processing information; Calculate the model occupied resources of the task processing model based on the first sample processing information and the model attribute information; determine the storage resources of the second processor based on the device attribute information of the second processor; and when it is determined that the model occupied resources are greater than the storage resources of the second processor, determine the number of unloading layers of the task processing model corresponding to the first sample processing information.

5. The method for determining a model deployment strategy according to claim 1, wherein The initial number of layers included in the type attribute information is used to determine the parameter offloading strategy of the task processing model, including: Calculating the number of unloaded layers and the initial number of layers included in the model attribute information according to a preset strategy algorithm to obtain an unload interval of the task processing model; A parameter offloading strategy of the task processing model is determined according to the number of offloading layers and the offloading interval.

6. The model deployment strategy determination method according to claim 2, further comprising: determining a transmission time for transmitting a model parameter of a single task processing layer in the task processing model corresponding to the sample processing information from the second processor to the first processor; Determining a calling time of the single task processing layer corresponding to the sample processing information; Calculate task processing information of the task processing model corresponding to the sample processing information according to the transmission time, the call time, the initial number of layers, and the number of unloaded layers.

7. The method for determining a model deployment strategy according to claim 6, wherein determining the model deployment strategy of the task processing model according to the parameter offloading strategy comprises: A model deployment strategy of the task processing model is determined according to the parameter offloading strategy and the task processing information.

8. The model deployment strategy determination method according to claim 1 or 7, after determining the model deployment strategy of the task processing model, further comprising: The task processing model is deployed in the task processing device according to the model deployment strategy.

9. The method for determining a model deployment strategy according to claim 8, wherein deploying the task processing model in the task processing device according to the model deployment strategy comprises: Determine, according to the parameter offloading strategy in the model deployment strategy, a target processing layer in the task processing model corresponding to the number of offloaded layers, and initially deploy the task processing model on the first processor; offload the target processing layer from the task processing model to obtain a target task processing model, wherein the target task processing model includes model parameters of task processing layers other than the target processing layer; and transfer the target task processing model from the first processor to the second processor for deployment.

10. A model deployment method, applied to a task processing device for deploying a task processing model, the task processing device comprising a first processor and a second processor, the method comprising: Receive a model deployment strategy for the task processing model, wherein the model deployment strategy is determined according to the model deployment strategy determination method according to any one of claims 1 to 9; determine a target processing layer in the task processing model corresponding to the number of unloaded layers according to a parameter unloading strategy in the model deployment strategy, and initially deploy the task processing model on the first processor; unload the target processing layer from the task processing model to obtain a target task processing model, wherein the target task processing model includes model parameters of task processing layers other than the target processing layer; and transfer the target task processing model from the first processor to the second processor for deployment.

11. The model deployment method according to claim 10, further comprising: after transferring the target task processing model from the first processor to the second processor for deployment; In response to calling the task processing model based on the task to be processed, the model parameters of the target processing layer are sequentially obtained from the first processor according to the processing order of the task processing layers included in the task processing model, and transmitted to the second processor, so that the second processor calls the target processing layer to process the task to be processed. Service.

12. A computing device comprising: memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the model deployment strategy determination method according to any one of claims 1 to 9 or the model deployment method according to claim 10 or 11 are implemented.

13. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the model deployment strategy determination method according to any one of claims 1 to 9 or the model deployment method according to claim 10 or 11.

14. A computer program product comprising a computer program / instructions, which, when executed by a processor, implements the steps of the model deployment strategy determination method according to any one of claims 1 to 9 or the model deployment method according to claim 10 or 11.

Images