Model deployment scheme generation method and device, model processing method and device and electronic equipment

Abstract

The embodiment of the invention provides a model deployment scheme generation method and device, a model processing method and device and electronic equipment, and relates to the technical field of model deployment, large language models and the like. According to the specific implementation scheme, processor parameters of all processors in target equipment are obtained, wherein the target equipment is used for deploying a to-be-deployed model; based on the model information and the processor parameters, a deployment scheme is generated, the deployment scheme comprises first specified information, the to-be-deployed model comprises a plurality of processing layers, and the first specified information is used for specifying processors deployed by model weights of the processing layers. Based on the deployment scheme provided by the embodiment of the invention, reasonable deployment of processor resources in the target equipment can be realized, so that reasonable deployment of the model in the target equipment is realized.

Description

Technical Field

[0001] The present application relates to technical fields such as model deployment and large language models. Specifically, the present application relates to a model deployment solution generation, model processing method, device and electronic device. Background Art

[0002] In recent years, with the rapid development of deep learning and artificial intelligence technology, the capabilities of artificial intelligence models have become increasingly powerful, and the scale of artificial intelligence models has also become larger and larger.

[0003] At present, large-scale artificial intelligence models are mainly deployed on servers with powerful processing capabilities, but it is difficult to deploy them on other devices. Therefore, how to deploy the model reasonably has become an important technical issue. Summary of the invention

[0004] This application provides a model deployment solution generation, model processing method, device and electronic device, aiming to solve at least one of the above technical defects. The technical solution adopted by this application is as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for generating a model deployment solution, the method comprising:

[0006] Get the model information of the model to be deployed;

[0007] Obtain processor parameters of each processor in a target device, where the target device is used to deploy the model to be deployed;

[0008] Based on the model information and processor parameters, a deployment plan is generated, where the deployment plan includes first specified information. The model to be deployed includes multiple processing layers, and the first specified information is used to specify the processor where the model weight of each processing layer is deployed.

[0009] In a second aspect, an embodiment of the present application provides a model deployment method, the method comprising:

[0010] Obtaining a deployment plan for the model to be deployed, the deployment plan including first specified information, the model to be deployed including multiple processing layers, the target device for deploying the model to be deployed including multiple processors, the first specified information is used to specify the processors to which the model weights of each processing layer are deployed, and the model deployment plan is generated by the server based on the model information of the model to be deployed and the processor parameters of the processors;

[0011] Based on the first specified information, the model weights of each processing layer are deployed to a specified processor in the target device.

[0012] In a third aspect, an embodiment of the present application provides a model deployment solution generation device, the device comprising:

[0013] A model information acquisition module is used to obtain model information of the model to be deployed;

[0014] A processor parameter acquisition module is used to obtain processor parameters of each processor in a target device, and the target device is used to deploy the model to be deployed;

[0015] A deployment plan generation module is used to generate a deployment plan based on model information and processor parameters. The deployment plan includes first specified information. The model to be deployed includes multiple processing layers. The first specified information is used to specify the processor where the model weight of each processing layer is deployed.

[0016] In a fourth aspect, an embodiment of the present application provides a model processing device, the device comprising:

[0017] A deployment scheme acquisition module is used to acquire a deployment scheme for a model to be deployed, wherein the deployment scheme includes first specified information, the model to be deployed includes multiple processing layers, and the target device for deploying the model to be deployed includes multiple processors. The first specified information is used to specify the processors to which the model weights of each processing layer are deployed. The model deployment scheme is generated by the server based on the model information of the model to be deployed and the processor parameters of the processors.

[0018] The model deployment module is used to deploy the model weights of each processing layer to a specified processor in the target device based on the first specified information.

[0019] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0020] In a sixth aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0021] one or more processors; and

[0022] A memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, execute the steps of the method.

[0023] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0024] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0025] The solution provided by the embodiment of the present application obtains the model information of the model to be deployed, obtains the processor parameters of each processor in the target device for deploying the model to be deployed, and generates a deployment solution based on the model information and the processor parameters, the deployment solution including first specified information for specifying the processors deployed by the model weights of each processing layer and second specified information for specifying the processors that perform the model calculations corresponding to each processing layer. Based on the deployment solution provided by the embodiment of the present application, it is possible to achieve reasonable allocation of processor resources in the target device, thereby achieving reasonable deployment of the model on the target device. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0027] Figure 1 A system architecture diagram applicable to the embodiments of the present application;

[0028] Figure 2 A flowchart of a method for generating a model deployment solution provided in an embodiment of the present application;

[0029] Figure 3 A flowchart of a specific implementation method of the model deployment solution generation method provided in an embodiment of the present application;

[0030] Figure 4 This is a schematic diagram of the process of model weight deployment in an embodiment of the present application;

[0031] Figure 5 This is a schematic diagram of the flow chart of the model inference calculation in the embodiment of the present application;

[0032] Figure 6 A flow chart of a model processing method provided in an embodiment of the present application;

[0033] Figure 7 A schematic diagram of the structure of a model deployment solution generation device provided in an embodiment of the present application;

[0034] Figure 8 A schematic diagram of the structure of a model processing device provided in an embodiment of the present application;

[0035] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0037] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0038] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0039] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0040] First, let’s introduce and explain several terms involved in this application:

[0041] Large Language Model (LLM) is a deep learning model used for natural language related tasks. It can receive text content input and return corresponding output. The specific tasks that can be completed include text generation, classification, summarization, rewriting, etc.

[0042] When processing natural language tasks, LLM can be divided into two different stages: understanding stage and generation stage.

[0043] In the understanding phase, LLM "understands" the meaning of the text input by the user through calculation, which is manifested by calculating the state tensor (KV-Cache) inside LLM and the first element (token) of the prediction.

[0044] In the generation phase, LLM uses the state tensor calculated in the language understanding phase as a parameter to continuously update the KVcache and generate a new next token.

[0045] The overall processing flow of LLM in the understanding stage and the generation stage is the same. The difference between the two stages is that the understanding stage processes multiple or all input data at a time and generates the first token, while the generation stage processes one token at a time.

[0046] The central processing unit (CPU) is a large-scale integrated circuit, which is the computing core and control core of a computer. Its main function is to interpret computer instructions and process data in computer software.

[0047] A graphics processing unit (GPU) is a processor specifically used for image and video processing. GPU was first used in the field of computer games and is currently widely used in the fields of big data analysis and artificial intelligence.

[0048] A neural network processor (NPU) is a specialized processor designed specifically for the computing needs of neural network models. It is designed to perform machine learning, especially deep learning tasks, efficiently and with low power consumption. As an artificial intelligence (AI) accelerator, NPU is a customized circuit for AI acceleration and contains some necessary control units and algorithms to execute deep learning algorithms.

[0049] A digital signal processor (DSP) is a microprocessor dedicated to (usually real-time) digital signal processing. The DSP chip uses a Harvard structure that separates program and data, has a dedicated hardware multiplier, widely uses pipeline operations, and provides special DSP instructions to quickly implement various digital signal processing algorithms. The characteristics of DSP chips include the ability to perform multiple operations in parallel, which makes DSP perform well when processing digital signals that require high-speed mathematical operations. Currently, DSP is also often used to perform deep learning model reasoning.

[0050] At present, the ability of artificial intelligence models to handle complex tasks is getting stronger and stronger. As a result, the number of model parameters is getting larger and larger, and the model's demand for computing power is also increasing, which greatly increases the demand for equipment when deploying models. At present, large-scale models are mainly deployed on servers with powerful processing capabilities, but there are difficulties in deploying them on other devices. Therefore, how to deploy the model reasonably has become an important technical issue.

[0051] Take the Large Language Model (LLM), which has made breakthrough progress in the field of natural language processing in recent years, as an example. The number of parameters of LLM is huge, usually including tens of billions or hundreds of billions of parameters, which makes LLM mainly deployed on servers with powerful graphics processing units (GPUs). In this deployment mode, users need to request LLM services through the network, which will lead to the following problems: (1) LLM services cannot be provided in time in a poor network environment or offline environment; (2) There are certain information security risks in network transmission.

[0052] Deploying LLM on the end-side device can solve the above problems caused by network requests for LLM services. However, the types of end-side devices are diverse, and the performance of processors in end-side devices is also uneven. The performance parameters of some processors cannot fully meet the requirements of deploying LLM, which makes it difficult to deploy LLM on the end-side device.

[0053] In view of this, the present application provides a new idea. In order to facilitate the understanding of the present application, the system architecture on which the present application is based is first described. Figure 1 An exemplary system architecture to which the embodiments of the present application can be applied is shown. Figure 1 As shown in , the system architecture may include: a target device and a server.

[0054] Among them, the deployment plan generation device on the server side can adopt the method provided in the embodiment of the present application to generate a deployment plan based on model information and processor parameters in the offline stage.

[0055] The target device can exchange information with the server through the network. For example, the target device can send processor parameters to the deployment plan generation device. The server can send model resources and deployment plans to the target device when model deployment is required. The model processing device of the target device can implement model deployment based on the deployment plan.

[0056] The deployment solution generation device can be set on a server or a server group, or on an independent or the same cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in a cloud computing service system that solves the defects of difficult management and weak service scalability in traditional physical hosts and virtual private servers (VPS) services.

[0057] It should be understood that Figure 1The number of target devices, deployment solution generating devices, and model processing devices in the embodiment is only illustrative. According to the implementation requirements, there may be any number of target devices, deployment solution generating devices, and model processing devices.

[0058] Figure 2 A schematic diagram of a flow chart of a method for generating a model deployment solution provided in an embodiment of the present application is shown. The method can be Figure 1 The deployment plan generation device in the system shown is executed. Figure 2 As shown, the method may mainly include:

[0059] Step 210: Obtain model information of the model to be deployed;

[0060] Step 220: Obtain processor parameters of each processor in a target device, where the target device is used to deploy the model to be deployed;

[0061] Step 230: Generate a deployment plan based on the model information and processor parameters, the deployment plan includes first specified information, the model to be deployed includes multiple processing layers, and the first specified information is used to specify the processor where the model weight of each processing layer is deployed.

[0062] The model to be deployed may include but is not limited to a large language model. The model to be deployed may include multiple processing layers, and the model weight of the model to be deployed is composed of the model weights corresponding to each processing layer. Based on the model weights corresponding to each processing layer, the model calculation corresponding to each processing layer can be performed.

[0063] The model information of the model to be deployed can reflect the processor performance requirements of the model to be deployed during deployment.

[0064] In the embodiment of the present application, the model to be deployed will be deployed in the target device. The target device may include but is not limited to the end-side device, the edge-side device, the terminal device, etc. As a preferred embodiment, the target device in this case is the end-side device, and the model service is run through the end-side device.

[0065] The target device includes multiple processors. The processors in the target device can be of various types, including but not limited to CPU, GPU, NPU, DSP, etc. The processor parameters can reflect the hardware performance of the processor.

[0066] In the embodiment of the present application, the model to be deployed can be run on the target device in the form of an executable program. The model deployment involved in the embodiment of the present application mainly refers to loading and storing the model weights into the processor of the target device.

[0067] Based on the model information and processor parameters, it is possible to reasonably allocate the processors in the target device according to the requirements of each processing layer of the model to be deployed and the performance of each processor in the target device, so that the model to be deployed can be reasonably deployed on the target device.

[0068] Specifically, the model weights of each processing layer are mainly considered during deployment based on the storage space of the processor. Therefore, by generating the first designated information, the model weights of each processing layer can be deployed to a processor whose storage space meets the requirements.

[0069] In the embodiment of the present application, by generating a deployment plan, effective allocation of resources within the processor in the target device can be achieved, which helps to achieve reasonable deployment of the model to be deployed on the target device.

[0070] In an embodiment of the present application, a deployment plan can be generated offline, and then the deployment plan can be sent to a target device, so that the target device deploys the model to be deployed based on the deployment plan.

[0071] The method provided in the embodiment of the present application obtains the model information of the model to be deployed, obtains the processor parameters of each processor in the target device for deploying the model to be deployed, and then generates a deployment plan based on the model information and the processor parameters, and the deployment plan includes first specified information for specifying the processors deployed by the model weights of each processing layer. Based on the deployment plan provided in the embodiment of the present application, it is possible to achieve reasonable allocation of processor resources in the target device, thereby achieving reasonable deployment of the model on the target device.

[0072] In an optional manner of an embodiment of the present application, the deployment plan also includes second specified information, and the second specified information is used to specify a processor that executes model calculations corresponding to each processing layer.

[0073] When executing the model calculations corresponding to each processing layer, the main consideration is the computing power of the processor. Therefore, by generating the second designated information, a processor with computing power that meets the requirements can be deployed to execute the model calculations corresponding to different processing layers.

[0074] It can be understood that, for any processing layer of the model to be deployed, the processor used to deploy the model weights corresponding to the processing layer may be the same as or different from the processor used to execute the model calculation of the processing layer.

[0075] Based on the deployment solution provided in the embodiment of the present application, it is possible to reasonably allocate the storage resources and computing resources of the target device, thereby realizing the reasonable deployment of the model on the target device.

[0076] In the embodiment of the present application, the second designated information can be generated on the server side and sent to the target device, so as to reasonably designate the processor to execute the model of each processing layer. The second designated information can also be converted into execution logic, which is automatically run on the target device to realize automatic selection of the processor to execute the model of each processing layer.

[0077] In an optional manner of an embodiment of the present application, the second specified information includes second specified information in the understanding stage, and the second specified information in the understanding stage is used to specify a processor that performs model calculations corresponding to each processing layer in the understanding stage, and generates a deployment plan based on the model information and processor parameters, including:

[0078] For any processing layer, determine a first duration corresponding to the first processor and a second duration corresponding to the second processor, wherein the model weight corresponding to the processing layer is deployed on the first processor, the computing power of the second processor is higher than that of the first processor, the first duration is the duration for the first processor to perform the model calculation corresponding to the processing layer, and the second duration is the sum of the transmission duration and the calculation duration, the transmission duration is the duration for the data required to perform the model calculation of the processing layer to be transmitted to the second processor, and the calculation duration is the duration for the second processor to perform the model calculation corresponding to the processing layer;

[0079] Based on the first duration and the second duration, second specified information of the understanding stage is determined.

[0080] In the embodiment of the present application, in addition to being a large language model, the model to be deployed can also be other types of large models whose processing process includes understanding and generation stages, including but not limited to a text-based image large model, a text-based video large model, etc.

[0081] In an embodiment of the present application, the requirements for processor performance in the understanding stage and the generation stage of the model to be deployed are different. Based on the requirements of the model to be deployed in the understanding stage, second specified information for the understanding stage can be generated to specify the processor that executes the model calculations corresponding to each processing layer in the understanding stage.

[0082] In the understanding phase of the model to be deployed, multiple or all of the data to be processed will be input at a time, and all of the input data to be processed will be processed at once, which has certain requirements for processor storage space and computing power. In actual situations, some processors of the target device may have poor computing power, while other processors may have insufficient storage space. In response to these situations, by reasonably specifying the processors that perform model calculations corresponding to each processing layer in the understanding phase, the model calculation efficiency can be effectively improved.

[0083] Specifically, for any processing layer, the processor that deploys the model weights corresponding to the processing layer can be recorded as the first processor, and the time taken by the first processor to process the model calculation corresponding to the processing layer is the first time. The second processor is a processor in the target device with a computing power higher than that of the first processor. When the second processor processes the model calculation corresponding to the processing layer, it is necessary to first transfer the data required for the model calculation of the processing layer to the second processor, and the time required for this data transmission is recorded as the transmission time. The second time corresponding to the second processor is actually the sum of the transmission time and the calculation time of the second processor to perform the model calculation corresponding to the processing layer.

[0084] In the embodiment of the present application, based on the first duration and the second duration, a processor that can more efficiently complete the corresponding processing process of the processing layer can be determined, thereby effectively generating the second designated information of the understanding stage.

[0085] In an optional method of an embodiment of the present application, the second specified information includes the second specified information of the generation stage, and the second specified information of the generation stage is used to specify that the processor that executes the model calculation corresponding to each processing layer in the generation stage is the first processor, and the first processor is the processor deployed for the model weights corresponding to each processing layer.

[0086] In an embodiment of the present application, during the generation phase of the model to be deployed, a token is input and processed each time. The generation phase has relatively low requirements for processor computing power. Therefore, the second specified information of the generation phase can be used to specify the use of the first processor that deploys the model weights corresponding to a certain processing layer to perform the model calculation of the processing layer. In other words, the second specified information of the generation phase indicates that during the generation phase, there is no need to transfer the model weights to a processor with higher computing power, nor is there a need to call a processor with higher computing power to perform the model calculation of the processing layer.

[0087] In the embodiment of the present application, since the generation stage inputs and processes one token each time, if the processor that executes the model calculation of each processing layer is still determined in the understanding stage, the model weight will be transmitted multiple times, resulting in a large amount of time spent on the transmission of the model weight, affecting the model processing efficiency. Considering that the demand for processor computing power in the generation stage is relatively low, the processor used for model deployment can generally meet the demand for processor computing power in the generation stage. Therefore, the first processor corresponding to the model weight of a certain processing layer can be specified to perform the model calculation of the processing layer.

[0088] In an optional manner of the embodiment of the present application, the processor parameters include the memory information of the processor, the model information includes the data volume of the model weight corresponding to each processing layer, and based on the model information and the processor parameters, a deployment plan is generated, including:

[0089] The first specified information is determined based on the memory information and the data volume of the model weights corresponding to each processing layer.

[0090] In an embodiment of the present application, the processor parameters may include memory information, and the memory information can reflect the ability of the processor to deploy model weights. Therefore, the first specified information can be determined based on the memory information of the processor and the amount of data of the model weights corresponding to each processing layer, that is, the processor used to deploy the model weights corresponding to each processing layer is determined.

[0091] In an embodiment of the present application, based on the memory information and the amount of data of the model weights corresponding to each processing layer, it is possible to determine whether the storage space of the processor can support the deployment of the model weight files of each processing layer, thereby determining the first specified information.

[0092] As an example, the computing power of different processors can be analyzed, and processors with higher computing power can be used to deploy the model weights of each processing layer. After the storage space of the processor with higher computing power is fully used, consider using models with lower computing power to deploy model weights.

[0093] In the embodiment of the present application, when the model weights of each processing layer are deployed on the processor in the target device based on the first specified information, the storage address of the model weights in the memory buffer can be set, and the processor identifier can be used as the attribute information of the model weights stored in the memory buffer. In this way, during the model calculation process, the processor to which the model weights belong can be determined based on the attributes of the data in the memory buffer.

[0094] In an optional manner of the embodiment of the present application, determining the first specified information based on the memory information and the data amount of the model weights corresponding to each processing layer includes:

[0095] Get the historical memory usage of each processor;

[0096] Determine available memory based on historical memory usage and memory information;

[0097] The first specified information is determined based on the available memory and the data volume of the model weights corresponding to each processing layer.

[0098] In the embodiment of the present application, the historical memory usage of the processor can reflect the memory usage in the processor. Based on the historical memory usage and memory information, the available memory in the processor that can be used for model weight deployment can be determined.

[0099] Since the target device may be in different application scenarios, based on the processor's historical memory usage, we can understand the processor's usual memory usage in application scenarios, and avoid allocating too much memory resources to deploy model services, causing the model service to compete with other services for memory resources and resulting in reduced processing performance.

[0100] As an example, when the target device contains multiple processors, if, based on the historical memory usage of a certain processor, it is determined that the memory resources in the processor are being used at a high load and cannot stably support the normal deployment and operation of the model, the processor can be disabled and other processors in the target device can be used to deploy the model.

[0101] In an optional manner of the embodiment of the present application, the processor parameters are collected based on a preset test program running on the target device.

[0102] In the embodiment of the present application, a preset test program can be run on the target device to obtain processor parameters of each processor. The preset test program may include but is not limited to a benchmark program, and the collected processor parameters may include the computing power, transmission bandwidth, and storage space of the processor.

[0103] As an example, the operating system of the target device may provide relevant application programming interfaces (APIs), through which the processor parameters may be obtained.

[0104] As an example, Figure 3 A schematic diagram of a specific implementation of the model deployment solution generation method provided in an embodiment of the present application is shown.

[0105] like Figure 3 As shown in , the model deployment solution generation method in this example includes the following steps:

[0106] Step S310: loading model information and processor parameters;

[0107] Step S320: the offline planner generates a deployment plan;

[0108] Step S330: Output the deployment plan.

[0109] In the embodiment of the present application, before deploying the large model, an offline planner can be run in an offline environment. The offline planner can read processor parameters through an API provided by the operating system of the target device.

[0110] After the offline planner loads the model information and processor parameters, it generates a deployment plan based on the model information and processor parameters, and formats and outputs the deployment plan, that is, outputs a deployment plan that can be directly used by the large language model inference framework.

[0111] As an example, Figure 4 A schematic diagram of the process of model weight deployment in an embodiment of the present application is shown, Figure 4 As shown in , the process includes the following steps:

[0112] Step S410: loading the model;

[0113] Step S420: Loading the deployment plan;

[0114] Step S430: Determine the model weight deployment strategy based on the deployment plan;

[0115] Step S440: According to the model weight deployment strategy, the model weights of each processing layer are loaded into the designated processor respectively, and the corresponding memory cache area is initialized in the processor, and the properties of the memory buffer are set.

[0116] In an embodiment of the present application, by loading the model, the model to be deployed runs on the target device in the form of an executable program. The target device can load the deployment plan and determine the model weight deployment strategy from the deployment plan. The model weight deployment strategy is equivalent to the aforementioned first specified information. Based on the first specified information, the model weights of each processing layer can be loaded into the specified processor respectively. At the same time, the corresponding memory cache area can be initialized in the processor for storing the model weights, and the attributes of the memory buffer can be set to the identification of each processor.

[0117] As an example, Figure 5 The schematic diagram of the flow chart of the model inference calculation in the embodiment of the present application is shown as follows: Figure 5 As shown in , the process includes the following steps:

[0118] Step S510: Loading model weights of each processing layer;

[0119] Step S520: for the current processing layer, determining a processor X for executing the model calculation of the current processing layer;

[0120] Step S530: Whether the model weights of the current processing layer are deployed on processor X;

[0121] Step S540: calling processor X to perform model calculation of the current processing layer;

[0122] Step S550: transmitting the weight of the current processing layer from processor Y to processor X, where the model weight of the current processing layer is deployed;

[0123] Step S560: calling processor X to perform model calculation of the current processing layer;

[0124] Step S570: The model calculation result of the current processing layer is transmitted to the processor that performs the model calculation of the next processing layer;

[0125] Step S580: Model calculation processing of all processing layers is completed;

[0126] Step S590: End.

[0127] In the embodiment of the present application, processor X and processor Y are different processors in the target device.

[0128] The model weights corresponding to each processing layer of the model to be deployed will be deployed on the specified processor. For the current processing layer that is about to perform model calculation, the processor X used to perform the model calculation of the current processing layer can be determined.

[0129] Determine whether the model weights of the current processing layer are deployed on processor X. If the model weights of the current processing layer are deployed on processor X, processor X can directly perform the model calculation of the current processing layer. If the model weights of the current processing layer are not deployed on processor X, such as deployed on processor Y, the weights of the current processing layer can be transferred from processor Y to processor X, and then processor X is called to perform the model calculation of the current processing layer.

[0130] After completing the model calculation of the current processing layer, the model calculation result of the current processing layer can be transmitted to the processor that executes the model calculation of the next processing layer for subsequent model calculation.

[0131] Determine whether the model calculations of all processing layers have been completed. If the model calculations of all processing layers have been completed, the process ends; if the model calculations of all processing layers have not been completed, steps S520 to S570 are performed in sequence according to the order of the processing layers.

[0132] Figure 6 A schematic diagram of a model processing method provided in an embodiment of the present application is shown. The method can be performed by Figure 1 The model processing device in the system shown is executed. Figure 6 As shown, the method may mainly include:

[0133] Step S610: obtaining a deployment scheme for the model to be deployed, the deployment scheme including first specified information, the model to be deployed including multiple processing layers, the target device for deploying the model to be deployed including multiple processors, the first specified information is used to specify the processors to which the model weights of each processing layer are deployed, and the model deployment scheme is generated by the server based on the model information of the model to be deployed and the processor parameters of the processors;

[0134] Step S620: Based on the first specified information, the model weights of each processing layer are deployed to the specified processor in the target device.

[0135] The model to be deployed may include but is not limited to a large language model. The model to be deployed may include multiple processing layers, and the model weight of the model to be deployed is composed of the model weights corresponding to each processing layer. Based on the model weights corresponding to each processing layer, the model calculation corresponding to each processing layer can be performed.

[0136] In the embodiment of the present application, the model to be deployed will be deployed in the target device. As a preferred embodiment, the target device in this case is a terminal device, and the model service is run through the terminal device.

[0137] The target device includes multiple processors. The processors in the target device can be of various types, including but not limited to CPU, GPU, NPU, DSP, etc. The processor parameters can reflect the hardware performance of the processor.

[0138] In the embodiment of the present application, the model to be deployed can be run on the target device in the form of an executable program. The model deployment involved in the embodiment of the present application mainly refers to loading and storing the model weights into the processor of the target device.

[0139] In an embodiment of the present application, the server side can reasonably allocate the processors in the target device based on the requirements of each processing layer of the model to be deployed and the performance of each processor in the target device based on the model information and processor parameters, so that the model to be deployed can be reasonably deployed on the target device.

[0140] Specifically, the model weights of each processing layer are mainly considered during deployment based on the storage space of the processor. Therefore, based on the first specified information in the deployment plan, the model weights of each processing layer can be deployed to a processor whose storage space meets the requirements.

[0141] In the embodiment of the present application, completing the model deployment based on the deployment plan can achieve effective allocation of processor resources in the target device, which helps to achieve reasonable deployment of the model to be deployed on the target device.

[0142] The method provided by the embodiment of the present application, by obtaining a deployment scheme for the model to be deployed, the deployment scheme includes first specified information and second specified information, the model to be deployed includes multiple processing layers, the target device for deploying the model to be deployed includes multiple processors, the first specified information is used to specify the processor where the model weights of each processing layer are deployed, the second specified information is used to specify the processor that performs the model calculation corresponding to each processing layer, and the model deployment scheme is generated by the server based on the model information of the model to be deployed and the processor parameters of the processor. Based on the first specified information, the model weights of each processing layer are deployed to the specified processor in the target device, and the model calculation of the model to be deployed is performed based on the second specified information. Model deployment based on the deployment scheme provided by the embodiment of the present application can achieve the reasonable allocation of processor resources in the target device, thereby achieving the reasonable deployment of the model on the target device.

[0143] In an embodiment of the present application, the deployment plan may also include second specified information, and the second specified information is used to specify a processor that executes model calculations corresponding to each processing layer.

[0144] When executing the model calculations corresponding to each processing layer, the main consideration is the computing power of the processor. Therefore, by generating the second designated information, a processor with computing power that meets the requirements can be deployed to execute the model calculations corresponding to different processing layers.

[0145] It can be understood that, for any processing layer of the model to be deployed, the processor used to deploy the model weights corresponding to the processing layer may be the same as or different from the processor used to execute the model calculation of the processing layer.

[0146] Based on the deployment solution provided in the embodiment of the present application, it is possible to reasonably allocate the storage resources and computing resources of the target device, thereby realizing the reasonable deployment of the model on the target device.

[0147] In the embodiment of the present application, the second designated information can be generated on the server side and sent to the target device, so as to reasonably designate the processor to execute the model of each processing layer. The second designated information can also be converted into execution logic, which is automatically run on the target device to realize automatic selection of the processor to execute the model of each processing layer.

[0148] In an optional implementation manner of the present application, after deploying the model weights of each processing layer to a specified processor in the target device, the method further includes:

[0149] For any processing layer, determining a current processor that performs model calculations of the processing layer in the understanding phase;

[0150] In response to the current processor being the first processor, reading the model weight corresponding to the processing layer from the memory buffer of the first processor, and performing the model calculation of the processing layer in the understanding phase based on the model weight corresponding to the processing layer, the first processor being a processor for deploying the model weight corresponding to the processing layer;

[0151] In response to the current processor not being the first processor, the model weights corresponding to the processing layer are transferred from the memory buffer of the first processor to the memory buffer of the current processor, the model weights corresponding to the processing layer are read from the memory buffer of the current processor, and the model calculations of the processing layer in the understanding phase are performed based on the model weights corresponding to the processing layer.

[0152] In the embodiment of the present application, in addition to being a large language model, the model to be deployed can also be other types of large models whose processing process includes understanding and generation stages, including but not limited to a text-based image large model, a text-based video large model, etc.

[0153] In the understanding phase of the model to be deployed, multiple or all of the data to be processed will be input at a time, and all of the input data to be processed will be processed at once, which places certain requirements on the storage space and computing power of the processor. In actual situations, some processors of the target device may have poor computing power, while other processors may have insufficient storage space. In view of these situations, if the processors corresponding to the model calculations of each processing layer in the understanding phase can be reasonably selected, the model calculation efficiency can be effectively improved.

[0154] In the embodiment of the present application, before performing model calculation of any processing layer, the current processor for executing the model calculation of the processing layer in the understanding phase can be determined. By reasonably selecting the current processor, its computing power can meet the computing power requirements of the processing layer, thereby improving the model calculation efficiency.

[0155] If the current processor is the first processor that deploys the model weights corresponding to the processing layer, it means that using the first processor to execute the model calculation corresponding to the processing layer can have a better processing speed. At this time, the first processor can be directly called to execute the model calculation corresponding to the processing layer.

[0156] If the current processor is the second processor, it means that using a second processor with higher computing power to execute the model calculation corresponding to the processing layer can have a better processing speed. At this time, the model weight corresponding to the processing layer can be transmitted to the second processor first, and then the second processor is called to execute the model calculation of the processing layer in the understanding stage.

[0157] In an optional implementation manner of the present application, the model weight corresponding to the processing layer is transferred from the memory buffer of the first processor to the memory buffer of the current processor, including:

[0158] Before calling the current processor to execute the model calculation of the processing layer, the current processor is controlled to start a transmission task, where the transmission task is used to transmit the model weights of the processing layer to the memory buffer of the current processor.

[0159] In an embodiment of the present application, for any processing layer, when the model weights corresponding to the processing layer are not stored in the memory of the current processor, it is necessary to execute a transmission task of transferring the model weights corresponding to the processing layer from the memory buffer of other processors to the memory buffer of the current processor. At this time, the transmission task can be started before the current processor executes the model calculation of the processing layer. This greatly reduces the additional waiting for the transmission of the model weights when the processing flow of the model to be deployed is transferred to the processing layer, thereby quickly performing the model calculation of the processing layer.

[0160] In an embodiment of the present application, by controlling the current processor to transmit the model weights in advance and reasonably setting the start time of the transmission, the transmission duration of the model weights can be effectively hidden, thereby greatly improving the processing efficiency of the model.

[0161] As an example, the transmission task can be controlled to be completed before the current processor executes the model calculation of the processing layer. By reasonably estimating the transmission time and setting the start time of the model weight, it is ensured that the transmission task is completed before the model calculation of the processing layer is executed, and the model weight of the processing layer has been stored in the memory buffer of the current processor. This ensures that when the processing flow of the model to be deployed is transferred to the processing layer, there is no need to wait for the model weight transmission, and the model calculation of the processing layer can be quickly performed.

[0162] As an example, the model calculations of the third and fourth processing layers of the model to be deployed are both performed by the current processor. The model weights of the third processing layer are deployed on the current processor, and the weights of the fourth processing layer are deployed on other processors. When performing model calculations on the third processing layer, the task of transmitting the model weights is started asynchronously, so that the transmission time of the model weights can be effectively hidden in the model calculations on the third processing layer, greatly improving the processing efficiency of the model. In this example, if the transmission time is long, the task of transmitting the model weights can also be started at an earlier time point before performing the model calculations on the third processing layer, so as to ensure that the model weights can be transmitted to the current processor in a timely manner, so as to ensure that the current processor can quickly perform the model calculations on this processing layer.

[0163] In an optional implementation manner of the present application, the above method further includes:

[0164] After the model calculation of the processing layer is completed, the model weights of the processing layer are moved out of the memory buffer of the current processor.

[0165] In an embodiment of the present application, after the current processor completes the model calculation of the processing layer, the model weights can be moved out of the memory buffer of the current processor in a timely manner, thereby releasing the memory space of the current processor.

[0166] As an example, for a processor in a target device, the memory buffer of the processor may include a static storage area and a dynamic storage area, wherein the static storage area is used to statically store the model weights stored in the processor memory buffer during initial deployment. The dynamic storage area is used to dynamically swap in and out the model weights, that is, to store the model weights transmitted to the processor memory buffer during the model operation process, and after the corresponding model calculation is completed, the corresponding model weights in the dynamic storage area are removed.

[0167] In an embodiment of the present application, when the target device is executing model processing, the model weights can be transmitted between the processor used to deploy the model weights and the processor used to execute the model calculation, so that the model calculation process of each processing layer can be constructed in the form of a pipeline composed of multiple processors, automatically performing the process of dynamically passing in and out the model weights, and effectively hiding the time-consuming transmission of the model weights, thereby improving the processing efficiency of the model.

[0168] In the embodiment of the present application, by providing a deployment plan and allocating processors for calculations at each processing layer, the processing efficiency of the model can be effectively improved without modifying the structure of the model itself, and has a wider applicability.

[0169] In an optional implementation manner of the present application, determining a current processor for executing model calculation of the processing layer in the understanding phase includes:

[0170] Determine a first duration corresponding to the first processor and a second duration corresponding to the second processor, wherein the computing power of the second processor is higher than the computing power of the first processor, the first duration is the duration for the first processor to perform the model calculation corresponding to the processing layer, and the second duration is the sum of the transmission duration and the calculation duration, the transmission duration is the duration for data required to perform the model calculation of the processing layer to be transmitted to the second processor, and the calculation duration is the duration for the second processor to perform the model calculation corresponding to the processing layer;

[0171] Based on the first duration and the second duration, a current processor that executes the model calculation of the processing layer in the understanding phase is determined.

[0172] In the embodiment of the present application, for any processing layer, a first duration used by the first processor to process the model calculation corresponding to the processing layer and a second duration used by the second processor to process the model calculation corresponding to the processing layer can be determined respectively.

[0173] The second processor is a processor with higher computing power than the first processor. When the second processor processes the model calculation corresponding to the processing layer, it needs to first transmit the data required for the model calculation of the processing layer to the second processor, and the time required for the data transmission is recorded as the transmission time. The second time corresponding to the second processor is actually the sum of the transmission time and the calculation time of the second processor performing the model calculation corresponding to the processing layer.

[0174] In the embodiment of the present application, based on the first duration and the second duration, a processor that can more efficiently complete the processing process corresponding to the processing layer can be determined.

[0175] In an optional implementation manner of the present application, determining a current processor for executing model calculation of the processing layer in the understanding phase based on the first duration and the second duration includes:

[0176] In response to the first duration being not greater than the second duration, determining the first processor as a current processor for executing the model calculation of the processing layer in the understanding phase;

[0177] In response to the first duration being greater than the second duration, the second processor is determined as the current processor for executing the model calculation of the processing layer in the understanding phase.

[0178] In an embodiment of the present application, when the first time length is not greater than the second time length, it means that when the first processor executes the model calculation of the processing layer, the processing time is shorter and the processing efficiency is higher. At this time, the first processor can be used as the current processor to execute the model calculation of the processing layer.

[0179] When the first duration is greater than the second duration, it means that when the second processor executes the model calculation of the processing layer, the processing time is shorter and the processing efficiency is higher. At this time, the second processor can be used as the current processor to execute the model calculation of the processing layer.

[0180] In an embodiment of the present application, the second processor is used as the current processor to execute the model calculation of the processing layer. It can be understood that the model weight of the processing layer is deployed on the first processor with lower computing power (such as CPU), and the second processor with higher computing power (such as NPU) is called to obtain the model weight of the processing layer from the memory of the first processor, and then execute the model calculation of the processing layer.

[0181] In an embodiment of the present application, after the current processor executes the model calculation of the processing layer, a calculation result will be obtained, and the calculation result needs to be stored for use in subsequent model calculations. Specifically, in the memory buffer of the first processor used to deploy the model weights of the processing layer, a portion of space for storing the calculation results can be reserved, and this portion of space can be used to store the calculation results. When the model calculation of the processing layer is executed by the first processor, the first processor can directly store the calculation result in the memory buffer. When the model calculation of the processing layer is executed by the second processor, the second processor can first send the calculation result to the first processor, and then the first processor stores the calculation result in the memory buffer.

[0182] In an optional implementation manner of the present application, the data required for the model calculation of the processing layer includes the model weight corresponding to the processing layer, and determining the transmission duration includes:

[0183] Obtaining the transmission bandwidth of the second processor and the data volume of the model weight corresponding to the processing layer;

[0184] Based on the transmission bandwidth and the data volume, determine the transmission time required for the second processor to obtain the data required to execute the model calculation of the processing layer.

[0185] In the embodiment of the present application, the data required to perform the model calculation of the processing layer includes the model weight corresponding to the processing layer and the calculation result of the previous processing layer. The data volume of the model weight corresponding to the processing layer is relatively large, which is the main data to be transmitted, so the transmission of the model weight corresponding to the processing layer can be used as the transmission duration.

[0186] The model information may include the data volume of the model weights corresponding to each processing layer, and the processor parameters may include the transmission bandwidth of the processor. Based on the transmission bandwidth and the data volume, it is possible to determine the transmission time for the second processor to obtain the data required to perform the model calculation of the processing layer.

[0187] In an optional implementation manner of the present application, determining a first duration corresponding to the first processor includes:

[0188] Obtaining the layer calculation amount of the model calculation corresponding to the processing layer and the computing power of the first processor;

[0189] Based on the layer calculation amount and the computing power of the first processor, a first duration for the first processor to perform the model calculation corresponding to the processing layer is determined.

[0190] In the embodiment of the present application, the layer calculation amount of each processing layer of the model to be deployed can be obtained and used as model information. As an example, the layer calculation amount can be estimated based on the data amount of each processing layer corresponding to the model weight.

[0191] In the embodiment of the present application, the computing power of the first processor can be obtained, and based on the computing power of the first processor and the layer calculation amount, the first time required for the first processor to process the model calculation corresponding to the processing layer can be determined.

[0192] Similarly, the computing power of the second processor can be obtained. Based on the computing power of the second processor and the layer calculation amount, the computing time required for the second processor to process the model calculation corresponding to the processing layer can be determined. By adding it to the transmission time, the second time can be obtained.

[0193] In an optional implementation manner of the present application, after deploying the model weights of each processing layer to a specified processor in the target device, the method further includes:

[0194] For any processing layer, the first processor is called to execute the model operation corresponding to the processing layer in the generation phase, wherein the first processor is a processor for deploying the model weights corresponding to the processing layer.

[0195] In the embodiment of the present application, in the generation phase of the model to be deployed, a token is input and processed each time. The generation phase has relatively low requirements for processor computing power. Therefore, in the generation phase, the first processor that deploys the model weight corresponding to a certain processing layer can be directly used to perform the model calculation of the processing layer. In other words, in the generation phase, there is no need to transfer the model weight to a processor with higher computing power to perform the model calculation of the processing layer, and higher model calculation efficiency can be achieved.

[0196] In the embodiment of the present application, since the generation stage inputs and processes one token each time, if the processor that executes the model calculation of each processing layer is still specified in the understanding stage, the model weight will be transmitted multiple times, resulting in a large amount of time spent on the transmission of the model weight, affecting the model processing efficiency. Considering that the demand for processor computing power in the generation stage is relatively low, the processor used for model deployment can generally meet the demand for processor computing power in the generation stage. Therefore, the first processor that deploys the model weight corresponding to a certain processing layer can be obtained, and the first processor can be called to perform the model calculation of the processing layer.

[0197] In an optional implementation manner of the present application, before obtaining the deployment solution for the model to be deployed, the method further includes:

[0198] Run the preset test program to collect processor parameters of each processor of the target device;

[0199] The processor parameters are sent to the server, so that the server generates a deployment plan based on the processor parameters and the model information of the model to be deployed.

[0200] In the embodiment of the present application, a preset test program can be run on the target device to obtain processor parameters of each processor. The preset test program may include but is not limited to a benchmark program, and the collected processor parameters include processor computing power, transmission bandwidth, and storage space.

[0201] As an example, the operating system of the target device may provide relevant APIs, and the processor parameters may be sent to the server through these interfaces.

[0202] Based on Figure 2 The same principle as shown in the method, Figure 7 A schematic diagram of the structure of a model deployment solution generation device provided in an embodiment of the present application is shown. Figure 7 As shown, the model deployment solution generating device 70 may include:

[0203] A model information acquisition module 710 is used to acquire model information of a model to be deployed;

[0204] A processor parameter acquisition module 720, used to acquire processor parameters of each processor in a target device, where the target device is used to deploy the model to be deployed;

[0205] The deployment plan generation module 730 is used to generate a deployment plan based on model information and processor parameters. The deployment plan includes first specified information. The model to be deployed includes multiple processing layers. The first specified information is used to specify the processor where the model weight of each processing layer is deployed.

[0206] The device provided in the embodiment of the present application obtains the model information of the model to be deployed, obtains the processor parameters of each processor in the target device for deploying the model to be deployed, and generates a deployment plan based on the model information and the processor parameters, and the deployment plan includes first specified information for specifying the processors deployed by the model weights of each processing layer. Based on the deployment plan provided in the embodiment of the present application, it is possible to achieve reasonable allocation of processor resources in the target device, thereby achieving reasonable deployment of the model in the target device.

[0207] Optionally, the deployment plan also includes second specified information, and the second specified information is used to specify a processor that executes model calculations corresponding to each processing layer.

[0208] Optionally, the second specified information includes second specified information in the understanding stage, and the second specified information in the understanding stage is used to specify a processor that performs model calculations corresponding to each processing layer in the understanding stage, and the deployment solution generation module is specifically used to:

[0209] For any processing layer, determine a first duration corresponding to the first processor and a second duration corresponding to the second processor, wherein the model weight corresponding to the processing layer is deployed on the first processor, the computing power of the second processor is higher than that of the first processor, the first duration is the duration for the first processor to perform the model calculation corresponding to the processing layer, and the second duration is the sum of the transmission duration and the calculation duration, the transmission duration is the duration for the data required to perform the model calculation of the processing layer to be transmitted to the second processor, and the calculation duration is the duration for the second processor to perform the model calculation corresponding to the processing layer;

[0210] Based on the first duration and the second duration, second specified information of the understanding stage is determined.

[0211] Optionally, the second designated information includes second designated information of the generation stage, and the second designated information of the generation stage is used to specify that the processor that performs the model calculation corresponding to each processing layer in the generation stage is the first processor, and the first processor is the processor deployed for the model weight corresponding to each processing layer.

[0212] Optionally, the processor parameters include memory information of the processor, the model information includes the data volume of the model weight corresponding to each processing layer, and the deployment solution generation module is specifically used for:

[0213] The first specified information is determined based on the memory information and the data volume of the model weights corresponding to each processing layer.

[0214] Optionally, when the deployment solution generation module determines the first specified information based on the memory information and the data volume of the model weights corresponding to each processing layer, it is specifically used to:

[0215] Get the historical memory usage of each processor;

[0216] Determine available memory based on historical memory usage and memory information;

[0217] The first specified information is determined based on the available memory and the data volume of the model weights corresponding to each processing layer.

[0218] Optionally, the processor parameters are collected based on a preset test program running on the target device.

[0219] Based on Figure 6 The same principle as shown in the method, Figure 8 A schematic diagram of the structure of a model processing device provided in an embodiment of the present application is shown. Figure 8 As shown, the model processing device 80 may include:

[0220] A deployment scheme acquisition module 810 is used to acquire a deployment scheme for a model to be deployed, wherein the deployment scheme includes first specified information, the model to be deployed includes multiple processing layers, and the target device for deploying the model to be deployed includes multiple processors. The first specified information is used to specify the processors to which the model weights of each processing layer are deployed. The model deployment scheme is generated by the server based on the model information of the model to be deployed and the processor parameters of the processors.

[0221] The model deployment module 820 is used to deploy the model weights of each processing layer to a specified processor in the target device based on the first specified information.

[0222] The device provided by the embodiment of the present application obtains a deployment scheme for the model to be deployed, and the deployment scheme includes first specified information and second specified information. The model to be deployed includes multiple processing layers, and the target device for deploying the model to be deployed includes multiple processors. The first specified information is used to specify the processor where the model weights of each processing layer are deployed, and the second specified information is used to specify the processor that performs the model calculation corresponding to each processing layer. The model deployment scheme is generated by the server based on the model information of the model to be deployed and the processor parameters of the processor. Based on the first specified information, the model weights of each processing layer are deployed to the specified processor in the target device, and the model calculation of the model to be deployed is performed based on the second specified information. Model deployment based on the deployment scheme provided by the embodiment of the present application can achieve reasonable allocation of processor resources in the target device, thereby achieving reasonable deployment of the model on the target device.

[0223] Optionally, the above device further includes a model calculation module, and the model calculation module is used to:

[0224] After deploying the model weights of each processing layer to the specified processor in the target device, for any processing layer, determining the current processor that performs the model calculation of the processing layer in the understanding phase;

[0225] In response to the current processor being the first processor, reading the model weight corresponding to the processing layer from the memory buffer of the first processor, and performing the model calculation of the processing layer in the understanding phase based on the model weight corresponding to the processing layer, the first processor being a processor for deploying the model weight corresponding to the processing layer;

[0226] In response to the current processor not being the first processor, the model weights corresponding to the processing layer are transferred from the memory buffer of the first processor to the memory buffer of the current processor, the model weights corresponding to the processing layer are read from the memory buffer of the current processor, and the model calculations of the processing layer in the understanding phase are performed based on the model weights corresponding to the processing layer.

[0227] Optionally, when the model calculation module transfers the model weight corresponding to the processing layer from the memory buffer of the first processor to the memory buffer of the current processor, it is specifically used to:

[0228] Before calling the current processor to execute the model calculation of the processing layer, the current processor is controlled to start a transmission task, where the transmission task is used to transmit the model weights of the processing layer to the memory buffer of the current processor.

[0229] Optionally, the above device further includes:

[0230] The model weight removal module is used to remove the model weight of the processing layer from the memory buffer of the current processor after the model calculation of the processing layer is completed.

[0231] Optionally, when determining the current processor for executing the model calculation of the processing layer in the understanding phase, the model calculation module is specifically configured to:

[0232] Determine a first duration corresponding to the first processor and a second duration corresponding to the second processor, wherein the computing power of the second processor is higher than the computing power of the first processor, the first duration is the duration for the first processor to perform the model calculation corresponding to the processing layer, and the second duration is the sum of the transmission duration and the calculation duration, the transmission duration is the duration for data required to perform the model calculation of the processing layer to be transmitted to the second processor, and the calculation duration is the duration for the second processor to perform the model calculation corresponding to the processing layer;

[0233] Based on the first duration and the second duration, a current processor that performs model calculation of the processing layer in the understanding phase is determined.

[0234] Optionally, when the model calculation module determines the current processor for executing the model calculation of the processing layer in the understanding stage based on the first duration and the second duration, it is specifically configured to:

[0235] In response to the first duration being not greater than the second duration, determining the first processor as a current processor for executing the model calculation of the processing layer in the understanding phase;

[0236] In response to the first duration being greater than the second duration, the second processor is determined as the current processor for executing the model calculation of the processing layer in the understanding phase.

[0237] Optionally, the data required for the model calculation of the processing layer includes a model weight corresponding to the processing layer, and the model calculation module is specifically used to:

[0238] Obtaining the transmission bandwidth of the second processor and the data volume of the model weight corresponding to the processing layer;

[0239] Based on the transmission bandwidth and the data volume, determine the transmission time required for the second processor to obtain the data required to execute the model calculation of the processing layer.

[0240] Optionally, when determining the first duration corresponding to the first processor, the model calculation module is specifically configured to:

[0241] Obtaining the layer calculation amount of the model calculation corresponding to the processing layer and the computing power of the first processor;

[0242] Based on the layer calculation amount and the computing power of the first processor, a first duration for the first processor to perform the model calculation corresponding to the processing layer is determined.

[0243] Optionally, the model calculation module is further used to:

[0244] After the model weights of each processing layer are deployed to the specified processor in the target device, for any processing layer, the first processor is called to execute the model operation corresponding to the processing layer in the generation phase, wherein the first processor is a processor for deploying the model weights corresponding to the processing layer.

[0245] Optionally, the above device further includes:

[0246] A processor parameter collection module is used to run a preset test program to collect processor parameters of each processor of the target device before obtaining a deployment plan for the model to be deployed;

[0247] The processor parameter reporting module is used to send the processor parameters to the server so that the server generates a deployment plan based on the processor parameters and the model information of the model to be deployed.

[0248] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0249] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0250] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the method embodiments described above.

[0251] Compared with the prior art, the computer-readable storage medium provided in the embodiment of the present application obtains the model information of the model to be deployed, obtains the processor parameters of each processor in the target device for deploying the model to be deployed, and generates a deployment plan based on the model information and the processor parameters, the deployment plan including first designated information for specifying the processors deployed by the model weights of each processing layer and second designated information for specifying the processors that perform the model calculations corresponding to each processing layer. Based on the deployment plan provided in the embodiment of the present application, it is possible to achieve reasonable allocation of processor resources in the target device, thereby achieving reasonable deployment of the model in the target device.

[0252] The present application also provides an electronic device, including:

[0253] one or more processors; and

[0254] A memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, executes the steps of any one of the method embodiments described above.

[0255] Compared with the prior art, the electronic device provided by the embodiment of the present application obtains the model information of the model to be deployed, obtains the processor parameters of each processor in the target device for deploying the model to be deployed, and generates a deployment plan based on the model information and the processor parameters, the deployment plan including first designated information for specifying the processors deployed by the model weights of each processing layer and second designated information for specifying the processors that perform the model calculations corresponding to each processing layer. Based on the deployment plan provided by the embodiment of the present application, it is possible to achieve reasonable allocation of processor resources in the target device, thereby achieving reasonable deployment of the model in the target device.

[0256] As an example, Fig. 9 A schematic diagram of the structure of an electronic device applicable to an embodiment of the present application is shown. Fig. 9 As shown, the electronic device 2000 includes: a processor 2001 and a memory 2003. The processor 2001 and the memory 2003 are connected, such as through a bus 2002. Optionally, the electronic device 2000 may also include a transceiver 2004. It should be noted that in actual applications, the transceiver 2004 is not limited to one, and the structure of the electronic device 2000 does not constitute a limitation on the embodiments of the present application.

[0257] The processor 2001 is applied in the embodiment of the present application to implement the method shown in the above method embodiment. The transceiver 2004 may include a receiver and a transmitter. The transceiver 2004 is applied in the embodiment of the present application to implement the function of the electronic device of the embodiment of the present application communicating with other devices when executed.

[0258] Processor 2001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 2001 may also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0259] The bus 2002 may include a path for transmitting information between the above components. The bus 2002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 2002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0260] The memory 2003 can be a ROM (Read On Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrical Erasable Programmable Read On Memory), a CD-ROM (Compact Disk Read On Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0261] Optionally, the memory 2003 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 2001. The processor 2001 is used to execute the application code stored in the memory 2003 to implement the steps of any one of the methods in the aforementioned method embodiments.

[0262] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the method embodiments described above when executed by a processor.

[0263] Compared with the prior art, the computer program product provided by the embodiment of the present application obtains the model information of the model to be deployed, obtains the processor parameters of each processor in the target device for deploying the model to be deployed, and generates a deployment plan based on the model information and the processor parameters, the deployment plan including first designated information for specifying the processors deployed by the model weights of each processing layer and second designated information for specifying the processors that perform the model calculations corresponding to each processing layer. Based on the deployment plan provided by the embodiment of the present application, it is possible to achieve reasonable allocation of processor resources in the target device, thereby achieving reasonable deployment of the model in the target device.

[0264] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as designated by the arrows, these steps are not necessarily executed in sequence in the order designated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a portion of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0265] The above descriptions are only some embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating a model deployment solution, characterized in that: include: Get the model information of the model to be deployed; Obtaining processor parameters of each processor in a target device, where the target device is used to deploy the model to be deployed; Based on the model information and the processor parameters, a deployment plan is generated, the deployment plan includes first specified information, the model to be deployed includes multiple processing layers, and the first specified information is used to specify the processor deployed by the model weight of each processing layer.

2. The method according to claim 1, characterized in that The deployment solution also includes second specified information, and the second specified information is used to specify a processor that executes model calculations corresponding to each processing layer.

3. The method according to claim 2, characterized in that The second specified information includes second specified information in the understanding stage, and the second specified information in the understanding stage is used to specify a processor that performs model calculations corresponding to each processing layer in the understanding stage. The generating of the deployment plan based on the model information and the processor parameters includes: For any of the processing layers, determine a first duration corresponding to the first processor and a second duration corresponding to the second processor, wherein the model weight corresponding to the processing layer is deployed on the first processor, the computing power of the second processor is higher than the computing power of the first processor, the first duration is the duration for the first processor to perform the model calculation corresponding to the processing layer, and the second duration is the sum of the transmission duration and the calculation duration, the transmission duration is the duration for the data required to perform the model calculation of the processing layer to be transmitted to the second processor, and the calculation duration is the duration for the second processor to perform the model calculation corresponding to the processing layer; Based on the first duration and the second duration, second designated information of the understanding stage is determined.

4. The method according to claim 2, characterized in that: The second designated information includes the second designated information of the generation stage, and the second designated information of the generation stage is used to specify that the processor that performs the model calculation corresponding to each processing layer in the generation stage is the first processor, and the first processor is the processor deployed for the model weight corresponding to each processing layer.

5. The method according to any one of claims 1 to 4, characterized in that The processor parameters include memory information of the processor, the model information includes data volume of model weights corresponding to each processing layer, and the generating of a deployment plan based on the model information and the processor parameters includes: The first specified information is determined based on the memory information and the data volume of the model weights corresponding to each processing layer.

6. The method according to claim 5, characterized in that The determining the first specified information based on the memory information and the data amount of the model weights corresponding to each processing layer includes: Obtaining historical memory usage of each of the processors; Determining available memory based on the historical memory usage and the memory information; The first specified information is determined based on the available memory and the data volume of the model weights corresponding to each processing layer.

7. The method according to any one of claims 1 to 4, characterized in that The processor parameters are collected based on a preset test program running on the target device.

8. A model processing method, characterized in that: include: Obtaining a deployment scheme for a model to be deployed, the deployment scheme comprising first specified information, the model to be deployed comprising multiple processing layers, the target device for deploying the model to be deployed comprising multiple processors, the first specified information being used to specify the processors to which the model weights of each processing layer are deployed, and the model deployment scheme is generated by the server based on the model information of the model to be deployed and the processor parameters of the processors; Based on the first specified information, the model weights of each processing layer are deployed to a specified processor in the target device.

9. The method according to claim 8, characterized in that After deploying the model weights of each processing layer to a specified processor in the target device, the method further includes: For any of the processing layers, determining a current processor that performs model calculations of the processing layer in the understanding phase; In response to the current processor being the first processor, reading the model weight corresponding to the processing layer from the memory buffer of the first processor, and performing the model calculation of the processing layer in the understanding phase based on the model weight corresponding to the processing layer, the first processor being a processor for deploying the model weight corresponding to the processing layer; In response to the current processor not being the first processor, the model weights corresponding to the processing layer are transferred from the memory buffer of the first processor to the memory buffer of the current processor, the model weights corresponding to the processing layer are read from the memory buffer of the current processor, and the model calculations of the processing layer in the understanding phase are performed based on the model weights corresponding to the processing layer.

10. The method according to claim 9, characterized in that The transferring the model weight corresponding to the processing layer from the memory buffer of the first processor to the memory buffer of the current processor includes: Before calling the current processor to perform model calculation of the processing layer, the current processor is controlled to start a transmission task, where the transmission task is used to transmit the model weights of the processing layer to the memory buffer of the current processor.

11. The method according to claim 10, characterized in that The method further comprises: After the model calculation of the processing layer is completed, the model weights of the processing layer are moved out of the memory buffer of the current processor.

12. The method according to any one of claims 9 to 11, characterized in that: The determining of the current processor for executing the model calculation of the processing layer in the understanding phase includes: Determine a first duration corresponding to the first processor and a second duration corresponding to the second processor, wherein the computing power of the second processor is higher than the computing power of the first processor, the first duration is the duration for the first processor to perform model calculations corresponding to the processing layer, and the second duration is the sum of a transmission duration and a calculation duration, the transmission duration is the duration for data required to perform the model calculations of the processing layer to be transmitted to the second processor, and the calculation duration is the duration for the second processor to perform the model calculations corresponding to the processing layer; Based on the first duration and the second duration, a current processor that performs model calculation of the processing layer in the understanding phase is determined.

13. The method according to claim 12, characterized in that The determining, based on the first duration and the second duration, a current processor for executing model calculation of the processing layer in the understanding phase includes: In response to the first duration being not greater than the second duration, determining the first processor as a current processor for executing model calculation of the processing layer in the understanding phase; In response to the first duration being greater than the second duration, the second processor is determined as a current processor for executing model calculation of the processing layer in the understanding phase.

14. The method according to claim 12, characterized in that The data required for the model calculation of the processing layer includes the model weight corresponding to the processing layer, and determining the transmission duration includes: Obtaining the transmission bandwidth of the second processor and the data volume of the model weight corresponding to the processing layer; Based on the transmission bandwidth and the data volume, determine the transmission time required for the second processor to obtain the data required to execute the model calculation of the processing layer.

15. The method according to claim 12, characterized in that The determining a first duration corresponding to the first processor includes: Obtaining a layer calculation amount of a model calculation corresponding to the processing layer and a computing power of the first processor; Based on the layer calculation amount and the computing power of the first processor, a first duration for the first processor to execute the model calculation corresponding to the processing layer is determined.

16. The method according to any one of claims 8 to 11, characterized in that: After deploying the model weights of each processing layer to a specified processor in the target device, the method further includes: For any of the processing layers, the first processor is called to execute the model operation corresponding to the processing layer in the generation phase, wherein the first processor is a processor for deploying the model weights corresponding to the processing layer.

17. The method according to any one of claims 8 to 11, characterized in that: Before obtaining the deployment plan for the model to be deployed, the method further includes: Running a preset test program to collect processor parameters of each processor of the target device; The processor parameters are sent to a server, so that the server generates the deployment solution based on the processor parameters and the model information of the model to be deployed.

18. A model deployment scheme generating device, characterized in that: include: A model information acquisition module is used to obtain model information of the model to be deployed; A processor parameter acquisition module, used to acquire processor parameters of each processor in a target device, where the target device is used to deploy the model to be deployed; A deployment plan generation module is used to generate a deployment plan based on the model information and the processor parameters, wherein the deployment plan includes first specified information, the model to be deployed includes multiple processing layers, and the first specified information is used to specify the processor to which the model weight of each processing layer is deployed.

19. A model processing device, characterized in that: include: A deployment scheme acquisition module, used to acquire a deployment scheme for a model to be deployed, wherein the deployment scheme includes first specified information, the model to be deployed includes multiple processing layers, and the target device for deploying the model to be deployed includes multiple processors. The first specified information is used to specify the processor to which the model weight of each processing layer is deployed. The model deployment scheme is generated by the server based on the model information of the model to be deployed and the processor parameters of the processor; A model deployment module is used to deploy the model weights of each processing layer to a specified processor in the target device based on the first specified information.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

21. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 17.

22. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images