Service deployment method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of computers, and discloses a service deployment method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining a first to-be-deployed model of a target service, the first to-be-deployed model comprising a first pre-filling unit and a first decoding unit; quantizing the first to-be-deployed model based on a plurality of preset quantization modes to obtain processing performance of a second pre-filling unit and a second decoding unit in a corresponding second to-be-deployed model; respectively obtaining target quantization modes of the first pre-filling unit and the first decoding unit based on a comparison result of the processing performance; based on the first target quantization mode and the second target quantization mode, respectively quantizing the first pre-filling unit and the first decoding unit to obtain a target deployment model; and deploying the target deployment model into the target equipment to deploy the target service. According to the method and the device, the problem that model quantification affects the performance of the deployed target service can be solved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a service deployment method, device, electronic device, storage medium, and program product. Background Art

[0002] In related technologies, when deploying the language model of the target service on a device, the language model is usually quantized to improve the efficiency of deploying the language model on the device. The reasoning process of the language model mainly includes the pre-filling stage and the decoding stage. Due to the different computational characteristics of the pre-filling stage and the decoding stage, the introduced quantization method may reduce the reasoning speed of the language model, thereby affecting the performance of the target service. Summary of the invention

[0003] In view of this, the present disclosure provides a service deployment method, apparatus, device, storage medium and program product to solve the problem that model quantization affects the performance of the deployed target service.

[0004] In a first aspect, the present disclosure provides a service deployment method, the method comprising:

[0005] Acquire a first to-be-deployed model of a target service, wherein the first to-be-deployed model includes a first pre-filling unit and a first decoding unit;

[0006] quantizing the first to-be-deployed model based on a plurality of preset quantization methods respectively, and obtaining processing performance of a second pre-filling unit and a second decoding unit in a corresponding second to-be-deployed model;

[0007] Based on the comparison result of the processing performance, respectively obtaining a first target quantization mode of the first pre-filling unit and a second target quantization mode of the first decoding unit;

[0008] Based on the first target quantization method and the second target quantization method, quantizing the first pre-filling unit and the first decoding unit respectively to obtain a target deployment model;

[0009] The target deployment model is deployed to a target device to deploy the target service.

[0010] In a second aspect, the present disclosure provides a service deployment device, the device comprising:

[0011] A model acquisition module, used to acquire a first to-be-deployed model of a target service, wherein the first to-be-deployed model includes a first pre-filling unit and a first decoding unit;

[0012] A model processing module, used to quantize the first model to be deployed based on a plurality of preset quantization methods, respectively, to obtain processing performance of a second pre-filling unit and a second decoding unit in a corresponding second model to be deployed;

[0013] a mode determination module, configured to obtain a first target quantization mode of the first pre-filling unit and a second target quantization mode of the first decoding unit based on the comparison result of the processing performance;

[0014] A unit quantization module, configured to quantize the first pre-filling unit and the first decoding unit respectively based on the first target quantization mode and the second target quantization mode to obtain a target deployment model;

[0015] The model deployment module is used to deploy the target deployment model to the target device to deploy the target service.

[0016] In a third aspect, the present disclosure provides an electronic device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the service deployment method of the first aspect by executing the computer instructions.

[0017] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the service deployment method of the first aspect.

[0018] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, where the computer instructions are used to enable a computer to execute the service deployment method of the first aspect.

[0019] The service deployment method provided by the embodiment of the present disclosure quantizes the first model to be deployed of the target service based on multiple preset quantization methods, and obtains the processing performance of the second pre-filling unit and the second decoding unit in the corresponding second model to be deployed. Then, based on the comparison result of the processing performance, the first target quantization method of the first pre-filling unit and the second target quantization method of the first decoding unit in the first model to be deployed are obtained respectively. Therefore, according to the calculation characteristics of the pre-filling stage and the decoding stage, the quantization methods suitable for the pre-filling stage and the decoding stage can be selected respectively. Then, based on the first target quantization method and the second target quantization method, the first pre-filling unit and the first decoding unit are quantized respectively to obtain the target deployment model. Therefore, the performance impact of model quantization on the target deployment model can be reduced to ensure the reasoning performance of the target deployment model at different reasoning stages. Then, the target deployment model is deployed to the target device to deploy the target service. Therefore, it is possible to improve the deployment efficiency of the target service through quantization while ensuring the performance of the deployed target service. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 is a flowchart of a first service deployment method according to an embodiment of the present disclosure;

[0022] Figure 2 is a flow chart of a second service deployment method according to an embodiment of the present disclosure;

[0023] Figure 3 is a flow chart of a third service deployment method according to an embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram of a deployment architecture of a target service according to an embodiment of the present disclosure;

[0025] Figure 5 is a structural block diagram of a service deployment device according to an embodiment of the present disclosure;

[0026] Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0028] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0029] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0030] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0032] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0033] The reasoning process of the language model mainly includes the prefill and decoding stages. In the prefill stage, the language model generates the corresponding word element identification sequence (input ids) according to the input prompt information, and generates the first word element through the language model decoder layer (LLM decoder layers). In this process, a key-value cache (KV Cache) is generated for all word element identification sequences. Therefore, in the subsequent decoding stage, there is no need to repeatedly generate the key-value cache, avoiding a large number of calculation processes. Since the key-value cache and self-attention mechanism (Self-Attention) need to be calculated for all input word element identification sequences in the prefill stage, the amount of calculation is large and it is in a calculation-intensive stage. Even if the batch size is increased, the throughput of the overall prefill stage will not increase significantly.

[0034] In the decoding stage, the next word is generated gradually based on the processing of the last generated word and the decoder layer of the language model. In this process, since the key-value cache has been obtained in the pre-filling stage, and the decoding is performed in an autoregressive manner, the length of each input sequence is one, and the amount of calculation at this time is smaller than that in the pre-filling stage. Therefore, the calculation speed of each step in the decoding stage is faster. The memory mapping input / output (Memory I / O) of the language model accounts for the vast majority of the time, so increasing the batch size can greatly improve the throughput of the model.

[0035] Taking into account the different computing characteristics of the pre-filling stage and the decoding stage, in the relevant technology, a separate deployment method is usually adopted for the deployment of the target service, and the parts of the language model of the target service corresponding to the pre-filling stage and the decoding stage are deployed to different devices (such as graphics processors) to run, so as to make better use of the computing performance of the device and improve throughput.

[0036] When deploying the parts of the language model corresponding to the pre-filling stage and the decoding stage on the device, the language model is usually quantized and applied to the pre-filling stage and the decoding stage to improve the efficiency of deploying the language model on the device. However, due to the different computing characteristics of the pre-filling stage and the decoding stage, the quantization algorithm introduced in this model quantization method may reduce the processing speed of the language model, thereby affecting the performance of the deployed target service.

[0037] In view of this, according to an embodiment of the present disclosure, a service deployment method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] In this embodiment, a service deployment method is provided, which can be used in a service deployment platform. The service deployment platform is used to deploy a model for implementing a target service. Figure 1 is a flow chart of a first service deployment method according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:

[0039] Step S101: Acquire a first to-be-deployed model of a target service, where the first to-be-deployed model includes a first pre-filling unit and a first decoding unit.

[0040] Specifically, the first model to be deployed is used to implement the target service, and the first model to be deployed is a language model, such as a large language model (LLM). In addition, the first model to be deployed can also use other models that perform model reasoning through the pre-filling stage and the decoding stage, which is not limited here.

[0041] Specifically, the first pre-filling unit corresponds to a part of a pre-filling phase in the first model to be deployed, and the first encoding unit corresponds to a part of a decoding phase in the first model to be deployed.

[0042] Step S102: quantize the first model to be deployed based on multiple preset quantization methods to obtain the processing performance of the second pre-filling unit and the second decoding unit in the corresponding second model to be deployed.

[0043] Specifically, the plurality of preset quantization methods include quantization algorithms that meet the quantization accuracy requirements.

[0044] Specifically, multiple preset quantization methods include but are not limited to W4, W8, W8A8, fp16 and other quantization algorithms.

[0045] Specifically, the processing performance of the second pre-filling unit and the second decoding unit may be characterized by one of processing time and processing efficiency.

[0046] Step S103 , based on the comparison result of the processing performance, respectively obtain a first target quantization mode of the first pre-filling unit and a second target quantization mode of the first decoding unit.

[0047] Specifically, the preset quantization mode corresponding to the second pre-filling unit with the best processing performance is determined as the first target quantization mode.

[0048] Specifically, the preset quantization mode corresponding to the second decoding unit with the best processing performance is determined as the second target quantization mode.

[0049] Step S104, based on the first target quantization method and the second target quantization method, the first pre-filling unit and the first decoding unit are quantized respectively to obtain a target deployment model.

[0050] Specifically, the first pre-filling unit is quantized based on the first target quantization method to obtain the third pre-filling unit. The first decoding unit is quantized based on the second target quantization method to obtain the third decoding unit. The target deployment model is formed based on the third pre-filling unit and the third decoding unit.

[0051] Step S105, deploying the target deployment model to the target device to deploy the target service.

[0052] Specifically, the entire target deployment model may be deployed to the same target device, or different target devices may be used to respectively deploy the third pre-filling unit and the third decoding unit in the target deployment model.

[0053] The service deployment method provided in this embodiment quantizes the first model to be deployed of the target service based on multiple preset quantization methods, and obtains the processing performance of the second pre-filling unit and the second decoding unit in the corresponding second model to be deployed. Then, based on the comparison result of the processing performance, the first target quantization method of the first pre-filling unit and the second target quantization method of the first decoding unit in the first model to be deployed are obtained respectively. Therefore, according to the calculation characteristics of the pre-filling stage and the decoding stage, the quantization methods suitable for the pre-filling stage and the decoding stage can be selected respectively. Then, based on the first target quantization method and the second target quantization method, the first pre-filling unit and the first decoding unit are quantized respectively to obtain the target deployment model. Therefore, the performance impact of model quantization on the target deployment model can be reduced to ensure the reasoning performance of the target deployment model at different reasoning stages. Then, the target deployment model is deployed to the target device to deploy the target service. Therefore, it is possible to improve the deployment efficiency of the target service through quantization while ensuring the performance of the deployed target service.

[0054] In this embodiment, a second service deployment method is provided, which can be used in a service deployment platform. The service deployment platform is used to deploy a model for implementing a target service. Figure 2 is a flow chart of a second service deployment method according to an embodiment of the present disclosure, such as Figure 2 As shown, the process includes the following steps:

[0055] Step S201, obtaining a first to-be-deployed model of a target service, the first to-be-deployed model comprising a first pre-filling unit and a first decoding unit. Please refer to the above step S101, which will not be described in detail here.

[0056] Step S202: quantize the first model to be deployed based on multiple preset quantization methods to obtain the processing performance of the second pre-filling unit and the second decoding unit in the corresponding second model to be deployed.

[0057] Specifically, the above step S202 includes:

[0058] Step S2021: quantize the first model to be deployed based on multiple preset quantization methods to obtain a corresponding second model to be deployed.

[0059] Exemplarily, if the multiple preset quantization methods include preset quantization method 1 and preset quantization method 2, the preset quantization method 1 is used to quantize the first model to be deployed, and the second model to be deployed corresponding to the preset quantization method 1 is obtained. The preset quantization method 2 is used to quantize the first model to be deployed, and the second model to be deployed corresponding to the preset quantization method 2 is obtained.

[0060] Step S2022: Determine the performance difference between each second model to be deployed and the first model to be deployed.

[0061] Specifically, a performance evaluation is performed on each of the second models to be deployed and the first models to be deployed, and a performance difference between each of the second models to be deployed and the first models to be deployed is determined according to the performance evaluation result.

[0062] Step S2023: If the performance difference is less than the preset difference threshold, the processing performance of the second pre-filling unit and the second decoding unit in the second to-be-deployed model is determined.

[0063] Optionally, the preset difference threshold is 0.5%. That is, the performance of the second model to be deployed is 0.5% different from the performance of the first model to be deployed. In actual operation, the preset difference threshold can be flexibly adjusted according to actual conditions, and the value of the preset difference threshold is not limited here.

[0064] It can be understood that if the performance difference is greater than or equal to the preset difference threshold, it means that after the model is quantized, the second model to be deployed no longer meets the expected model performance requirements, and the preset quantization method corresponding to the second model to be deployed has a poor quantization effect.

[0065] Step S203: Based on the comparison result of the processing performance, a first target quantization mode of the first pre-filling unit and a second target quantization mode of the first decoding unit are obtained respectively. Please refer to the above step S103, which will not be described in detail here.

[0066] Step S204: quantize the first pre-filling unit and the first decoding unit based on the first target quantization method and the second target quantization method to obtain a target deployment model. Please refer to the above step S104, which will not be described in detail here.

[0067] Step S205: deploy the target deployment model to the target device to deploy the target service. Please refer to the above step S105, which will not be described in detail here.

[0068] The service deployment method provided in this embodiment determines the processing performance of the second pre-filling unit and the second decoding unit in the second model to be deployed only when the performance difference between the second model to be deployed and the first model to be deployed is less than a preset difference threshold. Therefore, the performance of the quantified model can be guaranteed by screening the performance difference.

[0069] In some optional implementations, determining the performance difference between each second model to be deployed and the first model to be deployed in step S2022 includes:

[0070] Step a1, obtaining first test data.

[0071] It should be noted that the first test data is test data under the target service, and the first test data includes at least one first test sample for performance testing.

[0072] Step a2: Based on the first test data, a performance test is performed on the first model to be deployed and the second model to be deployed respectively to obtain a first performance indicator of the first model to be deployed and a second performance indicator of each second model to be deployed.

[0073] Optionally, the first performance indicator is the calculation accuracy of the first model to be deployed, and the second performance indicator is the calculation accuracy of the second model to be deployed.

[0074] Step a3: based on the difference between the second performance indicator and the first performance indicator, obtain the performance difference of the corresponding second model to be deployed.

[0075] The service deployment method provided in this embodiment performs performance tests on the first model to be deployed and the second model to be deployed based on the first test data, respectively, to obtain performance indicators corresponding to the first model to be deployed and the second model to be deployed, so as to determine the performance difference between the second model to be deployed and the first model to be deployed. Therefore, the performance difference can be used to effectively screen out the preset quantization method with poor quantization performance, thereby ensuring the model performance of the target deployment model obtained by the final quantization.

[0076] In some optional implementations, determining the processing performance of the second pre-filling unit and the second decoding unit in the second to-be-deployed model in the above step S2023 includes:

[0077] Step b1, obtaining second test data.

[0078] It should be noted that the second test data is test data under the target service, and the second test data includes at least one second test sample for performance testing.

[0079] Step b2: input the second test data into the second model to be deployed for processing, obtain the total processing time of the second model to be deployed and the first processing time of the second pre-filling unit, and the first processing time is used to characterize the processing performance of the second pre-filling unit.

[0080] It should be noted that, when the performance test is usually performed on the model to be deployed, the total processing time from input to output can be obtained, but the processing time of the pre-filling stage and the decoding stage is unknown. Therefore, in the service code of the second model to be deployed, a code for detecting the processing time of the second model to be deployed in the pre-filling stage can be added to obtain the first processing time.

[0081] Step b3: determining a second processing duration of the second decoding unit based on the difference between the total processing duration and the first processing duration, where the second processing duration is used to characterize the processing performance of the second decoding unit.

[0082] It should be noted that in addition to determining the second processing time by the difference between the total processing time and the first processing time, a code for detecting the processing time of the second model to be deployed in the decoding stage can also be added to the service code of the second model to be deployed to obtain the second processing time.

[0083] It is worth noting that, assuming that the second model to be deployed needs to infer and output 64 tokens (also known as tokens) when processing the second test data. Among them, the first token is generated in the pre-filling stage, and then the remaining 63 tokens are generated in the decoding stage. If the processing time of the second model to be deployed in the pre-filling stage and the decoding stage is detected at the same time, the processing time of 63 tokens needs to be detected separately in the decoding stage to obtain the processing time of the decoding stage. Therefore, it is necessary to add a lot of codes for detecting the processing time in the service code, which will increase the amount of additional code. Based on this, the total processing time of the second model to be deployed and the first processing time of the second pre-filling unit can be obtained first, and then the second processing time can be obtained by the difference between the total processing time and the first processing time, thereby reducing the amount of additional code, improving the detection efficiency of the processing time, improving the efficiency of model quantization, and improving the deployment efficiency of the target service.

[0084] The service deployment method provided in this embodiment inputs the second test data into the second model to be deployed for processing, and obtains the total processing time of the second model to be deployed and the first processing time of the second pre-filling unit. The first processing time is used to characterize the processing performance of the second pre-filling unit. Furthermore, based on the difference between the total processing time and the first processing time, the second processing time of the second decoding unit is determined, and the second processing time is used to characterize the processing performance of the second decoding unit. Therefore, the analysis time of the processing performance of the second pre-filling unit and the second decoding unit can be effectively reduced, and the analysis efficiency of the processing performance can be effectively improved.

[0085] In some optional implementations, the step b2 of inputting the second test data into the second model to be deployed for processing to obtain the total processing time of the second model to be deployed and the first processing time of the second pre-filled unit includes:

[0086] Step b21, obtaining input and output constraints of the first model to be deployed, where the input and output constraints include a target number of input words and a target number of output words.

[0087] Specifically, after determining the first model to be deployed of the target service, it is necessary to further analyze the number of input word units and the number of output word units of the first model to be deployed to obtain input and output constraints.

[0088] Step b22: Under the input and output constraints, the second test data is input into the second model to be deployed for processing to obtain the total processing time and the first processing time.

[0089] Specifically, the number of input words of the second model to be deployed is constrained to be the target number of input words, and the number of output words of the second model to be deployed is constrained to be the target number of output words. Under the same input and output conditions, the second test data is input into the second model to be deployed for processing, and the total processing time of each second model to be deployed and the first processing time are obtained.

[0090] The service deployment method provided in this embodiment inputs the second test data into the second to-be-deployed model for processing under the input-output constraint to obtain the total processing time and the first processing time. Therefore, the accuracy of the processing time of the second pre-filling unit and the second decoding unit can be improved.

[0091] In some optional embodiments, obtaining the input and output constraints of the first model to be deployed in the above step b21 includes: obtaining third test data; and analyzing the quantitative performance of the first model to be deployed under different numbers of input and output words based on the third test data to determine the input and output constraints.

[0092] It should be noted that the third test data is test data under the target service, and the third test data includes at least one third test sample for performance testing.

[0093] It is worth noting that in the case of long word input or output, the loss of model quantization is relatively large. At this time, the model calculation accuracy after model quantization may not meet expectations, so the first model to be deployed cannot be quantized, or a high-bit quantization method such as 16bit is required. In addition, in the case of long word input or output, the speed of model quantization will also be affected. Therefore, it is necessary to analyze the quantization performance of the first model to be deployed under different numbers of input and output words to determine the impact of the number of input words and output words on model quantization. Then, select the number of input words and the number of output words with the best quantization performance as input and output constraints.

[0094] The service deployment method provided in this embodiment analyzes the quantization performance of the first model to be deployed under different numbers of input and output word units to determine the input and output constraints. Therefore, it is possible to ensure that the processing performance of the second pre-filling unit and the second decoding unit in each second model to be deployed is analyzed under the same conditions, thereby improving the accuracy of the processing performance analysis. At the same time, by constraining the input word units and the output word units, the quantization speed and the calculation accuracy of the model after quantization are ensured.

[0095] In some optional implementations, the comparison result based on the processing performance in the above step S203 is used to obtain the first target quantization mode of the first pre-filling unit and the second target quantization mode of the first decoding unit, including:

[0096] Step c1, determining a first target quantization method based on a comparison result of the first processing duration.

[0097] Specifically, the preset quantization method corresponding to the shortest first processing duration is determined as the first target quantization method.

[0098] Step c2: determining a second target quantization method based on the comparison result of the second processing time.

[0099] Specifically, the preset quantization method corresponding to the shortest second processing duration is determined as the second target quantization method.

[0100] The service deployment method provided in this embodiment determines the first target quantization method and the second target quantization method based on the comparison results of the first processing time and the second processing time, respectively. Therefore, it can effectively reduce the time consumption of the quantized model in the reasoning process to improve the model reasoning efficiency.

[0101] In this embodiment, a third service deployment method is provided, which can be used in a service deployment platform. The service deployment platform is used to deploy a model for implementing a target service. Figure 3 is a flow chart of a second service deployment method according to an embodiment of the present disclosure, such as Figure 3 As shown, the process includes the following steps:

[0102] Step S301, obtaining a first to-be-deployed model of a target service, the first to-be-deployed model including a first pre-filling unit and a first decoding unit. Please refer to the above step S101, which will not be described in detail here.

[0103] Step S302, quantize the first model to be deployed based on multiple preset quantization methods to obtain the processing performance of the second pre-filling unit and the second decoding unit in the corresponding second model to be deployed. Please refer to the above step S202, which will not be repeated here.

[0104] Step S303: Based on the comparison result of the processing performance, the first target quantization mode of the first pre-filling unit and the second target quantization mode of the first decoding unit are obtained respectively. Please refer to the above step S103, which will not be described in detail here.

[0105] Step S304, based on the first target quantization method and the second target quantization method, the first pre-filling unit and the first decoding unit are quantized respectively to obtain a target deployment model, and the target deployment model includes a third pre-filling unit and a third decoding unit.

[0106] Specifically, the third pre-filling unit is obtained by quantizing the first pre-filling unit based on the first target quantization method.

[0107] Specifically, the third decoding unit is obtained by quantizing the first decoding unit based on the second target quantization method.

[0108] Step S305: deploy the target deployment model to the target device to deploy the target service.

[0109] Specifically, the above step S305 includes:

[0110] Step S3051, determining deployable devices for the target service.

[0111] Specifically, the deployable device is a computing node used to deploy the target service.

[0112] Optionally, if the target service is related to graphics processing, the deployable device may be a graphics processing unit (GPU).

[0113] Step S3052: Determine, from the deployable devices, a first target device for deploying the third pre-filling unit and a second target device for deploying the third decoding unit.

[0114] Specifically, a performance evaluation is performed on the deployable device to determine the device performance of the deployable device, and a first target device and a second target device are determined in the deployable device based on the device performance.

[0115] Step S3053: deploy the third pre-filling unit into the first target device, and deploy the third decoding unit into the second target device, so as to deploy the target service.

[0116] Specifically, the third pre-filling unit is deployed to the first target device and the third decoding unit is deployed to the second target device in a separate deployment manner, and the first target device and the second target device jointly process the target service.

[0117] The service deployment method provided in this embodiment takes into account the different computing characteristics of the pre-filling stage and the decoding stage, and determines a first target device for deploying the third pre-filling unit and a second target device for deploying the third decoding unit in the deployable device of the target service. Then, the third pre-filling unit and the third decoding unit are deployed to the first target device and the second target device, respectively. Therefore, the target deployment model can be run on different devices in the pre-filling stage and the decoding stage of the reasoning process, so as to make better use of the computing performance of the device and improve the throughput.

[0118] As a specific application example, a target application is installed on the service deployment platform, and the target application is used to deploy a target service. The target service can be deployed in the target application using the service deployment method disclosed in the present disclosure to improve the deployment efficiency of the target service while ensuring the performance of the target service.

[0119] For example, the deployment architecture of the target service of the present disclosure is as follows: Figure 4 As shown. Assume that the target service corresponds to 3 first target devices for the calculation process of the pre-filling stage, and 3 second target devices for the calculation process of the decoding stage. The service deployment method disclosed in the present invention can be used to obtain the third pre-filling unit and the third decoding unit corresponding to the target service. Then, the third pre-filling unit is deployed to the first target device 11, the first target device 12, and the first target device 13. And, the third decoding unit is deployed to the second target device 21, the second target device 22, and the second target device 23. If the inference middleware (also known as Inference MiddleWare) receives a request for the target service, such as requests 1 to 4, requests 1 to 4 are respectively assigned to the corresponding first target devices for processing. The processing result of the first target device is then sent to the corresponding second target device, which is processed by the second target device, and the processing result of the second target device is returned through the inference middleware.

[0120] Exemplarily, taking the first model to be deployed as the language model Yi-6B and Yi-34B as an example, three different quantization algorithms, fp16, W8, and W8A8, are introduced under batch sizes of 1 and 16, respectively, and the time consumption of Yi-6B and Yi-34B in the pre-filling stage and the decoding stage are statistically analyzed to obtain the time consumption data shown in Tables 1 and 2. As shown in Table 1, when the batch size is 1, Yi-6B has the least computational time after using W8A8 for model quantization in the pre-filling stage, and Yi-6B has the least computational time after using W8 for model quantization in the decoding stage. Yi-34B has the least computational time after using W8A8 for model quantization in the pre-filling stage, and Yi-34B has the least computational time after using W8 for model quantization in the decoding stage. As shown in Table 2, when the batch size is 16, Yi-6B uses W8A8 to quantize the model in the pre-filling stage, and the calculation time of Yi-6B is the least after using W8 to quantize the model in the decoding stage. Yi-34B uses W8A8 to quantize the model in the pre-filling stage, and the calculation time of Yi-34B uses W8 to quantize the model in the decoding stage. It can be seen that due to the intensive calculation in the pre-filling stage, the use of a quantization algorithm that only quantizes the weights, such as fb16, will reduce the processing speed of the language model. In the decoding stage, the W8A8 quantization algorithm that uses quantization technology to introduce additional computing nodes is slightly slower than the processing speed of the W8 quantization algorithm. Based on this, in order to reduce the inference delay of the language model and further improve the throughput, the present disclosure adopts a quantization algorithm suitable for the current inference stage in different inference stages, so that different inference stages can obtain the best inference performance as much as possible.

[0121] Table 1 Time consumption when batch size is 1

[0122]

[0123] Table 2 Time consumption when the batch size is 16

[0124]

[0125] It is worth noting that the adaptive quantization method of the pre-filling stage and the decoding stage proposed by the service deployment method of the present invention can be well combined with the separate deployment method. According to the difference in the computational characteristics of the pre-filling stage and the decoding stage in the model reasoning process, the present invention selects quantization methods for the pre-filling stage and the decoding stage based on performance analysis, which can ensure the best reasoning performance while ensuring the availability of the quantization method. Combined with the separate deployment method, the pre-filling stage and the decoding stage in the model reasoning process are deployed separately. Therefore, it is possible to improve throughput while reducing reasoning delays, thereby effectively improving the performance of the target service.

[0126] It is worth noting that, in the actual service deployment process, the service deployment method disclosed in the present invention can also be integrated with optimization strategies such as continuous batching to further improve the model service throughput.

[0127] In this embodiment, a service deployment device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0128] This embodiment provides a service deployment device, such as Figure 5 As shown, including:

[0129] A model acquisition module 501 is used to acquire a first to-be-deployed model of a target service, where the first to-be-deployed model includes a first pre-filling unit and a first decoding unit;

[0130] The model processing module 502 is used to quantize the first model to be deployed based on multiple preset quantization methods to obtain the processing performance of the second pre-filling unit and the second decoding unit in the corresponding second model to be deployed;

[0131] A mode determination module 503, used to obtain a first target quantization mode of the first pre-filling unit and a second target quantization mode of the first decoding unit based on the comparison result of the processing performance;

[0132] The unit quantization module 504 is used to quantize the first pre-filling unit and the first decoding unit respectively based on the first target quantization method and the second target quantization method to obtain a target deployment model;

[0133] The model deployment module 505 is used to deploy the target deployment model to the target device to deploy the target service.

[0134] In some optional implementations, the model processing module 502 includes:

[0135] A first processing unit, configured to quantize the first model to be deployed based on a plurality of preset quantization methods to obtain a corresponding second model to be deployed;

[0136] A second processing unit, used to determine the performance difference between each second model to be deployed and the first model to be deployed;

[0137] The third processing unit is used to determine the processing performance of the second pre-filling unit and the second decoding unit in the second to-be-deployed model if the performance difference is less than a preset difference threshold.

[0138] In some optional implementations, the second processing unit includes:

[0139] A first data acquisition subunit, used to acquire first test data;

[0140] A first performance testing subunit is used to perform performance tests on the first model to be deployed and the second model to be deployed based on the first test data, respectively, to obtain a first performance indicator of the first model to be deployed and a second performance indicator of each second model to be deployed;

[0141] The difference operation subunit is used to obtain the performance difference of the corresponding second model to be deployed based on the difference between the second performance indicator and the first performance indicator.

[0142] In some optional implementations, the third processing unit includes:

[0143] A second data acquisition subunit, used to acquire second test data;

[0144] A first duration determination subunit is used to input the second test data into the second model to be deployed for processing, to obtain a total processing duration of the second model to be deployed and a first processing duration of the second pre-filling unit, wherein the first processing duration is used to characterize the processing performance of the second pre-filling unit;

[0145] The second duration determination unit is used to determine a second processing duration of the second decoding unit based on the difference between the total processing duration and the first processing duration, where the second processing duration is used to characterize the processing performance of the second decoding unit.

[0146] In some optional embodiments, the first duration determination subunit is specifically used to: obtain input and output constraints of the first model to be deployed, the input and output constraints including a target number of input words and a target number of output words; under the input and output constraints, input the second test data into the second model to be deployed for processing to obtain a total processing duration and a first processing duration.

[0147] In some optional embodiments, the first duration determination subunit is used to obtain input and output constraints of the first model to be deployed, including: obtaining third test data; based on the third test data, analyzing the quantitative performance of the first model to be deployed under different input and output word numbers to determine the input and output constraints.

[0148] In some optional implementations, the mode determination module 503 includes:

[0149] A preset quantization mode determining unit, configured to determine a first target quantization mode based on a comparison result of the first processing duration;

[0150] The second quantization mode determining unit is used to determine a second target quantization mode based on the comparison result of the second processing time length.

[0151] In some optional implementations, the target deployment model includes a third pre-filling unit and a third decoding unit. Then, the model deployment module 505 includes:

[0152] A deployable device determining unit, used to determine a deployable device for a target service;

[0153] a target device determining unit, configured to determine, in the deployable device, a first target device for deploying the third pre-filling unit and a second target device for deploying the third decoding unit;

[0154] The target service deployment unit is used to deploy the third pre-filling unit into the first target device and deploy the third decoding unit into the second target device to deploy the target service.

[0155] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0156] The service deployment device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0157] The present disclosure also provides an electronic device having the above Figure 5 The service deployment device shown.

[0158] See also Figure 6 , Figure 6 is a structural block diagram of an electronic device provided by an optional embodiment of the present disclosure, such as Figure 6As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0159] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0160] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0161] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0163] The electronic device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0164] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0165] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0166] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0167] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A service deployment method, characterized in that: The method comprises: Acquire a first to-be-deployed model of a target service, wherein the first to-be-deployed model includes a first pre-filling unit and a first decoding unit; quantizing the first to-be-deployed model based on a plurality of preset quantization methods respectively, and obtaining processing performance of a second pre-filling unit and a second decoding unit in a corresponding second to-be-deployed model; Based on the comparison result of the processing performance, respectively obtaining a first target quantization mode of the first pre-filling unit and a second target quantization mode of the first decoding unit; Based on the first target quantization method and the second target quantization method, quantizing the first pre-filling unit and the first decoding unit respectively to obtain a target deployment model; The target deployment model is deployed to a target device to deploy the target service.

2. The method according to claim 1, characterized in that The step of quantizing the first model to be deployed based on a plurality of preset quantization methods to obtain processing performance of a second pre-filling unit and a second decoding unit in a corresponding second model to be deployed includes: Quantize the first models to be deployed respectively based on the multiple preset quantization methods to obtain the corresponding second models to be deployed; Determine a performance difference between each of the second models to be deployed and the first models to be deployed; If the performance difference is less than a preset difference threshold, the processing performance of the second pre-filling unit and the second decoding unit in the second to-be-deployed model is determined.

3. The method according to claim 2, characterized in that The determining the performance difference between each of the second models to be deployed and the first models to be deployed includes: Acquire first test data; Based on the first test data, a performance test is performed on the first model to be deployed and the second model to be deployed respectively to obtain a first performance indicator of the first model to be deployed and a second performance indicator of each of the second models to be deployed; Based on the difference between the second performance indicator and the first performance indicator, a corresponding performance difference of the second model to be deployed is obtained.

4. The method according to claim 2, characterized in that: The determining the processing performance of the second pre-filling unit and the second decoding unit in the second to-be-deployed model includes: Acquire second test data; Inputting the second test data into the second model to be deployed for processing, obtaining a total processing time of the second model to be deployed and a first processing time of the second pre-filling unit, wherein the first processing time is used to characterize the processing performance of the second pre-filling unit; Based on the difference between the total processing time and the first processing time, a second processing time of the second decoding unit is determined, where the second processing time is used to characterize the processing performance of the second decoding unit.

5. The method according to claim 4, characterized in that The step of inputting the second test data into the second model to be deployed for processing, and obtaining a total processing time of the second model to be deployed and a first processing time of the second pre-filling unit, includes: Obtaining input and output constraints of the first model to be deployed, wherein the input and output constraints include a target number of input tokens and a target number of output tokens; Under the input-output constraints, the second test data is input into the second model to be deployed for processing to obtain the total processing time and the first processing time.

6. The method according to claim 5, characterized in that The obtaining of input and output constraints of the first model to be deployed includes: Acquire third test data; Based on the third test data, the quantitative performance of the first model to be deployed under different numbers of input and output word units is analyzed to determine the input and output constraints.

7. The method according to claim 4, characterized in that The first target quantization mode of the first pre-filling unit and the second target quantization mode of the first decoding unit are obtained based on the comparison result of the processing performance, including: Determining the first target quantization method based on the comparison result of the first processing time; Based on the comparison result of the second processing time, the second target quantization method is determined.

8. The method according to claim 1, characterized in that The target deployment model includes a third pre-filling unit and a third decoding unit, and deploying the target deployment model to the target device to deploy the target service includes: Determining deployable devices for the target service; Determining, in the deployable device, a first target device for deploying the third pre-filling unit and a second target device for deploying the third decoding unit; The third pre-filling unit is deployed in the first target device, and the third decoding unit is deployed in the second target device, so as to deploy the target service.

9. A service deployment device, characterized in that: The device comprises: A model acquisition module, used to acquire a first to-be-deployed model of a target service, wherein the first to-be-deployed model includes a first pre-filling unit and a first decoding unit; A model processing module, used to quantize the first model to be deployed based on a plurality of preset quantization methods, respectively, to obtain processing performance of a second pre-filling unit and a second decoding unit in a corresponding second model to be deployed; a mode determination module, configured to obtain a first target quantization mode of the first pre-filling unit and a second target quantization mode of the first decoding unit based on the comparison result of the processing performance; A unit quantization module, configured to quantize the first pre-filling unit and the first decoding unit respectively based on the first target quantization mode and the second target quantization mode to obtain a target deployment model; The model deployment module is used to deploy the target deployment model to the target device to deploy the target service.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the service deployment method according to any one of claims 1 to 8 by executing the computer instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the service deployment method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the service deployment method according to any one of claims 1 to 8.

Images