A model deployment method, device, equipment, storage medium and product

Abstract

The application discloses a model deployment method and device, equipment, storage medium and product, relates to the technical field of multi-element heterogeneous computing systems, and determines total time consumption of each expert model in performing an inference task on a corresponding heterogeneous computing node according to distributed inference task information of a mixed expert model and performance information of a heterogeneous computing system. Based on a load balancing principle, total time consumption, communication time consumption and calculation time consumption of all expert models are analyzed to determine a compression ratio. Each expert model is iteratively compressed according to a model compression strategy to obtain each compressed expert model that meets error requirements and compression ratio requirements. Each compressed expert model is deployed on a corresponding heterogeneous computing node. By compressing the expert model and determining the compression degree of the expert model based on the compression ratio, the calculation time consumption of different heterogeneous computing power in the expert operation layer is balanced as much as possible, and the utilization rate of hardware resources is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-element heterogeneous computing system, and particularly relates to a model deployment method and device, equipment, storage medium and product. BACKGROUND

[0002] In recent years, the concept of multi-element heterogeneous computing system has been proposed. In such a system, heterogeneous computing nodes with different computing performance are integrated into the same distributed computing environment and cooperatively complete the distributed inference task of the mixed expert model. The mixed expert (Mixture of Experts, MoE) is a model parallelization method, which divides a large model into multiple expert models and deploys each expert model on a different heterogeneous computing node, thereby improving the training and inference efficiency.

[0003] Due to the differences in performance and communication capabilities of different heterogeneous computing nodes in the heterogeneous computing system, the end times of the expert models executing the computing task on different heterogeneous computing nodes are inconsistent. The process of expert model parallel computing needs to wait for the slowest expert model to complete, which results in low utilization of hardware resources.

[0004] Therefore, how to improve the utilization of hardware resources is a problem to be solved by those skilled in the art. SUMMARY

[0005] The present application provides a model deployment method, device, equipment, storage medium and product to at least solve the problem of low utilization of hardware resources in the related art.

[0006] The present application provides a model deployment method, comprising:

[0007] obtaining distributed inference task information of a mixed expert model and performance information of a heterogeneous computing system; wherein the heterogeneous computing system comprises a plurality of heterogeneous computing nodes; the mixed expert model comprises a plurality of expert models;

[0008] determining the total time consumption of each expert model in executing the inference task on its corresponding heterogeneous computing node according to the performance information of the heterogeneous computing system and the distributed inference task information; wherein the total time consumption comprises communication time consumption and computing time consumption;

[0009] analyzing the total time consumption, communication time consumption and computing time consumption of all expert models based on the load balancing principle to determine the compression ratio;

[0010] iteratively compressing each expert model according to the model compression strategy to obtain each compressed expert model meeting the error requirement and the compression ratio requirement;

[0011] According to the correspondence between the expert models and the heterogeneous computing nodes, each compressed expert model is deployed on a corresponding heterogeneous computing node.

[0012] The application also provides a model deployment device, including an acquisition unit, a time consumption determination unit, a compression ratio determination unit, a compression unit and a deployment unit.

[0013] The acquisition unit is used to acquire distributed inference task information of the mixed expert model and performance information of the heterogeneous computing system; wherein the heterogeneous computing system includes a plurality of heterogeneous computing nodes; and the mixed expert model includes a plurality of expert models.

[0014] The time consumption determination unit is used to determine total time consumption of each expert model in performing an inference task on a corresponding heterogeneous computing node according to the performance information of the heterogeneous computing system and the distributed inference task information; wherein the total time consumption includes communication time consumption and calculation time consumption.

[0015] The compression ratio determination unit is used to analyze the total time consumption, the communication time consumption and the calculation time consumption of all the expert models based on a load balancing principle to determine a compression ratio.

[0016] The compression unit is used to iteratively compress each expert model according to a model compression strategy to obtain each compressed expert model meeting an error requirement and a compression ratio requirement.

[0017] The deployment unit is used to deploy each compressed expert model on a corresponding heterogeneous computing node according to the correspondence between the expert models and the heterogeneous computing nodes.

[0018] The application also provides an electronic device, including a memory for storing a computer program and a processor for executing the computer program to implement the steps of any one of the above model deployment methods.

[0019] The application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the steps of any one of the above model deployment methods.

[0020] The application also provides a computer program product, including a computer program, and the computer program is executed by a processor to implement the steps of any one of the above model deployment methods.

[0021] The distributed inference task information of the mixed expert model and the performance information of the heterogeneous computing system are acquired through the application. The heterogeneous computing system includes a plurality of heterogeneous computing nodes. The mixed expert model includes a plurality of expert models. The distributed inference task information contains the architecture information of the expert model. The architecture is different, and the time consumption of executing the inference task is also different. The performance of the heterogeneous computing node also affects the time consumption of the inference task. Therefore, according to the performance information of the heterogeneous computing system and the distributed inference task information, the total time consumption of each expert model executing the inference task on the corresponding heterogeneous computing node can be determined. The total time consumption includes communication time consumption and calculation time consumption. In order to make the time consumption of different expert models executing the inference task as balanced as possible, the total time consumption, the communication time consumption and the calculation time consumption of all expert models can be analyzed based on the load balancing principle to determine the compression ratio. Each expert model is iteratively compressed according to the model compression strategy to obtain each compressed expert model that meets the error requirement and the compression ratio requirement. According to the correspondence between the expert model and the heterogeneous computing node, each compressed expert model is deployed on the corresponding heterogeneous computing node. In the application, the expert model deployed on different heterogeneous computing nodes is adjusted by model compression, and the compression degree of the expert model is determined based on the compression ratio, so that the calculation time consumption of different heterogeneous computing power in the expert operation layer is balanced as much as possible, the expert neural network inference task is accelerated, and the need for long-time waiting for the slowest expert model to complete calculation is effectively avoided, thereby improving the utilization rate of hardware resources. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 A schematic diagram of a mixed expert model executing distributed inference on a heterogeneous computing system;

[0024] Figure 2 A flowchart of a model deployment method provided by an embodiment of the present application;

[0025] Figure 3 A time consumption schematic diagram of a plurality of compressed expert models executing a calculation task provided by an embodiment of the present application;

[0026] Figure 4 A flowchart of a method for compressing an expert model provided by an embodiment of the present application;

[0027] Figure 5 A structural schematic diagram of a model deployment device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0029] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0030] In the MoE architecture, the mixed expert model is split into multiple expert models, each of which is responsible for a part of the task, and the expert model is usually a small feedforward neural network (FNN). The heterogeneous computing system includes multiple heterogeneous computing nodes, which can be accelerator cards of different manufacturers and different computing performance. The communication between the heterogeneous computing nodes can be intra-server communication or inter-server communication.

[0031] Figure 1 A schematic diagram of distributed inference of a mixed expert model on a heterogeneous computing system, the mixed expert model comprising multiple expert models, Figure 1 In the specific embodiment, four expert models are taken as examples, namely expert model 1, expert model 2, expert model 3 and expert model 4. Each expert model has its corresponding input and output, Figure 1 In the specific embodiment, different labels are used to distinguish the inputs and outputs of different expert models. Each expert model is deployed on a corresponding heterogeneous computing node, and one heterogeneous computing node is one heterogeneous computing power, Figure 2 In the specific embodiment, four heterogeneous computing powers are taken as examples, namely heterogeneous computing power 1, heterogeneous computing power 2, heterogeneous computing power 3 and heterogeneous computing power 4. Multiple expert models share one large model space, but each time the input data only activates a part of the expert models for calculation, and the corresponding task is allocated to the activated expert model by the expert selection gate function. Figure 1The MoE architecture can realize parallel computing between the expert models by assigning different expert models to different heterogeneous computing nodes. This parallelization can greatly improve the inference speed and the scalability of the model.

[0032] Due to the differences in performance and communication capabilities of the heterogeneous computing system, the end times of the expert models executing the inference task on different heterogeneous computing nodes are different. The expert parallel computing process needs to wait for the slowest expert neural network to finish computing, resulting in low utilization of hardware resources.

[0033] The present application considers that if the time consumption of all expert models can be consistent or similar, the load of each heterogeneous computing power can be balanced as much as possible, thereby improving the utilization of hardware resources. Therefore, the model deployment method, device, equipment, storage medium and product provided by the embodiments of the present application adjust the expert models deployed on different heterogeneous computing nodes through model compression, and determine the compression degree of the expert models based on the compression rate, so that the computing time consumption of different heterogeneous computing powers in the expert operation layer is as balanced as possible, the expert neural network inference task is accelerated, and the need to wait for the slowest expert model to finish computing for a long time is effectively avoided, thereby improving the utilization of hardware resources.

[0034] When performing model compression, fine-tuning training can be performed on the target expert model after each compression. Fine-tuning refers to small-scale training of a pre-trained model based on specific task targets (downstream tasks) and task data (downstream data), which realizes the fine adjustment of the pre-trained model parameters, and finally obtains a model adapted to specific tasks and data.

[0035] In order to enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Figure 2 The flowchart of the model deployment method provided by the embodiments of the present application includes:

[0037] S201: Obtain the distributed inference task information of the mixed expert model and the performance information of the heterogeneous computing system.

[0038] The heterogeneous computing system includes a plurality of heterogeneous computing nodes, and one heterogeneous computing node can be regarded as one heterogeneous computing power. The mixed expert model includes a plurality of expert models.

[0039] The distributed inference task information includes architecture information of the expert model, such as network structure of each expert model, input and output size, number of activated expert models in each round of inference, correspondence relationship between the expert model and the heterogeneous computing node, and the like.

[0040] The network structure of the expert model can include how many neural layers the expert model contains, type of each layer, input and output structure of each layer, and the like. Most of the current expert neural networks are fully connected layers.

[0041] According to the correspondence relationship between the expert model and the heterogeneous computing node, it can be determined that each expert model is deployed on which heterogeneous computing node. The correspondence relationship can be directly obtained through the task demand input by the user.

[0042] The performance information of the heterogeneous computing system is used to represent the network communication capability of the heterogeneous computing power in the heterogeneous computing system. The performance information can include uplink and downlink bandwidth, delay information, and the like of each external link of the heterogeneous computing power. The uplink and downlink bandwidth includes uplink bandwidth and downlink bandwidth. The delay information includes uplink delay and downlink delay.

[0043] In actual application, the performance information of each heterogeneous computing node can be obtained through server performance test tool (benchmark) test.

[0044] S202: According to the performance information of the heterogeneous computing system and the distributed inference task information, the total time consumption of each expert model in executing the inference task on the corresponding heterogeneous computing node is determined.

[0045] The total time consumption can include communication time consumption and calculation time consumption.

[0046] In the embodiment of the present application, the calculation time consumption of each expert model in executing the inference task on the corresponding heterogeneous computing node can be obtained. According to the input size and output size included in the distributed inference task information, the uplink bandwidth, the downlink bandwidth, and the delay information included in the performance information, and the number of activated expert models in each round of inference, the communication time consumption is determined. The sum of the calculation time consumption and the communication time consumption of each expert model is taken as the total time consumption of each expert model.

[0047] For determination of the communication time consumption, the downlink data amount can be determined according to the input size and the number of activated expert models in each round of inference; the ratio of the downlink data amount to the downlink bandwidth is taken as the downlink communication time consumption. The uplink data amount is determined according to the output size and the number of activated expert models in each round of inference; the ratio of the uplink data amount to the uplink bandwidth is taken as the uplink communication time consumption. The downlink communication time consumption, the uplink communication time consumption, and the uplink delay and the downlink delay included in the performance information are summed to obtain the communication time consumption.

[0048] In a specific implementation, the total time consumption can be calculated according to the following formula:

[0049] ;

[0050] wherein, denotes the total time consumption of the ith expert model, denotes the calculation time consumption of the ith expert model, , denotes the communication time consumption of the ith expert model, , M i denotes the ith expert model, A denotes the number of expert models activated per round of reasoning, D in denotes the input size, D out denotes the output size, denotes the uplink bandwidth, denotes the downlink bandwidth, L i denotes the time delay. Since the uplink time delay and the downlink time delay are the same, the time delay in the communication time consumption is 2L i .

[0051] S203: Based on the load balancing principle, the total time consumption, the communication time consumption and the calculation time consumption of all the expert models are analyzed to determine the compression ratio.

[0052] Since the correspondence between the expert models and the heterogeneous computing nodes has been fixed, the communication time consumption of each expert model is difficult to change when the heterogeneous computing power is specified, and therefore the calculation time consumption is changed in the model compression in the embodiments of the present application, so that each expert model can complete the reasoning task as much as possible at the same time. By obtaining the calculation time consumption and the communication time consumption of each expert model, the sum of the calculation time consumption and the communication time consumption is taken as the total time consumption, which provides a reference basis for determining the compression ratio.

[0053] In order to evaluate the compression degree of the model, the compression ratio can be determined first. The compression ratio can be regarded as the ratio of the calculation amount of the compressed expert model to the calculation amount of the expert model before compression.

[0054] In the embodiments of the present application, the total time consumption and the communication time consumption of all the expert models can be compared to determine the optimal load balancing time consumption; according to the total time consumption of each expert model, a target expert model with the maximum total time consumption is selected. In the case where the total time consumption of the target expert model is less than or equal to the optimal load balancing time consumption, the target expert model can be directly output. In the case where the total time consumption of the target expert model is greater than the optimal load balancing time consumption, the compression ratio is determined according to the optimal load balancing time consumption and the calculation time consumption and the total time consumption of the target expert model.

[0055] In order to balance the total time consumption of each expert model, the optimal load balancing time consumption can be determined according to the minimum value of the total time consumption of all expert models. Considering that some expert models may have a long communication time consumption, in order to take this situation into account, the minimum value of the communication time consumption of all expert models can be considered when determining the optimal load balancing time consumption.

[0056] Therefore, in the embodiment of the present application, the minimum value of the total time consumption of all expert models can be screened out according to the total time consumption of all expert models, and the maximum value of the communication time consumption of all expert models can be screened out according to the communication time consumption of all expert models. The maximum value of the total time consumption and the communication time consumption is selected as the optimal load balancing time consumption.

[0057] In a specific implementation, the optimal load balancing time consumption can be determined according to the following formula:

[0058] ;

[0059] Wherein, O represents the optimal load balancing time consumption.

[0060] After the optimal load balancing time consumption is determined, the difference between the total time consumption of the target expert model and the optimal load balancing time consumption is divided by the calculation time consumption of the target expert model to obtain the calculation amount compression ratio. According to the calculation amount compression ratio and the compression ratio, the compression ratio is determined.

[0061] In a specific implementation, the compression ratio can be determined according to the following formula: ;

[0062] Wherein, q represents the compression ratio, C i represents the calculation time consumption of the i th expert model.

[0063] S204: iteratively compressing each expert model according to the model compression strategy to obtain each compressed expert model meeting the error requirement and meeting the compression ratio requirement.

[0064] The model compression strategy is used to represent the way of compressing the model, which can include the compression step and which neurons are compressed. In the embodiment of the present application, the model compression can be realized by model pruning.

[0065] In practical application, the importance of each neuron in the expert model to the overall output of the model can be evaluated, and the neurons with low importance are pruned according to the compression step, so as to obtain the single compressed expert model.

[0066] In order to avoid the influence of single compression on the accuracy of the model, the value of the compression step can be set to be small. Through multiple compression, the final compressed expert model meeting the error requirement and meeting the compression ratio requirement is determined.

[0067] Figure 3 A time-consuming diagram of multiple compressed expert models performing a computing task is provided for an embodiment of the present application, Figure 3 Three expert models are taken as examples in the embodiment of the present application, which are expert model 1, expert model 2 and expert model 3. The three expert models are respectively deployed on different heterogeneous computing nodes. The time consumption of the expert models in performing a computing task mainly includes communication time consumption and computing time consumption. The communication time consumption can include communication data receiving time consumption and communication data sending time consumption. The communication data receiving time consumption can be referred to as communication receiving time consumption, and the communication data sending time consumption can be referred to as communication sending time consumption. By compressing the expert models, the computing time consumption of each expert model can be adjusted, so that the total time consumption of the three expert models is approximately consistent.

[0068] S205: According to the correspondence between the expert models and the heterogeneous computing nodes, each compressed expert model is deployed on the corresponding heterogeneous computing node.

[0069] According to the correspondence between the expert models and the heterogeneous computing nodes, the deployment position of each expert model can be determined, that is, which heterogeneous computing node each expert model is deployed on. After the compression of the expert models is completed, each compressed expert model can be deployed on the corresponding heterogeneous computing node according to the correspondence.

[0070] In the embodiment of the present application, by comprehensively considering the performance of the heterogeneous computing system, the architecture of the mixed expert model and the total time consumption of each expert model in performing an inference task on its corresponding heterogeneous computing node, the compression of the expert model is realized, so that the computing time consumption of different heterogeneous computing power in the expert operation layer is balanced as much as possible, thereby accelerating the expert neural network inference task, achieving the effect of completing the inference request faster with the same computing cost, and improving the utilization rate of the hardware resources of the heterogeneous computing system.

[0071] After each compressed expert model is deployed on the corresponding heterogeneous computing node, these expert models can be used to perform a text response task. For example, a question can be input to an expert model, and the expert model can output a corresponding answer.

[0072] According to the technical solution, distributed inference task information of a mixed expert model and performance information of a heterogeneous computing system are obtained, the heterogeneous computing system includes a plurality of heterogeneous computing nodes, and the mixed expert model includes a plurality of expert models. The distributed inference task information contains architecture information of the expert models. The architecture is different, and the time consumption of performing an inference task is also different. The performance of the heterogeneous computing nodes also affects the time consumption of the inference task. Therefore, according to the performance information of the heterogeneous computing system and the distributed inference task information, the total time consumption of each expert model in performing an inference task on the corresponding heterogeneous computing node can be determined. The total time consumption includes communication time consumption and calculation time consumption. In order to balance the time consumption of different expert models in performing an inference task, the total time consumption, the communication time consumption, and the calculation time consumption of all expert models are analyzed based on a load balancing principle to determine a compression ratio. Each expert model is iteratively compressed according to a model compression strategy to obtain each compressed expert model that meets an error requirement and a compression ratio requirement. According to the correspondence between the expert models and the heterogeneous computing nodes, each compressed expert model is deployed on the corresponding heterogeneous computing node. In this application, the expert models deployed on different heterogeneous computing nodes are adjusted by model compression, and the compression degree of the expert models is determined based on the compression ratio, so that the calculation time consumption of different heterogeneous computing power in the expert operation layer is balanced as much as possible, the expert neural network inference task is accelerated, and the need for long-time waiting for the slowest expert model to complete calculation is effectively avoided, thereby improving the utilization rate of hardware resources.

[0073] Figure 4 A flowchart of a method for compressing an expert model is provided in this application. The method includes:

[0074] S401: Pruning a plurality of neurons with the lowest importance score in the target expert model based on a compression step to obtain a compressed target expert model.

[0075] In each execution of model compression, the expert model with the maximum total time consumption can be selected from all expert models that have not been compressed as the target expert model.

[0076] In the execution of model compression, in order to prevent changing the shape of the input and output and affecting the performance of the expert model, the importance of the output neurons in the full connection layer of the first layer and the last layer of the expert model is evaluated.

[0077] In the embodiment of the present application, the L1 norm of the weight column can be taken as the importance score, and the higher the importance score is, the higher the importance of the neuron is. The L1 norm of the weight of each neuron in the target expert model is taken as the importance score of the neuron. In ascending order of the importance score of each neuron, a plurality of to-be-pruned neurons with the lowest importance score consistent with the number of compression steps are determined. Each to-be-pruned neuron is pruned from the target expert model to obtain a compressed target expert model.

[0078] For example, for the neural network M i In addition to the output neurons in the full connection layer of the first layer and the last layer (to prevent changing the shape of the input and output and affecting the performance of the model), the L1 norm of the weight column is calculated as the importance score: ;

[0079] wherein, represents the L1 norm, W represents the weight matrix of the full connection layer of the target expert model, W j represents the weight of the jth output neuron, represents the importance score of the jth output neuron.

[0080] In the embodiment of the present application, the product value of the set search step and the pruning times can be taken as the compression step. The search step can be represented by K, the pruning times can be represented by u, and the compression step is u*K.

[0081] After determining the importance score of each neuron, u*K neurons can be pruned to obtain a single-compressed expert model. In order to facilitate the distinction, M i represents the expert model before compression, and M represents the single-compressed expert model, and M i represents the compressed expert model that meets the error requirement and meets the compression ratio requirement.

[0082] S402: According to the deviation between the output vector of the target expert model and the output vector of the compressed target expert model, the output error of the compressed target expert model is determined.

[0083] In the embodiment of the present application, the can be fine-tuned based on the distillation loss, wherein the distillation loss is:

[0084] ;

[0085] wherein, represents the distillation loss value, represents the L2 norm, represents the distillation loss function, an output vector of the expert model after single compression, an output vector of the expert model before compression, represents a constant. In order to prevent the denominator from being zero, 0.000001 can be set.

[0086] In order to evaluate the output error of the compressed target expert model, in actual application, the distillation loss value can be taken as the output error.

[0087] S403: Taking the ratio of the calculation amount of the compressed target expert model and the calculation amount of the target expert model as the calculation amount change value.

[0088] For ease of description, the calculation amount of the compressed target expert model can be represented as the calculation amount of the compressed target expert model, and the calculation amount of the target expert model before compression can be represented as the calculation amount of the target expert model before compression.

[0089] S404: In the case that the output error of the compressed target expert model is less than the error threshold and the calculation amount change value of the compressed target expert model is greater than the compression ratio, adjusting the compression step, and taking the compressed target expert model as the latest target expert model.

[0090] The value of the error threshold can be flexibly set based on actual demand. When the accuracy requirement of the expert model is high, the error threshold can be set to be small; when the accuracy requirement of the expert model is not very high, the error threshold can be set to be large. In actual application, the error threshold can be set to 0.1% by default.

[0091] In a specific implementation, in the case that the output error of the compressed target expert model is less than the error threshold, , it is indicated that the compressed target expert model does not meet the compression requirement, and at this time, the compressed target expert model can be further compressed.

[0092] Each time the pruning operation of the expert model is performed, the pruning number is increased by one; wherein, in the initial state, the value of the pruning number is one; and the product value of the set search step and the current value of the pruning number is taken as the adjusted compression step.

[0093] When the compressed target expert model needs to be further compressed, the compression step can be adjusted, the compressed target expert model is taken as the latest target expert model, and then the operation step of S501 is returned to perform pruning on the multiple neurons with the lowest importance score in the target expert model based on the compression step, to obtain the compressed target expert model.

[0094] S405: In a case where the output error of the compressed target expert model is less than the error threshold and the change value of the computational amount of the compressed target expert model is less than or equal to the compression ratio, the compressed target expert model is output.

[0095] In a specific implementation, in a case where the output error of the compressed target expert model is less than the error threshold, if , it is indicated that the compressed target expert model meets the compression requirement, and at this time, the compression of the target expert model can be ended, and the compressed target expert model is directly output.

[0096] It is considered that in actual application, there can be a case where the output error of the compressed target expert model is greater than or equal to the error threshold. Next, the compression processing flow of the model in this case will be introduced.

[0097] In a case where the output error of the compressed target expert model is greater than or equal to the error threshold, it is indicated that the current compressed target expert model is not available, and at this time, the total time consumption of the target expert model can be adjusted according to the computational amount of the target expert model and the computational amount of the compressed target expert model.

[0098] In the embodiment of the present application, the ratio of the computational amount of the compressed target expert model to the computational amount of the target expert model can be multiplied by the computational time consumption of the target expert model to obtain an adjusted computational time consumption; and the sum of the adjusted computational time consumption and the communication time consumption of the target expert model is used as the adjusted total time consumption of the target expert model.

[0099] For example, the ratio of the computational amount of the compressed target expert model to the computational amount of the target expert model is multiplied by C i to obtain the adjusted computational time consumption c i . c i is used to modify and update the current total time consumption: .

[0100] In a case where the change value of the computational amount of the compressed target expert model is less than or equal to the compression ratio, the operation step of selecting the target expert model with the maximum total time consumption can be returned to perform the compression operation on the newly selected target expert model.

[0101] In a case where the change value of the computational amount of the compressed target expert model is greater than the compression ratio, it is indicated that the optimal load balancing time consumption setting is unreasonable, and at this time, the adjusted total time consumption of the target expert model can be used as the optimal load balancing time consumption, and the operation step of selecting the target expert model with the maximum total time consumption is returned to perform the compression operation on the newly selected target expert model.

[0102] In the embodiment of the present application, when the output error of the compressed target expert model is large and the change value of the calculation amount is greater than the compression ratio, the smooth execution of the expert model compression process can be ensured by adjusting the optimal load balancing time consumption, thereby improving the inference speed of the expert model while ensuring the accuracy of the expert model.

[0103] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and necessary general hardware platforms, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0104] Figure 5 A structural schematic diagram of a model deployment device provided in an embodiment of the present application includes an acquisition unit 51, a time consumption determination unit 52, a ratio determination unit 53, a compression unit 54, and a deployment unit 55.

[0105] The acquisition unit 51 is configured to acquire distributed inference task information of a mixed expert model and performance information of a heterogeneous computing system; the heterogeneous computing system includes a plurality of heterogeneous computing nodes; the mixed expert model includes a plurality of expert models.

[0106] The time consumption determination unit 52 is configured to determine total time consumption of each expert model in performing an inference task on a corresponding heterogeneous computing node according to the performance information of the heterogeneous computing system and the distributed inference task information; the total time consumption includes communication time consumption and calculation time consumption.

[0107] The ratio determination unit 53 is configured to analyze the total time consumption, the communication time consumption, and the calculation time consumption of all the expert models based on a load balancing principle to determine a compression ratio.

[0108] The compression unit 54 is configured to iteratively compress each expert model according to a model compression strategy to obtain each compressed expert model that meets error requirements and compression ratio requirements.

[0109] The deployment unit 55 is configured to deploy each compressed expert model on a corresponding heterogeneous computing node according to a correspondence between the expert models and the heterogeneous computing nodes.

[0110] In some embodiments, the distributed inference task information includes network structures of the expert models, input and output sizes, numbers of the expert models activated in each round of inference, and correspondences between the expert models and the heterogeneous computing nodes.

[0111] The time consumption determination unit includes a calculation time consumption acquisition subunit, a communication time consumption determination subunit, and a total time consumption determination subunit.

[0112] The computing time consumption subunit is configured to acquire a computing time consumption of each expert model in performing an inference task on the corresponding heterogeneous computing node.

[0113] The communication time consumption determination subunit is configured to determine a communication time consumption according to an input size and an output size included in the distributed inference task information, uplink bandwidth, downlink bandwidth and time delay information included in the performance information, and a number of activated expert models in each round of inference.

[0114] The total time consumption determination subunit is configured to take a sum of the computing time consumption and the communication time consumption of each expert model as a total time consumption of each expert model.

[0115] In some embodiments, the communication time consumption determination subunit is configured to determine a downlink data volume according to the input size and the number of activated expert models in each round of inference.

[0116] The ratio of the downlink data volume to the downlink bandwidth is taken as a downlink communication time consumption.

[0117] The uplink data volume is determined according to the output size and the number of activated expert models in each round of inference.

[0118] The ratio of the uplink data volume to the uplink bandwidth is taken as an uplink communication time consumption.

[0119] The downlink communication time consumption, the uplink communication time consumption, and uplink time delay and downlink time delay included in the performance information are summed to obtain the communication time consumption.

[0120] In some embodiments, the rate determination unit includes a comparison subunit, a selection subunit, an output subunit and a compression rate determination subunit.

[0121] The comparison subunit is configured to compare the total time consumption and the communication time consumption of all expert models to determine an optimal load balancing time consumption.

[0122] The selection subunit is configured to select a target expert model with the maximum total time consumption according to the total time consumption of each expert model.

[0123] The output subunit is configured to output the target expert model in a case where the total time consumption of the target expert model is less than or equal to the optimal load balancing time consumption.

[0124] The compression rate determination subunit is configured to determine a compression rate according to the optimal load balancing time consumption and the computing time consumption and the total time consumption of the target expert model in a case where the total time consumption of the target expert model is greater than the optimal load balancing time consumption.

[0125] In some embodiments, the comparison subunit is configured to screen a minimum total time consumption according to the total time consumption of all expert models.

[0126] According to the communication time of all expert models, the maximum value of the communication time is screened out;

[0127] The maximum value of the total time and the maximum value of the communication time is selected as the optimal load balancing time.

[0128] In some embodiments, the compression ratio determination subunit is configured to divide the difference between the total time of the target expert model and the optimal load balancing time by the calculation time of the target expert model to obtain a calculation amount compression ratio; and determine the compression ratio according to the calculation amount compression ratio and the compression ratio being equal to one.

[0129] In some embodiments, the compression unit includes a pruning subunit, an error determination subunit, a change value determination subunit, an adjustment subunit, and a model output subunit.

[0130] The pruning subunit is configured to prune a plurality of neurons with the lowest importance score in the target expert model based on the compression step to obtain a compressed target expert model.

[0131] The error determination subunit is configured to determine the output error of the compressed target expert model according to the deviation between the output vector of the target expert model and the output vector of the compressed target expert model.

[0132] The change value determination subunit is configured to take the ratio of the calculation amount of the compressed target expert model to the calculation amount of the target expert model as the calculation amount change value.

[0133] The adjustment subunit is configured to adjust the compression step in the case that the output error of the compressed target expert model is less than the error threshold and the calculation amount change value of the compressed target expert model is greater than the compression ratio, take the compressed target expert model as the latest target expert model, and return to the operation step of pruning a plurality of neurons with the lowest importance score in the target expert model based on the compression step to obtain a compressed target expert model.

[0134] The model output subunit is configured to output the compressed target expert model in the case that the output error of the compressed target expert model is less than the error threshold and the calculation amount change value of the compressed target expert model is less than or equal to the compression ratio.

[0135] In some embodiments, the pruning subunit is configured to take the L1 norm of the weight of each neuron in the target expert model as the importance score of each neuron; determine a plurality of neurons with the lowest importance score in accordance with the compression step in ascending order of the importance score of each neuron; and prune each neuron to be pruned from the target expert model to obtain a compressed target expert model.

[0136] In some embodiments, the adjusting unit is configured to add one to the pruning number each time the pruning operation of the expert model is performed; wherein the pruning number is initially set to one; and the product of the set search step and the current value of the pruning number is used as the adjusted compression step.

[0137] In some embodiments, the system further comprises a time consumption adjusting unit and a selecting unit;

[0138] The time consumption adjusting unit is configured to, in a case where the output error of the compressed target expert model is greater than or equal to the error threshold, adjust the total time consumption of the target expert model according to the calculation amount of the target expert model and the calculation amount of the compressed target expert model; and in a case where the change value of the calculation amount of the compressed target expert model is less than or equal to the compression ratio, trigger the selecting unit to perform the operation step of selecting the target expert model with the maximum total time consumption.

[0139] The selecting unit is configured to, in a case where the change value of the calculation amount of the compressed target expert model is greater than the compression ratio, use the adjusted total time consumption of the target expert model as the optimal load balancing time consumption, and trigger the selecting unit to perform the operation step of selecting the target expert model with the maximum total time consumption.

[0140] In some embodiments, the time consumption adjusting unit is configured to perform a multiplication operation on the ratio of the calculation amount of the compressed target expert model to the calculation amount of the target expert model and the calculation time consumption of the target expert model to obtain an adjusted calculation time consumption; and use the sum of the adjusted calculation time consumption and the communication time consumption of the target expert model as the adjusted total time consumption of the target expert model.

[0141] According to the technical solution, distributed inference task information of a mixed expert model and performance information of a heterogeneous computing system are obtained. The heterogeneous computing system includes a plurality of heterogeneous computing nodes. The mixed expert model includes a plurality of expert models. The distributed inference task information includes architecture information of the expert models. The architecture is different, and the time consumption of performing an inference task is also different. The performance of the heterogeneous computing nodes also affects the time consumption of the inference task. Therefore, according to the performance information of the heterogeneous computing system and the distributed inference task information, the total time consumption of each expert model in performing an inference task on the corresponding heterogeneous computing node can be determined. The total time consumption includes communication time consumption and calculation time consumption. In order to balance the time consumption of different expert models in performing an inference task, the total time consumption, the communication time consumption, and the calculation time consumption of all expert models are analyzed based on a load balancing principle to determine a compression ratio. Each expert model is iteratively compressed according to a model compression strategy to obtain each compressed expert model that meets an error requirement and a compression ratio requirement. According to the correspondence between the expert models and the heterogeneous computing nodes, each compressed expert model is deployed on the corresponding heterogeneous computing node. In this application, the expert models deployed on different heterogeneous computing nodes are adjusted by model compression, and the compression degree of the expert models is determined based on the compression ratio, so that the calculation time consumption of different heterogeneous computing power in the expert operation layer is balanced as much as possible, the expert neural network inference task is accelerated, and the need for long-time waiting for the slowest expert model to complete calculation is effectively avoided, thereby improving the utilization rate of hardware resources.

[0142] The description of the features in the embodiments of the model deployment device can be referred to the related description of the embodiments of the model deployment method, which will not be repeated here.

[0143] Embodiments of the present application also provide an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the steps in any of the above model deployment method embodiments.

[0144] Embodiments of the present application also provide a computer readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above model deployment method embodiments when running.

[0145] In an exemplary embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0146] The embodiment of the present application further provides a computer program product, the computer program product comprising a computer program, the computer program being executed by a processor to implement the steps in any of the model deployment method embodiments.

[0147] The embodiment of the present application further provides another computer program product, comprising a non-volatile computer readable storage medium, the non-volatile computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps in any of the model deployment method embodiments.

[0148] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0149] The above describes in detail a model deployment method, device, equipment, storage medium and product provided by the present application. The principles and implementation modes of the present application are described by applying specific examples in this paper. The above description of the examples is only applicable to help understand the method and core idea of the present application. It should be pointed out that for ordinary skilled person in the technical field, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A model deployment method, characterized by, The method comprises the following steps: obtaining distributed inference task information of a mixed expert model and performance information of a heterogeneous computing system; wherein the heterogeneous computing system comprises a plurality of heterogeneous computing nodes; the mixed expert model comprises a plurality of expert models; determining total time consumption of each expert model in performing an inference task on a corresponding heterogeneous computing node according to the performance information of the heterogeneous computing system and the distributed inference task information; wherein the total time consumption comprises communication time consumption and calculation time consumption; analyzing the total time consumption, the communication time consumption and the calculation time consumption of all the expert models based on a load balancing principle to determine a compression ratio; iteratively compressing each expert model according to a model compression strategy to obtain each compressed expert model meeting an error requirement and a compression ratio requirement; deploying each compressed expert model on a corresponding heterogeneous computing node according to a correspondence between the expert model and the heterogeneous computing node; analyzing the total time consumption, the communication time consumption and the calculation time consumption of all the expert models based on a load balancing principle to determine a compression ratio, comprising: comparing the total time consumption of all the expert models with the communication time consumption to determine an optimal load balancing time consumption; selecting a target expert model with the maximum total time consumption according to the total time consumption of each expert model; outputting the target expert model in the case that the total time consumption of the target expert model is less than or equal to the optimal load balancing time consumption; determining a compression ratio according to the optimal load balancing time consumption and the calculation time consumption and the total time consumption of the target expert model in the case that the total time consumption of the target expert model is greater than the optimal load balancing time consumption.

2. The model deployment method of claim 1, wherein, The distributed inference task information comprises network structure, input and output size, number of activated expert models per round of inference, and correspondence between the expert model and the heterogeneous computing node of each expert model; determining total time consumption of each expert model in performing an inference task on a corresponding heterogeneous computing node according to the performance information of the heterogeneous computing system and the distributed inference task information, comprising: obtaining calculation time consumption of each expert model in performing an inference task on a corresponding heterogeneous computing node; determining communication time consumption according to input size and output size included in the distributed inference task information, uplink bandwidth, downlink bandwidth and time delay information included in the performance information, and number of activated expert models per round of inference; taking a sum of the calculation time consumption and the communication time consumption of each expert model as the total time consumption of each expert model.

3. The model deployment method of claim 2, wherein, determining communication time consumption according to input size and output size included in the distributed inference task information, uplink bandwidth, downlink bandwidth and time delay information included in the performance information, and number of activated expert models per round of inference, comprising: determining downlink data volume according to the input size and the number of activated expert models per round of inference; taking a ratio of the downlink data volume to the downlink bandwidth as downlink communication time consumption; determining uplink data volume according to the output size and the number of activated expert models per round of inference; taking a ratio of the uplink data volume to the uplink bandwidth as uplink communication time consumption; Summing the downlink communication time consumption, the uplink communication time consumption and the uplink and downlink time delays contained in the performance information to obtain a communication time consumption.

4. The model deployment method of claim 1, wherein, Comparing the total time consumption of all expert models with the communication time consumption to determine an optimal load balancing time consumption, including: Screening a minimum total time consumption from all expert models; Screening a maximum communication time consumption from all expert models; Selecting a maximum value from the minimum total time consumption and the maximum communication time consumption as the optimal load balancing time consumption.

5. The model deployment method of claim 1, wherein, Determining a compression ratio according to the optimal load balancing time consumption and the calculation time consumption and the total time consumption of the target expert model, including: Dividing the difference between the total time consumption of the target expert model and the optimal load balancing time consumption by the calculation time consumption of the target expert model to obtain a calculation amount compression ratio value; Determining the compression ratio according to the calculation amount compression ratio value and the compression ratio and value being one.

6. The model deployment method of claim 1, wherein, Iteratively compressing each expert model according to a model compression strategy to obtain each compressed expert model meeting an error requirement and a compression ratio requirement, including: Pruning a plurality of neurons with the lowest importance scores in the target expert model based on a compression step to obtain a compressed target expert model; Determining an output error of the compressed target expert model according to the deviation between the output vector of the target expert model and the output vector of the compressed target expert model; Taking the ratio of the calculation amount of the compressed target expert model to the calculation amount of the target expert model as a calculation amount change value; In the case that the output error of the compressed target expert model is less than an error threshold and the calculation amount change value of the compressed target expert model is greater than the compression ratio, adjusting the compression step, taking the compressed target expert model as the latest target expert model, and returning to the operation step of pruning the plurality of neurons with the lowest importance scores in the target expert model based on the compression step to obtain a compressed target expert model; In the case that the output error of the compressed target expert model is less than an error threshold and the calculation amount change value of the compressed target expert model is less than or equal to the compression ratio, outputting the compressed target expert model.

7. The model deployment method of claim 6, wherein, Pruning a plurality of neurons with the lowest importance scores in the target expert model based on a compression step to obtain a compressed target expert model, including: Taking the L1 norm of the weight of each neuron in the target expert model as the importance score of each neuron; Determining a plurality of neurons with the lowest importance scores in the order of ascending importance scores of each neuron to be consistent with the number of compression steps; Trimming each neuron to be pruned from the target expert model to obtain a compressed target expert model.

8. The model deployment method of claim 6, wherein, Adjusting the compression step, including: Each time the pruning operation of the expert model is performed, the pruning number is incremented by one; wherein, in the initial state, the pruning number is one; The product of the set search step and the current value of the pruning number is taken as an adjusted compression step.

9. The model deployment method of claim 6, wherein, Further comprising: In a case where an output error of the compressed target expert model is greater than or equal to an error threshold, adjusting a total time consumption of the target expert model according to a calculation amount of the target expert model and a calculation amount of the compressed target expert model; In a case where the calculation amount change value of the compressed target expert model is less than or equal to the compression ratio, returning to the operation step of selecting the target expert model with the maximum total time consumption; In a case where the calculation amount change value of the compressed target expert model is greater than the compression ratio, taking the adjusted total time consumption of the target expert model as an optimal load balancing time consumption, and returning to the operation step of selecting the target expert model with the maximum total time consumption.

10. The model deployment method of claim 9, wherein, Adjusting the total time consumption of the target expert model according to the calculation amount of the target expert model and the calculation amount of the compressed target expert model, comprising: Performing a multiplication operation on a ratio of the calculation amount of the compressed target expert model to the calculation amount of the target expert model and a calculation time consumption of the target expert model to obtain an adjusted calculation time consumption; Taking a sum of the adjusted calculation time consumption and a communication time consumption of the target expert model as the adjusted total time consumption of the target expert model.

11. A model deployment apparatus characterized by comprising: Comprising an acquisition unit, a time consumption determination unit, a compression ratio determination unit, a compression unit, and a deployment unit; The acquisition unit is configured to acquire distributed inference task information of a mixed expert model and performance information of a heterogeneous computing system; the heterogeneous computing system comprises a plurality of heterogeneous computing nodes; the mixed expert model comprises a plurality of expert models; The time consumption determination unit is configured to determine, according to the performance information of the heterogeneous computing system and the distributed inference task information, total time consumptions of each expert model in performing an inference task on a corresponding heterogeneous computing node; the total time consumption comprises a communication time consumption and a calculation time consumption; The compression ratio determination unit is configured to analyze, based on a load balancing principle, the total time consumptions, the communication time consumptions, and the calculation time consumptions of all the expert models to determine a compression ratio; The compression unit is configured to iteratively compress each expert model according to a model compression strategy to obtain each compressed expert model that meets an error requirement and a compression ratio requirement; The deployment unit is configured to deploy each compressed expert model on a corresponding heterogeneous computing node according to a correspondence between the expert model and the heterogeneous computing node. The compression ratio determination unit comprises a comparison subunit, a selection subunit, an output subunit, and a compression ratio determination subunit; The comparison subunit is configured to compare the total time consumptions and the communication time consumptions of all the expert models to determine an optimal load balancing time consumption; The selection subunit is configured to select, according to the total time consumptions of the expert models, a target expert model with the maximum total time consumption; The output subunit is configured to output the target expert model in a case where the total time consumption of the target expert model is less than or equal to the optimal load balancing time consumption. The compression ratio determining subunit is configured to determine a compression ratio according to the optimal load balancing time and the calculation time and the total time of the target expert model, in a case where the total time of the target expert model is greater than the optimal load balancing time.

12. An electronic device, comprising: The model deployment method comprises the following steps: a memory for storing a computer program; a processor for implementing the steps of the model deployment method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the model deployment method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the model deployment method according to any one of claims 1 to 10.

Images