Model deployment methods, electronic devices, storage media, and application products

Abstract

This application discloses a model deployment method, electronic device, storage medium, and program product, relating to the field of computer technology. The method includes: splitting the model to be deployed into multiple operator modules; obtaining the capability indicators of a computing power card; determining the operators supported by the computing power card based on the capability indicators; determining the target computing power card corresponding to the operator module based on the supported operators; and deploying the operator module to the target computing power card. This solves the problem of difficulty in distinguishing the capability differences of different computing power cards when deploying a neural network model across multiple computing power cards, leading to an increase in the overall execution time of the model. This method merges related operators into operator modules when splitting the model to be deployed, avoiding time loss due to data transmission. It determines the operators supported by the computing power card, the energy consumption cost and communication cost of the computing power card through capability indicators, and then determines the target computing power card most suitable for executing the operator module. This enables the deployment of the model on multiple computing power cards, reducing model execution time and improving computing power utilization.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to model deployment methods, electronic devices, storage media, and program products. Background Technology

[0002] As the scale of neural network models continues to increase, a single computing card can no longer meet the comprehensive requirements of computing power, energy consumption, and real-time performance for inference or training. Therefore, it is necessary to deploy neural network models across multiple computing cards. Currently, when deploying neural network models across multiple computing cards, it is difficult to distinguish the capabilities of different cards and to ignore the energy consumption and communication costs of the cards, leading to an increase in the overall execution time of the neural network model. Summary of the Invention

[0003] This invention provides a model deployment method, electronic device, storage medium, and program product to solve the problem that when deploying a neural network model to multiple computing cards, it is difficult to distinguish the capability differences of different computing cards, which leads to an increase in the overall execution time of the model.

[0004] Firstly, this application provides a model deployment method, which includes: The model to be deployed is split into multiple operator modules, wherein each operator module contains at least one operator, and the operator is contained in the model to be deployed; Obtain the capability metrics of multiple computing cards, whereby the capability metrics are used to determine the operators supported by the computing cards; Based on the capability indicators, determine the target computing power card corresponding to the operator module, wherein the target computing power card supports the operators contained in the operator module; Deploy the operator module to the target computing card.

[0005] Secondly, this application provides a model deployment apparatus, the apparatus comprising: The model splitting module is used to split the model to be deployed into multiple operator modules, wherein each operator module contains at least one operator, and the operator is contained in the model to be deployed. The indicator acquisition module is used to acquire the capability indicators of multiple computing cards. The capability indicators are used to determine the operators supported by the computing cards. The computing power card determination module is used to determine the target computing power card corresponding to the operator module based on the capability indicators. The target computing power card supports the operators contained in the operator module. The deployment module is used to deploy the operator module to the target computing card.

[0006] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the model deployment method of the first aspect or any corresponding embodiment described above.

[0007] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the model deployment method of the first aspect or any corresponding embodiment described above.

[0008] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the model deployment method of the first aspect or any corresponding embodiment described above.

[0009] This application addresses the problem of increased overall model execution time when deploying neural network models across multiple computing cards. The method involves splitting the model to be deployed into multiple operator modules; obtaining capability indicators for multiple computing cards; determining the operators supported by each computing card based on these indicators; identifying the target computing card for each operator module based on the supported operators; and deploying the operator modules to the target computing cards. This solution resolves the difficulty in distinguishing the capability differences between different computing cards when deploying a neural network model across multiple computing cards. Furthermore, this method merges related operators into operator modules during the model splitting process, avoiding time losses due to data transmission and improving the utilization rate of computing resources. It determines the capability indicators of the computing cards to differentiate their capabilities. Based on the operators supported by the computing cards, the energy consumption cost, and the communication cost of the computing cards, it identifies the most suitable target computing card for executing the operator modules. This enables the deployment of models across multiple computing cards, achieving unified modeling and fusion scheduling, reducing overall model inference or training time, and improving computing resource utilization. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this application, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of a model deployment system according to an embodiment of this application; Figure 2 This is a flowchart illustrating a model deployment method according to an embodiment of this application; Figure 3 This is a schematic diagram of the heterogeneous computing power fusion process according to an embodiment of this application; Figure 4 This is a schematic diagram of the computing power fusion process according to an embodiment of this application; Figure 5 This is a schematic diagram of the computing power allocation process according to an embodiment of this application; Figure 6 This is a structural block diagram of a model deployment apparatus according to an embodiment of this application; Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0013] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] The specific application environment architecture or specific hardware architecture on which the execution of the model deployment method depends is described here.

[0016] The model deployment method, electronic device, storage medium, and program product provided in this application are applicable to deploying neural network models on multiple computing cards and to the reasonable partitioning and efficient execution of models among different computing cards, thereby reducing overall inference or training latency and improving computing power utilization. Figure 1The diagram shows the structure of the model deployment system of this application. This system is used to deploy the model to be deployed onto multiple computing cards in a computing device, such as computing card 1, computing card 2… computing card N, where N represents multiple cards. The computing device can be a mobile terminal, a fixed terminal, or a portable terminal, etc., including but not limited to mobile phones, desktop computers, laptops, multimedia tablets, e-readers, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Computing cards can be, for example, graphics processing units (GPUs), neural network processing units (NPUs), central processing units (CPUs), tensor processing units (TPUs), or field-programmable gate arrays (FPGAs). The process of deploying the model to be deployed onto the computing device can be as follows: dividing the model to be deployed into multiple operator modules; obtaining the capability indicators of multiple computing cards; determining the target computing card corresponding to the operator module based on the capability indicators; and deploying the operator module onto the target computing card.

[0017] Currently, computing devices typically deploy multiple types of computing cards simultaneously. However, neural network models are usually deployed entirely on a single card, resulting in low utilization of the computing resources of the computing cards in the device. Alternatively, current methods involve simply partitioning the neural network model and deploying it separately on different computing cards. However, this simple model partitioning method does not consider the differences in computing card capabilities, leading to some computing cards being idle. Furthermore, current model partitioning methods ignore the data communication costs between computing cards, increasing the overall execution time. Moreover, due to the different operator types and precision supported by computing cards, current model partitioning methods are prone to producing unexecutable partitioning schemes. Therefore, traditional model deployment methods have the following problems: multiple computing cards cannot share computing power for the same model; the model can only be computed on a single computing card; the same model can only be computed on a single computing card, making it impossible to distribute a large model evenly across different computing cards based on their characteristics; and the impact of actual effective computing power and inter-card communication costs on the overall computing performance is not considered.

[0018] Based on the above, this application provides a model deployment method that establishes a unified computational cost model, energy consumption cost model and communication cost model to achieve reasonable partitioning and efficient execution of neural networks among various computing power cards, thereby reducing overall inference or training latency and improving computing power utilization.

[0019] According to an embodiment of this application, a model deployment method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of executable instructions, such as the model deployment system described above. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0020] This embodiment provides a model deployment method. Figure 2 This is a flowchart of a model deployment method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: Step S201: The model to be deployed is split into multiple operator modules, wherein each operator module contains at least one operator, and the operator is contained in the model to be deployed.

[0021] Specifically, the models to be deployed include, for example, neural network models such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).

[0022] The model to be deployed is broken down to determine the operators contained in the model and the data dependencies between them. If the data dependency between two or more operators exceeds a certain threshold, these two or more operators are merged into one operator module. Therefore, an operator module contains at least one operator. The data dependency between operators is determined based on the amount of data interaction between the two operators. When the neural network model is built, there may be residual relationships, multiplicative relationships, or no relationships between operators.

[0023] Step S202: Obtain the capability indicators of multiple computing cards, wherein the capability indicators are used to determine the operators supported by the computing cards.

[0024] Specifically, currently, computing devices typically deploy multiple types of computing cards simultaneously, such as graphics processing units, neural network processors, central processing units, tensor processing units, and field-programmable gate arrays.

[0025] Obtain multiple capability metrics for the computing card, such as: peak computing power, memory bandwidth, supported operator set, supported data format set, and power consumption. These metrics, especially the supported data format and operator sets, allow you to determine the operators supported by the computing card.

[0026] Step S203: Based on the capability indicators, determine the target computing power card corresponding to the operator module, wherein the target computing power card supports the operators contained in the operator module.

[0027] Specifically, determine the operators included in the operator module. For example, the operator module includes operator 1 and operator 2. The calculation data format required by operator 1 is data format A, and the calculation data format required by operator 2 is data format B.

[0028] Based on the capability indicators, determine the target computing card corresponding to the operator module. For example: if computing card 1 supports operator 1 and operator 2, and supports data format A and data format B, then computing card 1 can be the target computing card for this operator module; if computing card 2 supports operator 1 but not operator 1, then computing card 2 cannot be the target computing card for this operator module; if computing card 3 supports data format A but not data format A, then computing card 3 cannot be the target computing card for this operator module.

[0029] Step S204: Deploy the operator module to the target computing card.

[0030] Specifically, the multiple operator modules of the model to be deployed are split and deployed to the corresponding target computing cards, thus completing the deployment of the model to be deployed to multiple or various computing cards.

[0031] The above process is as follows Figure 3 As shown, the process involves: a computing power fusion process to obtain the capability indicators of multiple computing power cards; a model splitting process to divide the model to be deployed into multiple operator modules; and a computing power allocation process to determine the target computing power cards corresponding to the operator modules based on the capability indicators and deploy the operator modules to the target computing power cards. Additionally, after model deployment, the system checks whether the model output results are normal. If the output results are normal, it checks whether the model's computational output efficiency is normal. If the output efficiency is abnormal, it checks whether the transmission rate between computing power cards is normal. If the transmission rate is normal, it checks whether the operating status of the allocated computing power cards is normal. Furthermore, it checks whether the computing power card output data format is correct and whether the computing power card's data reception / output is normal. If the computing power card's operating status is abnormal, it reallocates the computing power cards. If the computing power card's output data format is incorrect, it performs data format conversion. If the computing power card's data reception / output is abnormal, it checks the network bandwidth.

[0032] The model deployment method provided in this embodiment splits the model to be deployed into multiple operator modules; obtains the capability indicators of multiple computing cards; determines the operators supported by the computing cards based on the capability indicators; determines the target computing card corresponding to the operator module based on the operators supported by the computing cards; and deploys the operator module to the target computing card. This method merges related operators into operator modules when splitting the model to be deployed, avoiding time loss caused by data transmission and improving the utilization rate of computing resources. The capability indicators of the computing cards are determined to distinguish the capability differences between different computing cards. Based on the operators supported by the computing cards, the energy consumption cost and communication cost of the computing cards, the most suitable target computing card for executing the operator module is determined. This enables the deployment of the model on multiple computing cards, achieving unified modeling and fusion scheduling, reducing the overall inference or training time of the model, and improving computing power utilization. It solves the problem that when deploying neural network models on multiple computing cards, it is difficult to distinguish the capability differences of different computing cards, leading to an increase in the overall execution time of the model.

[0033] As an optional implementation, the model to be deployed is divided into multiple operator modules, including: Determine the operators contained in the model to be deployed and the dependency values ​​between operators; If the dependency value is greater than a preset threshold, the operators corresponding to the dependency value will be merged into an operator module; If the dependency value is less than or equal to a preset threshold, the operators corresponding to the dependency values ​​will be treated as operator modules.

[0034] Specifically, this embodiment decomposes the model to be deployed into operators. The operators in the model to be deployed are determined when the model is built, that is, the operators contained in the model to be deployed are experimental parameters or hyperparameters, which can be directly obtained.

[0035] The dependency value between operators is determined based on the amount of data interaction between the two operators. When building a neural network model, there may be residual relationships, multiplicative relationships, or no relationship between operators. For example, if three intervals are set: 0-1MB / s, 2-99MB / s, and greater than 100MB / s, the corresponding dependency values ​​are 0.1, 0.5, and 0.9, respectively. If the data interaction between operator 1 and operator 2 is 101MB / s, then the dependency value between operator 1 and operator 2 is 0.9.

[0036] Based on the above, digital modeling is performed on the model to be deployed to determine the model to be deployed. operators Data dependencies between operators ,For example:

[0037]

[0038] in, , represents an operator node in the network; It represents the data dependency between any two operators.

[0039] Additionally, the feature parameters of the operators can be determined based on the model's input and the amount of data (eigenvalues ​​or feature maps) passed between the two operators. For example: Operator computational cost, measured in FLOPs; Storage space required by the operator, in GB; The size of the operator's output data, in bytes; The type of computational precision required by the operator.

[0040] The model is modularized based on the data dependencies of operators in the model to be deployed. If the dependency is greater than a set threshold, the operators are merged into one module. If there are modules with data backflow or residual connections in some neural networks, their size is calculated to see if it is greater than a set threshold. If so, they are considered to be computed as a whole. Otherwise, they are split according to the parallel lines of the model.

[0041] The preset threshold is set according to actual needs, such as 0.8, 0.9, or other values. The dependency value is compared with the preset threshold. If the dependency value is greater than the preset threshold, the operators corresponding to the dependency value are merged into an operator module. For example, if the preset threshold is 0.8, and the model to be deployed is ResNet50 (including residual connections), a residual block of ResNet50 is selected, containing the operator chain: Conv2d, BN, ReLU, Conv2d, BN, Add. Here, Conv2d is a two-dimensional convolution operator for feature extraction; BN is a batch normalization operator for stable training; ReLU is a ReLU activation operator for introducing non-linearity; and Add is an addition operator for summing residual connections. The dependency between operators within this residual block is 0.95 > 0.8; therefore, the operators in this residual block are merged into a complete operator module.

[0042] In this embodiment, based on the characteristics of the model and the data dependencies between operators, models with data fusion or data backflow are not split, or are split in their parallel direction, to avoid time loss caused by data transmission and improve the utilization rate of computing resources.

[0043] As an optional implementation, the capability metrics of multiple computing cards are obtained, including: Obtain the attribute information and supported operator set of the computing power card; Based on the attribute information, determine the current computing power of the computing card and the execution time of the operators in the supported operator set on the computing card; Based on attribute information, the set of supported operators, current computing power, and execution time, capability metrics are obtained.

[0044] Specifically, this embodiment performs computing power fusion on multiple computing power cards. All computing power cards are initialized; for example, if the computing device contains K computing power cards, the fused computing power is denoted as: , This represents the Kth computing power card.

[0045] Obtain the attribute information and supported operator set of the computing power card. Attribute information such as: video memory bandwidth, video memory capacity, supported data format set, power, etc.

[0046] Perform capability modeling on computing power cards, for example: for any computing power card Define its capability vector as ,in, For computing power cards Peak computing power, in FLOPs / s; For computing power cards The video memory bandwidth, measured in bytes per second; For computing power cards The video memory capacity, in MB; For computing power cards The set of supported operators; For computing power cards Supported data format set; For computing power cards Power, measured in W.

[0047] Computing power cards are all labeled with computing power values, but these values ​​vary in their processing capabilities for different operators. Some cards with high nominal computing power are actually slower at processing certain operators. Therefore, it is necessary to calculate the effective or actual computing power of a computing power card. Thus, based on the same operators supported by each computing power card, the effective or actual computing power of different operators on the computing power card is evaluated, and computing power is subdivided, for example, for the same operator i.

[0048] Determine the peak computing power of the computing card from its attribute information. and memory bandwidth .according to and Determine the current computing power of the computing power card relative to operator i. The execution time of operator i on the computing card Current computing power This indicates the effective or actual computing power of the computing power card.

[0049] By integrating the above attribute information, supported operator set, current computing power, and execution time, the capability indicators of each computing power card are obtained.

[0050] The above process is as follows Figure 4 As shown, the process includes: initializing the computing power card; modeling the computing power card's capabilities; evaluating the actual computing power of the computing power card based on the same operators supported by the computing power card; updating the computing power card's capabilities model; and modeling the communication between different computing power card components.

[0051] In this embodiment, by constructing a multi-dimensional capability vector model for the computing power card, combining operator characteristics to evaluate effective computing power and achieve computing power subdivision, the actual processing capability of the computing power card can be accurately characterized, adapting to the differences in execution of different operators, providing accurate data support for computing power fusion, communication modeling and scheduling, and improving the accuracy of computing power utilization and fusion.

[0052] As an optional embodiment, based on attribute information, the current computing power of the computing card and the execution time of operators in the supported operator set on the computing card are determined, including: Based on the attribute information, the computing power threshold, memory bandwidth, and computing power utilization of the computing power card are obtained; Determine the computational power and bandwidth requirements for the operators in the supported operator set; The first data item is obtained based on the computing power utilization rate and computing power threshold; Based on the computing power requirements and the first data item, the second data item is obtained; The third data item is obtained based on bandwidth requirements and video memory bandwidth; The execution time is obtained based on the second and third data items; The ratio of preset parameters to execution time is used as the current computing power.

[0053] Specifically, the attribute information determines the computing power threshold, memory bandwidth, and computing power utilization of the computing card. For example, the computing power threshold is the peak computing power of the computing card. ,Sure computing power threshold Memory bandwidth and computing power utilization .

[0054] Determine the computational power and bandwidth requirements of the operators in the set of operators that support it. For example, the computational power requirement of operator i is... The bandwidth requirement of operator i .

[0055] Based on the computing power utilization rate and computing power threshold, the first data item is obtained, for example: First data item = .

[0056] Based on the computing power requirement and the first data item, the second data item is obtained, for example: Second data item = .

[0057] Based on bandwidth requirements and video memory bandwidth, the third data item is obtained, for example: Third data item = .

[0058] Based on the second and third data items, the execution time is obtained, and operator i is in the computing power card. The execution time is shown in formula (1).

[0059] (1) The default parameters are hyperparameters. , The value is determined based on experiments or actual needs, for example: 1, 2, or other values. The ratio of the preset parameters to the execution time is used as the current computing power, for example: computing power card. Effective computing power relative to operator i As shown in formula (2).

[0060] (2) Update the capability vector of the computing power card. For operator i, the computing power card... The capability vector can be represented as: .

[0061] In this embodiment, by combining the peak computing power of the computing card, the memory bandwidth and the operator requirements, the operator execution time is accurately calculated and the effective computing power is obtained. The capability vector is updated, which can quantify the actual computing power at the operator level, take into account both computing power and bandwidth bottlenecks, improve the accuracy of computing power evaluation of heterogeneous computing cards, and adapt to the differentiated execution characteristics of operators.

[0062] As an optional embodiment, the target computing power card corresponding to the operator module is determined based on capability indicators, including: Based on the capability indicators, the set of supported operators and the set of supported data formats for the computing power card are obtained; Determine the operator data format for the operators in the operator module; The operators in the operator module are matched with the set of supported operators to obtain the first matching result, and the operator data format is matched with the set of supported data formats to obtain the second matching result; Based on the first and second matching results, candidate computing cards are determined from the computing cards; Based on the performance parameters of the candidate computing cards, the target computing card is determined from the candidate computing cards. The performance parameters are obtained based on the capability indicators.

[0063] Specifically, based on the capability indicators, the set of supported operators and the set of supported data formats for computing power cards are obtained, for example: computing power cards. Supported operator set is Computing power card Supported data format set is .

[0064] Determine the operator data format for the operators in the operator module, for example: the operator data format is... , This specifies the computational data format required by the operators in operator module m. For example, the operators in operator module m are... .

[0065] For all computing cards in the system Perform capability matching to calculate the number of supported operator modules. Match the operators in the operator modules with the set of supported operators to obtain the first matching result. Based on the first matching result, the operator can be determined. Is it included in the set of supporting operators? In the middle, it is to determine whether the condition is satisfied. The operator data format is matched against the set of supported data formats to obtain a second matching result. The operator data format can be determined based on the second matching result. Is it included in the set of supported data formats? In the middle, it is to determine whether the condition is satisfied. ".

[0066] Based on the first and second matching results, candidate computing cards are determined from the computing cards. For example, as shown in formula (3).

[0067] (3) Additionally, another filtering condition can be added, such as: specifying the operator. Required storage space Determine the video memory capacity of the computing card. The video memory capacity of the candidate computing card needs to be greater than the storage space required by the operator. .

[0068] Determine the peak computing power, computational efficiency, and energy consumption of the candidate computing cards from their performance parameters. Based on these parameters, determine the target computing card from among the candidate computing cards. For example, determine the computing card whose peak computing power and computational efficiency both meet the requirements of the corresponding computing module from among the candidate computing cards, and then select the computing card with the lowest power consumption as the target computing card to reduce the overall energy consumption of the system.

[0069] In this embodiment, candidate computing cards are screened by multiple matching of operators, data formats and video memory capacity, and then the target card is selected by combining computing power, computing efficiency and energy consumption. This ensures the feasibility of operator module deployment, reduces system energy consumption, and improves the accuracy and rationality of computing power allocation.

[0070] As an optional embodiment, the performance parameters include the current computing power, the computing efficiency of the candidate computing cards, and their energy consumption. Based on the performance parameters of the candidate computing cards, the target computing card is determined from the candidate computing cards, including: Based on the first matching result, it is determined whether there is a first intermediate computing card among the candidate computing cards, wherein the set of supported operators of the first intermediate computing card includes the operators in the operator module, and the current computing power of the first intermediate computing card is greater than the first preset threshold. In the presence of a first intermediate computing card, a second intermediate computing card is determined from the first intermediate computing card, wherein the computing efficiency of the second intermediate computing card is greater than a second preset threshold. In the second intermediate computing card, the one with the lowest energy consumption is selected as the target computing card.

[0071] Specifically, this embodiment implements a central node-less scheduling approach, without setting a central scheduling node. Idle computing cards are used as scheduling nodes to perform the entire system scheduling. The computing card with the fewest supported operators and the largest computing power is selected as the scheduling center; the scheduling center is randomly selected based on the different models; to avoid failures during the scheduling center's operation, two computing cards, a primary and a secondary one, are configured for the scheduling center. The allocation of computing cards is handled through the scheduling center.

[0072] Performance parameters include current computing power, computational efficiency of candidate computing cards, and energy consumption. The current computing power refers to the peak computing capacity of the computing card. The computational efficiency of candidate computing cards can be calculated after the model to be deployed is started, based on the time from data input to data output. The energy consumption of candidate computing cards can be determined by statistically analyzing their power consumption and runtime. Since power fluctuates instantaneously, power value graphs at different times can be generated, and the area of ​​the time-power graph can be calculated to obtain the energy consumption value.

[0073] In the above embodiments, the effective computing power of the candidate computing cards, i.e., the current computing power, is calculated. A first preset threshold for computing power is set to determine whether to allocate the model to different computing cards. If the computing power is very large, there is no need to allocate to multiple cards, as the computing efficiency may already be very high. If the effective computing power is very small, then computing power allocation is required to improve computing efficiency. This is a balance between computing power calculation time and data transmission time. This first preset threshold is also a hyperparameter, which can be set according to experimental values ​​or actual needs, such as 128 FLOPs / s, 256 FLOPs / s, etc.

[0074] Based on the first matching result, it is determined whether there is a first intermediate computing card among the candidate computing cards. For example, computing card 3 is a candidate computing card, and the operators in the operator module include operator 3 and operator 4. The operator set supported by computing card 3 also includes operator 3 and operator 4. The first preset threshold is 128 FLOPs / s, and the current computing power of computing card 3 is 256 FLOPs / s. Therefore, computing card 3 can be used as the first intermediate computing card.

[0075] In the presence of a first intermediate computing card, a second intermediate computing card with a computing efficiency greater than a second preset threshold is selected from among the first intermediate computing cards. The second preset threshold may be, for example, 80%, 90%, or other values. Among the second intermediate computing cards, the one with the lowest energy consumption is selected as the target computing card.

[0076] Alternatively, the process for determining the target computing card from the candidate computing cards can be as follows: First, determine if the operating efficiency of the candidate computing card is greater than a second preset threshold. If it is, continue; otherwise, the computing card cannot meet the requirements of the model to be deployed. Then, among the candidate computing cards, determine if the operating efficiency of the candidate computing card with the second-highest effective computing power is greater than the second preset threshold. If it is, calculate the energy consumption of that card; otherwise, directly select the previous computing card. Calculate the efficiency of all individual cards that meet the threshold and calculate their energy consumption. If the operating efficiency is greater than the second preset threshold, select the computing card with the lowest power consumption to reduce the overall system energy consumption.

[0077] In this embodiment of the application, the solution determines the target computing power card through three levels of screening: computing power threshold, computing efficiency, and energy consumption. It takes into account both computing power requirements and execution efficiency, prioritizes the selection of low-energy-consumption cards, balances computing power allocation and energy consumption, reduces the overall energy consumption of the system, and ensures the efficient and stable operation of the model.

[0078] As an optional embodiment, after determining whether a first intermediate computing card exists among the candidate computing cards, the method further includes: In the absence of a first intermediate computing card, the candidate computing cards are sorted according to the number of operators in the set of supported operators of the candidate computing cards to obtain a candidate computing card queue. Based on the candidate computing power card queue and the current computing power of the candidate computing power cards, multiple first computing power card combinations are generated. Each first computing power card combination contains multiple candidate computing power cards. The total set of supported operators of the first computing power card combination includes the operators in the operator module. Furthermore, the sum of the current computing power of the candidate computing power cards in the first computing power card combination is greater than a first preset threshold. Based on the first computing power card combination and operator module, determine the total running time of the model to be deployed; Based on the total running time, a second computing power card combination is determined from the first computing power card combination, wherein the total running time corresponding to the second computing power card combination is less than a third preset threshold. Determine the overall computing efficiency and overall energy consumption of the candidate computing cards in the second computing card combination, and determine the first target computing card combination in the second computing card combination based on the overall computing efficiency and overall energy consumption; The candidate computing cards in the first target computing card combination are used as target computing cards.

[0079] Specifically, if the first intermediate computing card is not among the candidate computing cards, the operator model needs to be deployed on the computing card combination, and the multiple computing cards in the computing card combination can be used to meet the computing power requirements of the operator model.

[0080] The candidate computing cards are sorted according to the number of operators in the set of supported operators of the candidate computing cards, resulting in a candidate computing card queue.

[0081] Based on the candidate computing power card queue and the current computing power of the candidate computing power cards, multiple first computing power card combinations are generated. For example, in the candidate computing power card queue, computing power cards that support the same number of operators are identified, and the computing power card with the largest effective computing power is selected from them. The computing power card with the largest effective computing power is added to the computing power card combination. It is determined whether the computing power card combination supports all operators in the operator module. If it does, the computing power card combination is output as the first computing power card combination. If some operators are not supported by the computing power card combination, the computing power cards are re-sorted according to the missing operators, and then the computing power card with the largest effective computing power is selected and added to the computing power card combination until all operators are supported. The computing power card combination is then output as the first computing power card combination.

[0082] Based on the first computing power card combination and operator module, determine the total running time of the model to be deployed. For example, the total running time is the total running time of a single run of the model to be deployed. , As shown in formula (4).

[0083] (4) in, Any computing card in the first computing card combination for Between execution, For computing power cards and computing power card Communication duration between them For the division of operator modules, This represents the total number of operator modules. A collection of computing power cards used for all models.

[0084] The third preset threshold is set according to actual needs, such as 1 minute, 5 minutes, or other durations. In the first computing card combination, the first computing card combination whose total running time is less than the third preset threshold is selected as the second computing card combination.

[0085] Determine the overall computing efficiency and overall energy consumption of the candidate computing cards in the second computing card combination, and based on the overall computing efficiency and overall energy consumption, determine the first target computing card combination in the second computing card combination. For example, if the overall computing efficiency meets the user's needs, such as exceeding the threshold set by the user, select the second computing card combination with the lowest overall energy consumption as the first target computing card combination to reduce the overall energy consumption of the system.

[0086] The candidate computing cards in the first target computing card combination are used as target computing cards.

[0087] In this embodiment, when a single card does not meet the requirements, a combination of computing power cards is constructed. The selection is completed by operator coverage and total computing power. The qualified combination is determined by combining execution and communication time. Then, the optimal combination is selected based on computing efficiency and energy consumption to meet the computing power requirements of the model, take into account runtime efficiency and energy consumption, and improve the adaptability of heterogeneous computing power scheduling.

[0088] As an optional embodiment, performance parameters include execution time, communication duration between candidate computing cards, and power of candidate computing cards. Based on the performance parameters of the candidate computing cards, the target computing card is determined from the candidate computing cards, including: Based on the candidate computing power cards, generate multiple combinations of second computing power cards; Obtain the target computing power card selection function; The execution time, communication duration, and power of the candidate computing cards in the second computing card combination are input into the target computing card selection function to obtain the evaluation index corresponding to the second computing card combination. The second computing power card combination with the lowest evaluation index is selected as the second target computing power card combination, and the candidate computing power cards in the second target computing power card combination are selected as target computing power cards.

[0089] Specifically, based on the candidate computing power cards, multiple second computing power card combinations are generated. For example, there are 5 operator modules, namely operator modules 5-9, which contain operators 5-15. Candidate computing power cards that support operators 5-15 are selected to form a second computing power card combination. That is, the computing power cards in the second computing power card combination can support the operators in all operator models.

[0090] Obtain the target computing power card selection function, for example: formula (5).

[0091] (5) in, , , This is a hyperparameter, measured experimentally. This refers to the power consumption loss of the computing card a and the operator M. This is an evaluation index.

[0092] The execution time, communication duration, and power consumption of the candidate computing cards in the second computing card combination are input into the target computing card selection function to obtain the evaluation index corresponding to the second computing card combination. Evaluation index The smallest second computing power card combination is taken as the second target computing power card combination, and the candidate computing power cards in the second target computing power card combination are taken as the target computing power cards.

[0093] In this embodiment of the application, by constructing a combination of computing power cards, the evaluation index of execution time, communication duration and power consumption loss is comprehensively calculated using the objective selection function to screen the optimal combination, thereby achieving multi-objective optimization of computing power scheduling, taking into account runtime efficiency, communication overhead and energy consumption, and improving the rationality and overall performance of heterogeneous computing power scheduling.

[0094] As an optional embodiment, obtaining the target computing card selection function includes: Based on the execution time and the operators in the operator module, the total execution time of the second computing power card combination is obtained, and based on the communication duration, the total communication duration of the second computing power card combination is obtained. Based on the total execution time and total communication time, the total runtime of the model to be deployed is obtained; The total power consumption loss of the second computing card combination is obtained based on the power, execution time, and operators in the operator module of the candidate computing cards. The target computing card selection function is obtained based on the total communication time, total runtime, total power consumption loss, and preset coefficients.

[0095] Specifically, the total execution time of the second computing power card combination is obtained based on the execution time and the operators in the operator module. For example, the total execution time is... Based on the communication duration, the total communication duration of the second computing card combination is obtained. For example, the total communication duration is... The calculation method for communication duration between different computing cards is as shown in formula (6).

[0096] (6) in, The data that needs to be transferred between computing power card a and computing power card b; The effective communication bandwidth between computing power card a and computing power card b; Let be the communication delay constant between computing power card a and computing power card b. Based on the preceding computing power card allocation and model input, the data flow direction is determined. This data flow direction determines the data transmission between the computing power cards, and the transmission data is then used to determine... , , .

[0097] Based on total execution time Total communication duration The total runtime of the model to be deployed is obtained. .

[0098] First, based on the power of the candidate computing cards and execution time The power consumption loss of each computing card in the second computing card combination was obtained. For example, as shown in formula (7).

[0099] (7) By combining the operators in the operator module, the total power consumption loss of the second computing card combination is obtained. .

[0100] The preset coefficients include the hyperparameters mentioned above. , , .

[0101] Combining the total communication time, total runtime, total power consumption loss and preset coefficients, the target computing card selection function is obtained. The target computing card selection function is, for example, the above formula (5).

[0102] In this embodiment, the total execution time, total communication time and total power consumption loss of the computing power card combination are accurately calculated, a multi-objective comprehensive selection function is constructed, and the scheduling overhead is quantitatively evaluated. This provides a reliable basis for selecting the optimal computing power combination, effectively balances runtime latency, communication overhead and energy consumption, and improves the scheduling accuracy of heterogeneous computing power.

[0103] As an optional implementation, another computing power allocation process is provided, such as Figure 5 As shown, the process includes: importing the model to be run; splitting the model by operators; calculating the number of computing cards that support the operators; sorting the cards according to the number of supported operators and the effective computing power; determining whether a single card can support the model independently; if not, then executing "sorting computing cards by the number of supported operators"; if yes, then selecting the computing card with the largest effective computing power among the single cards; determining whether the effective computing power of the accelerator card with the largest computing power is greater than a set threshold among all accelerator cards; if not, then executing "sorting computing cards by the number of supported operators"; if yes, then determining the computing card to be used. The process involves: sorting the computing cards by the number of supported operators; selecting the graphics card with the highest effective computing power; determining if all operators are supported, and if not, confirming that the computing power pool does not support the model; if so, selecting a computing card combination based on communication and computing power losses; calculating the total runtime T of the model in a single run; calculating the energy consumption loss of the computing card combination; determining if the selected computing cards meet the user's needs; selecting computing card combinations with lower computing power until all computing card combinations have been traversed; and finally, selecting the computing card combination with the lowest energy consumption loss.

[0104] In this embodiment, effective computing power is used to evaluate the actual capabilities of the computing cards. Different computing cards can be effectively ranked for different models, allowing for the selection of more suitable cards and improving system efficiency. This enables the deployment of a single model across computing cards with different architectures, achieving unified modeling and fusion scheduling. A globally optimal balance is achieved between computing, communication, and energy consumption, minimizing system power consumption while meeting user needs.

[0105] As an optional embodiment, determining candidate computing cards among computing cards based on the first matching result and the second matching result includes: determining a first type of computing card among computing cards based on the first matching result, wherein the supported operator set of the first type of computing card includes operators in the operator module; determining a second type of computing card among computing cards based on the second matching result, wherein the supported data format set of the second type of computing card includes operator data formats; obtaining a third intermediate computing card based on the first type of computing card and the second type of computing card, wherein the third intermediate computing card is a first type of computing card and is also a second type of computing card; determining the storage capacity requirements of the operators included in the operator module and determining the video memory capacity of the third intermediate computing card; if the video memory capacity is greater than the storage capacity requirement, the third intermediate computing card is selected as a candidate computing card.

[0106] Specifically, the set of supported operators for the first type of computing power card includes the operators in the operator module. For example, if the operator module includes operators 5, 6, and 7, and the set of supported operators for computing power card a includes operator 5, then computing power card a is a first-type computing power card. If the set of supported operators for computing power card b does not include operators 5-7, then computing power card b is not a first-type computing power card. Based on the first matching result, it can be determined whether the operators in the operator module are included in the set of supported operators for each computing power card. Based on the determination result, the first-type computing power card is identified among the computing power cards.

[0107] The second type of computing power card supports a set of data formats that include operator data formats. For example, if the operator data format is format A, and the set of supported data formats for computing power card a includes format A, then computing power card a is a second type of computing power card. Based on the second matching result, it can be determined whether the operator data format is included in the set of supported data formats. Based on the determination result, the second type of computing power card is identified among the computing power cards.

[0108] The third intermediate computing card is both a first-class computing card and a second-class computing card. That is, the third intermediate computing card supports both the operators in the operator module and the operator data format. For example, if the set of operators supported by computing card a includes operator 5, computing card a is a first-class computing card. If the set of data formats supported by computing card a includes format A, computing card a is a second-class computing card. Therefore, computing card a is a third-class computing card.

[0109] Additionally, a further filtering criterion for computing cards can be added, such as: determining the storage space required by the operator, determining the video memory capacity of the computing card, and ensuring that the video memory capacity of the candidate computing cards is greater than the storage space required by the operator. The storage capacity requirements of the operators included in the operator module are determined, and the video memory capacity of the third intermediate computing card is also determined. If the video memory capacity is greater than the storage capacity requirements, the third intermediate computing card is selected as a candidate computing card.

[0110] As an optional embodiment, based on the candidate computing cards, multiple combinations of second computing cards are generated, including: Determine the total set of operators contained in multiple operator modules; A fourth intermediate computing card is determined from the candidate computing cards. The set of supported operators of the fourth intermediate computing card includes the operators to be matched, and the operators to be matched are included in the total set of operators. The fourth intermediate computing power card is filtered according to the matching operator to obtain the filtered computing power card; Based on the selected computing power cards, a second computing power card combination is generated, wherein the total set of supported operators of the second computing power card combination includes the total set of operators.

[0111] Specifically, determine the total set of operators contained in multiple operator modules. For example, if there are 5 operator modules, namely operator modules 5-9; operator module 5 contains operators 5-8, operator module 6 contains operators 9-10, operator module 7 contains operator 11, operator module 8 contains operators 12-14, and operator module 9 contains operator 15, then the total set of operators is operator 5-15.

[0112] Each operator in the total set of operators is regarded as an operator to be matched. Among all candidate computing power cards, the candidate computing power card that supports at least one operator to be matched in the set of operators is selected as the fourth intermediate computing power card. That is, the fourth intermediate computing power card supports at least one operator to be matched.

[0113] Based on the candidate computing power cards, multiple combinations of second computing power cards are generated. For example, if there are 5 operator modules, namely operator modules 5-9, and these 5 operator modules contain operators 5-15, then candidate computing power cards that support operators 5-15 are selected to form the second computing power card combinations. The fourth intermediate computing power card is filtered based on the operators to be matched. For example, computing power card d supports operator 5, and computing power card e supports operators 5, 6, and 7. In this case, computing power card e can support operator 5, so computing power card d can be filtered out. Computing power card f supports operator 15, and computing power card g supports operator 15. Since they are the same, the computing power card with higher power consumption between f and g can be filtered out to save subsequent power consumption. After filtering, the filtered computing power cards are obtained.

[0114] By combining the selected computing power cards, a second computing power card combination is generated. The total set of supported operators in the second computing power card combination includes the total set of operators, meaning that the computing power cards in the second computing power card combination can support operators in all operator models.

[0115] As an optional embodiment, multiple first computing card combinations are generated based on the candidate computing card queue and the current computing power of the candidate computing cards, including: Create an initial computing card combination based on the first candidate computing card in the candidate computing card queue; The first missing operator of the initial computing power card combination is determined, wherein the first missing operator is included in the operator module and is not included in the total set of supporting operators of the initial computing power card combination; Based on the first missing operator, adjust the candidate computing power card queue to obtain the updated queue; Add the first candidate computing power card in the updated queue to the initial computing power card combination to obtain the intermediate computing power card combination; If the intermediate computing card combination does not contain the second missing operator, the intermediate computing card combination is regarded as the first computing card combination, wherein the second missing operator is included in the operator module and the second missing operator is not included in the total set of supported operators of the intermediate computing card combination. If the intermediate computing card combination contains the second missing operator, adjust the updated queue according to the second missing operator, use the adjusted updated queue as the updated queue, and start the subsequent steps from adding the first candidate computing card in the updated queue to the initial computing card combination, until the intermediate computing card combination does not contain the second missing operator, then end.

[0116] Specifically, an initial computing card combination is created based on the first candidate computing card in the candidate computing card queue. At this time, the candidate computing card queue is arranged in descending order of the number of supported operators. That is, the computing card with the most supported operators is added to a certain computing card combination as the initial computing card combination.

[0117] The first missing operator is the operator among all operators contained in the operator module that cannot be supported by the initial computing power card combination. Determine the first missing operator of the initial computing power card combination.

[0118] Based on the first missing operator, the candidate computing power card queue is adjusted to obtain an updated queue. For example, computing power cards that can support the first missing operator are selected from the candidate queue, and then sorted in descending order of the number of operators supported, resulting in the updated queue. Furthermore, during resorting, if computing power cards with the same number of supported operators exist, the computing power card with the highest effective computing power is selected and placed at the front of the queue; this facilitates the subsequent addition of the computing power card with the highest effective computing power to the computing power card combination.

[0119] Add the first candidate computing card in the updated queue to the initial computing card combination to obtain the intermediate computing card combination.

[0120] The second missing operator is the operator that cannot be supported by the intermediate computing card combination among all operators included in the operator modules. If the intermediate computing card combination does not contain the second missing operator, it is determined that the intermediate computing card combination supports all operators included in the operator modules. The intermediate computing card combination is then designated as the first computing card combination.

[0121] If the intermediate computing card combination contains the second missing operator, adjust the updated queue according to the second missing operator, and use the adjusted updated queue as the new updated queue. For example, select computing cards that can support the second missing operator from the updated queue, sort the selected computing cards in descending order of the number of operators they support, and use the adjusted updated queue as the new updated queue.

[0122] Repeat the above steps: "Add the first candidate computing card in the updated queue to the initial computing card combination to obtain the intermediate computing card combination" until the intermediate computing card combination does not contain the second missing operator, that is, until all operators are supported by the first computing card combination, then end.

[0123] As an optional embodiment, in the above step "determine candidate computing cards in the computing cards according to the first matching result and the second matching result", the computing cards are screened according to the operator data format. Since the data format is inconsistent, a large number of idle computing cards are easily eliminated, causing the idle computing cards to remain idle and wasting computing resources.

[0124] In this embodiment, a lightweight dynamic adaptation layer is added, which enables data conversion between various operator data formats. This allows computing cards with incompatible operator data formats to indirectly execute operator modules through low-overhead adaptation, significantly improving the computing power utilization of heterogeneous computing clusters. At the same time, the adaptation overhead is incorporated into joint optimization, balancing adaptation feasibility and execution efficiency.

[0125] Based on the lightweight dynamic adaptation layer, the specific process of the above step "determining candidate computing cards in computing cards" may also include steps A1 to A5.

[0126] Step A1: Determine the compatibility of the computing card with the operator module.

[0127] Specifically, first check whether the set of supported operators and data formats of the computing power card matches the operator and format requirements of the operator module, and determine whether it is compatible or incompatible. If incompatible, call the lightweight dynamic adaptation layer and start the dynamic adaptation process.

[0128] Step A2: From the preset operator mapping library, find an equivalent combinatorial operator that has the same function as the incompatible operator, replace the original operator, and ensure that the calculation logic and result remain unchanged.

[0129] Step A3 utilizes a lightweight dynamic adaptation layer to adapt and convert the data format in real time.

[0130] Specifically, the input and output data of the operator module are converted in real time to a format supported by the computing card (such as precision and encoding type), thus eliminating format incompatibility issues.

[0131] In step A4, between the two operations of re-determining the computing power card and using the lightweight dynamic adaptation layer for data format conversion, select the target operation and continue to execute the target operation.

[0132] Specifically, a comparative evaluation of the costs of adaptation and card replacement is conducted. The total costs of operator replacement and format conversion, as well as the execution and communication costs of replacing with other computing cards, are calculated separately to determine which operation is more cost-effective and efficient, and this is taken as the target operation.

[0133] Step A5: After determining the target operation, perform a secondary verification of the compatibility between the computing card and the operator. If the verification passes, the current computing card is selected as a candidate computing card.

[0134] Specifically, after completing the operator replacement or format conversion, re-check whether the adapted operators and data formats meet the requirements of the computing power card, confirm that it can be executed normally, and use the current computing power card as a candidate computing power card.

[0135] In this embodiment, the incompatibility problem of computing cards is solved by compatibility judgment, equivalent replacement of operators and data format conversion. Combined with overhead evaluation and optimal adaptation, the adaptation overhead is included in the optimization calculation, which can greatly improve the utilization rate of heterogeneous computing power, ensure the accuracy of calculation and execution efficiency, and enhance the versatility of scheduling.

[0136] This embodiment also provides a model deployment apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0137] This embodiment provides a model deployment device, such as Figure 6 As shown, it includes: The model splitting module 601 is used to split the model to be deployed into multiple operator modules, wherein the operator module contains at least one operator, and the operator is contained in the model to be deployed. The indicator acquisition module 602 is used to acquire the capability indicators of multiple computing cards, wherein the capability indicators are used to determine the operators supported by the computing cards; The computing power card determination module 603 is used to determine the target computing power card corresponding to the operator module based on the capability indicators, wherein the target computing power card supports the operators contained in the operator module; Deployment module 604 is used to deploy the operator module to the target computing card.

[0138] As an optional embodiment, the model splitting module 601 includes: The first determining unit is used to determine the operators contained in the model to be deployed and the dependency values ​​between the operators; The fusion unit is used to merge the operators corresponding to the dependency values ​​into an operator module when the dependency value is greater than a preset threshold. The setting unit is used to set the operators corresponding to the dependency values ​​as operator modules when the dependency values ​​are less than or equal to a preset threshold.

[0139] As an optional embodiment, the indicator acquisition module 602 includes: The acquisition unit is used to acquire the attribute information and supported operator set of the computing power card; The second determining unit is used to determine the current computing power of the computing power card and the execution time of the operators in the set of supported operators on the computing power card based on the attribute information. The first obtaining unit is used to obtain capability indicators based on attribute information, the set of supported operators, current computing power, and execution time.

[0140] As an optional embodiment, the second determining unit includes: The first submodule is used to obtain the computing power threshold, memory bandwidth and computing power utilization of the computing power card based on the attribute information. The first determination submodule is used to determine the computing power and bandwidth requirements of operators in the set of supported operators; The second submodule is used to obtain the first data item based on the computing power utilization rate and the computing power threshold; The third submodule is used to obtain the second data item based on the computing power requirements and the first data item; The fourth submodule is used to obtain the third data item based on the bandwidth requirements and video memory bandwidth. The first setting submodule is used to obtain the execution time based on the second and third data items; The second setting submodule is used to set the ratio of preset parameters to execution time as the current computing power.

[0141] As an optional embodiment, the computing power card determination module 603 includes: The second obtaining unit is used to obtain the set of supported operator sets and the set of supported data formats of the computing power card based on the capability indicators; The third determining unit is used to determine the operator data format of the operators in the operator module; The third obtaining unit is used to match the operators in the operator module with the set of supporting operators to obtain the first matching result, and to match the operator data format with the set of supporting data formats to obtain the second matching result; The fourth determining unit is used to determine candidate computing cards from among the computing cards based on the first matching result and the second matching result; The fifth determining unit is used to determine the target computing card from the candidate computing cards based on the performance parameters of the candidate computing cards, wherein the performance parameters are obtained based on the capability indicators.

[0142] As an optional embodiment, the performance parameters include the current computing power, the computing efficiency of the candidate computing cards, and energy consumption. The fifth determining unit includes: The judgment submodule is used to determine whether there is a first intermediate computing power card among the candidate computing power cards based on the first matching result. The first intermediate computing power card's support operator set includes the operators in the operator module, and the current computing power of the first intermediate computing power card is greater than a first preset threshold. The second determining submodule is used to determine the second intermediate computing card from the first intermediate computing card when the first intermediate computing card exists, wherein the computing efficiency of the second intermediate computing card is greater than the second preset threshold. The third setting submodule is used to select the second intermediate computing card with the lowest energy consumption as the target computing card in the second intermediate computing card.

[0143] As an optional embodiment, the fifth determining unit further includes: The sorting submodule is used to sort the candidate computing cards according to the number of operators in the set of supported operators of the candidate computing cards when there is no first intermediate computing card, and obtain a candidate computing card queue. The first generation submodule is used to generate multiple first computing card combinations based on the candidate computing card queue and the current computing power of the candidate computing card. The first computing card combination contains multiple candidate computing cards, the total set of supported operators of the first computing card combination contains the operators in the operator module, and the sum of the current computing power of the candidate computing cards in the first computing card combination is greater than a first preset threshold. The third determining submodule is used to determine the total running time of the model to be deployed based on the first computing power card combination and the operator submodule; The fourth determining submodule is used to determine the second computing card combination from the first computing card combination based on the total running time, wherein the total running time corresponding to the second computing card combination is less than the third preset threshold. The fifth determining submodule is used to determine the overall computing efficiency and overall energy consumption of the candidate computing cards in the second computing card combination, and to determine the first target computing card combination in the second computing card combination based on the overall computing efficiency and overall energy consumption. The fourth setting submodule is used to select candidate computing cards from the first target computing card combination as target computing cards.

[0144] As an optional embodiment, the performance parameters include execution time, communication duration between candidate computing cards, and power of candidate computing cards. The fifth determining unit further includes: The second generation submodule is used to generate multiple combinations of second computing cards based on the candidate computing cards; The acquisition submodule is used to obtain the target computing power card selection function; The input submodule is used to input the execution time, communication duration, and power of the candidate computing cards in the second computing card combination into the target computing card selection function to obtain the evaluation index corresponding to the second computing card combination. The fifth setting submodule is used to select the second computing power card combination with the smallest evaluation index as the second target computing power card combination, and to select the candidate computing power cards in the second target computing power card combination as the target computing power cards.

[0145] As an optional embodiment, the acquisition submodule includes: The first subunit is used to obtain the total execution time of the second computing card combination based on the execution time and the operators in the operator module, and to obtain the total communication time of the second computing card combination based on the communication duration. The second sub-unit is used to obtain the total runtime of the model to be deployed based on the total execution time and total communication duration. The third subunit is used to obtain the total power consumption loss of the second computing card combination based on the power, execution time and operators in the operator module of the candidate computing cards. The fourth sub-unit is used to obtain the target computing card selection function based on the total communication duration, total runtime, total power consumption loss, and preset coefficients.

[0146] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0147] In this embodiment, the model deployment device is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0148] Figure 7This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0149] The following is a detailed reference. Figure 7 This diagram illustrates a suitable structural schematic for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 701, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 702 or a program loaded from memory 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device. The processor 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0150] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 7 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0151] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a memory 708, or installed from a ROM 702. When the computer program is executed by the processor 701, it performs the functions defined in the model deployment method of the embodiments of the present invention.

[0152] Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0153] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described model deployment method embodiments at runtime.

[0154] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0155] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described model deployment method embodiments.

[0156] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model deployment method embodiments.

[0157] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0158] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0159] The foregoing has provided a detailed description of a model deployment method, electronic device, storage medium, and program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A model deployment method, characterized in that, The method includes: The model to be deployed is divided into multiple operator modules, wherein each operator module contains at least one operator, and the operator is contained in the model to be deployed; Obtain capability metrics for multiple computing power cards, wherein the capability metrics are used to determine the operators supported by the computing power cards; The process of obtaining capability metrics for multiple computing power cards includes: obtaining attribute information and a set of supported operators for the computing power cards; determining the current computing power of the computing power cards and the execution time of the operators in the set of supported operators on the computing power cards based on the attribute information; and obtaining the capability metrics based on the attribute information, the set of supported operators, the current computing power, and the execution time. Based on the capability indicators, the target computing power card corresponding to the operator module is determined, wherein the target computing power card supports the operators included in the operator module; The step of determining the target computing card corresponding to the operator module based on the capability indicators includes: obtaining the set of supported operators and the set of supported data formats of the computing card based on the capability indicators; determining the operator data format of the operators in the operator module; matching the operators in the operator module with the set of supported operators to obtain a first matching result, and matching the operator data format with the set of supported data formats to obtain a second matching result; determining candidate computing cards in the computing card set based on the first matching result and the second matching result; and determining the target computing card in the candidate computing card set based on the performance parameters of the candidate computing card set, wherein the performance parameters are obtained based on the capability indicators. The performance parameters include the execution time, the communication duration between candidate computing cards, and the power of the candidate computing cards. Determining the target computing card from the candidate computing cards based on the performance parameters of the candidate computing cards includes: generating multiple second computing card combinations based on the candidate computing cards; obtaining a target computing card selection function; inputting the execution time, communication duration, and power of the candidate computing cards in the second computing card combinations into the target computing card selection function to obtain an evaluation index corresponding to the second computing card combinations; selecting the second computing card combination with the smallest evaluation index as the second target computing card combination, and selecting the candidate computing cards in the second target computing card combination as the target computing card. The operator module is deployed to the target computing card.

2. The method according to claim 1, characterized in that, The step of splitting the model to be deployed into multiple operator modules includes: Determine the operators contained in the model to be deployed and the dependency values ​​between the operators; If the dependency value is greater than a preset threshold, the operators corresponding to the dependency value are merged into the operator module; When the dependency value is less than or equal to a preset threshold, the operators corresponding to the dependency value are respectively used as the operator modules.

3. The method according to claim 1, characterized in that, The step of determining the current computing power of the computing card and the execution time of the operators in the set of supported operators on the computing card based on the attribute information includes: Based on the attribute information, the computing power threshold, memory bandwidth, and computing power utilization of the computing power card are obtained. Determine the computational power and bandwidth requirements of the operators in the set of supported operators; The first data item is obtained based on the computing power utilization rate and the computing power threshold; Based on the computing power requirement and the first data item, the second data item is obtained; Based on the bandwidth requirements and the video memory bandwidth, the third data item is obtained; The execution time is obtained based on the second data item and the third data item; The ratio of the preset parameters to the execution time is used as the current computing power.

4. The method according to claim 1, characterized in that, The performance parameters include the current computing power, the computational efficiency of the candidate computing cards, and energy consumption. The step of determining the target computing card from the candidate computing cards based on the performance parameters of the candidate computing cards includes: Based on the first matching result, it is determined whether there is a first intermediate computing card among the candidate computing cards, wherein the set of supported operators of the first intermediate computing card includes the operators in the operator module, and the current computing power of the first intermediate computing card is greater than a first preset threshold. In the presence of the first intermediate computing card, a second intermediate computing card is determined from the first intermediate computing card, wherein the computing efficiency of the second intermediate computing card is greater than a second preset threshold. In the second intermediate computing card, the second intermediate computing card with the lowest energy consumption is selected as the target computing card.

5. The method according to claim 4, characterized in that, After determining whether a first intermediate computing card exists among the candidate computing cards, the method further includes: In the absence of the first intermediate computing card, the candidate computing cards are sorted according to the number of operators in the set of supported operators of the candidate computing cards to obtain a candidate computing card queue. Based on the candidate computing power card queue and the current computing power of the candidate computing power cards, multiple first computing power card combinations are generated. Each first computing power card combination contains multiple candidate computing power cards. The total set of supported operators of the first computing power card combination includes the operators in the operator module. Furthermore, the sum of the current computing power of the candidate computing power cards in the first computing power card combination is greater than the first preset threshold. Based on the first computing card combination and the operator module, determine the total running time of the model to be deployed; Based on the total running time, an intermediate first computing card combination is determined in the first computing card combination, wherein the total running time corresponding to the intermediate first computing card combination is less than a third preset threshold. Determine the overall computing efficiency and overall energy consumption of the candidate computing cards in the intermediate first computing card combination, and determine the first target computing card combination in the intermediate first computing card combination based on the overall computing efficiency and the overall energy consumption; The candidate computing cards in the first target computing card combination are used as the target computing cards.

6. The method according to claim 1, characterized in that, The function for obtaining the target computing power card selection includes: Based on the execution time and the operators in the operator module, the total execution time of the second computing card combination is obtained, and based on the communication duration, the total communication duration of the second computing card combination is obtained. The total runtime of the model to be deployed is obtained based on the total execution time and the total communication duration. The total power consumption loss of the second computing card combination is obtained based on the power of the candidate computing cards, the execution time, and the operators in the operator module. The target computing card selection function is obtained based on the total communication duration, the total runtime, the total power consumption loss, and a preset coefficient.

7. The method according to claim 1, characterized in that, The step of determining candidate computing cards from the computing cards based on the first matching result and the second matching result includes: Based on the first matching result, a first type of computing card is determined in the computing card, wherein the set of supported operators of the first type of computing card includes the operators in the operator module; Based on the second matching result, a second type of computing card is determined in the computing card, wherein the set of supported data formats of the second type of computing card includes the operator data format; Based on the first type of computing power card and the second type of computing power card, a third intermediate computing power card is obtained, wherein the third intermediate computing power card is the first type of computing power card and the third intermediate computing power card is the second type of computing power card; Determine the storage capacity requirements of the operators included in the operator module, and determine the video memory capacity of the third intermediate computing card; If the video memory capacity is greater than the storage capacity requirement, the third intermediate computing card will be selected as the candidate computing card.

8. The method according to claim 1, characterized in that, The step of generating multiple combinations of second computing power cards based on the candidate computing power cards includes: Determine the total set of operators contained in multiple operator modules; A fourth intermediate computing card is determined from the candidate computing cards, wherein the set of supporting operators of the fourth intermediate computing card includes operators to be matched, and the operators to be matched are included in the total set of operators; The fourth intermediate computing power card is filtered according to the matching operator to obtain the filtered computing power card; Based on the filtered computing power cards, a second computing power card combination is generated, wherein the total set of supported operators of the second computing power card combination includes the total set of operators.

9. The method according to claim 5, characterized in that, The step of generating multiple first computing card combinations based on the candidate computing card queue and the current computing power of the candidate computing card includes: An initial computing card combination is created based on the first candidate computing card in the candidate computing card queue. Determine the first missing operator of the initial computing power card combination, wherein the first missing operator is included in the operator module and is not included in the total set of supported operators of the initial computing power card combination; Based on the first missing operator, adjust the candidate computing power card queue to obtain the updated queue; Add the first candidate computing card in the updated queue to the initial computing card combination to obtain the intermediate computing card combination; If the intermediate computing card combination does not contain the second missing operator, the intermediate computing card combination is used as the first computing card combination, wherein the second missing operator is included in the operator module and the second missing operator is not included in the total set of supported operators of the intermediate computing card combination; If the intermediate computing card combination contains the second missing operator, the updated queue is adjusted according to the second missing operator, and the adjusted updated queue is used as the updated queue. The subsequent steps are executed starting from adding the first candidate computing card in the updated queue to the initial computing card combination, until the intermediate computing card combination does not contain the second missing operator, and then the process ends.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the model deployment method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the model deployment method as described in any one of claims 1 to 9.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the model deployment method as described in any one of claims 1 to 9.

Images