Search-based deep learning model deployment method and apparatus

Through the search-based deep learning model deployment method, iteratively optimizes hardware resource allocation, the problem of inefficiency in deep learning models running in multi-hardware deployment is solved, and higher computing efficiency and hardware performance utilization is achieved.

WO2024234534A1PCT designated stage expired Publication Date: 2024-11-21ZHEJIANG LAB +1
View PDF 5 Cites -1 Cited by

Patent Information

Application Number
PCT/CN2023/123366
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-17
Filing Date
2023-10-08
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

When the deep learning model is deployed on multiple hardware, it is difficult for the prior art to fully utilize the hardware performance, resulting in inefficient model operation.

Method used

Using search-based deep learning model deployment method, we can obtain calculation graphs and determine the hardware resources matching by operators, build a search space, select target samples and iterate and optimize the allocation scheme until the preset iteration termination conditions are met, so as to achieve the optimal hardware Resource allocation.

Benefits of technology

It improves the computing efficiency of deep learning models, makes full use of the performance advantages of each hardware, and achieves higher deployment performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023123366_21112024_PF_FP_ABST
    Figure CN2023123366_21112024_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed in the present disclosure are a search-based deep learning model deployment method and apparatus. The method comprises: acquiring a computation graph corresponding to a deep learning model; determining operators included in the computation graph, and determining a hardware resource matched with each operator; then, according to the hardware resource matched with each operator, constructing a search space; selecting a target sample from the search space, determining an operation duration corresponding to the target sample, determining a neighborhood sample corresponding to the target sample, determining an operation duration corresponding to the neighborhood sample, if the operation duration corresponding to the neighborhood sample is shorter than the operation duration of the target sample, determining the neighborhood sample as a new target sample, and continuing to determine a new neighborhood sample corresponding to the new target sample and an operation duration corresponding to the new neighborhood sample until a preset iteration termination condition is met; and, according to a finally determined allocation solution corresponding to the target sample, allocating hardware resources to the operators of the deep learning model so as to perform deployment.
Need to check novelty before this filing date? Find Prior Art

Description

A search-based deep learning model deployment method and device Technical Field

[0001] The present disclosure relates to the cross-technical fields of deep learning, compilation technology, and high-performance computing, and in particular to a search-based deep learning model deployment method and device. Background Art

[0002] Deep learning technology has been widely used in fields such as machine vision, natural language processing, and autonomous driving, achieving remarkable results. As deep learning models become increasingly complex and require larger and larger parameters, ensuring model performance becomes a key issue.

[0003] In practical applications, deep learning models can be run on higher-performance hardware. Typically, dedicated neural network acceleration chips are available on the market. Furthermore, deep learning compilers can be used to build and optimize deep learning models. Using deep learning compilers, deep learning models can be run on CPUs (central processing units), graphics cards (GPUs), and other processors.

[0004] In actual deployment scenarios, deep learning models can be deployed on multiple hardware, such as CPUs, GPUs, and multiple neural network acceleration chips. However, when deploying deep learning models on multiple hardware, it is usually based on human experience to assign specified parts of the deep learning model to certain specified hardware for execution. This may not necessarily maximize the performance of the hardware to efficiently run the deep learning model.

[0005] Therefore, how to improve the operating efficiency of deep learning models is an urgent problem to be solved.

[0006] Summary of the Invention

[0007] The present disclosure provides a search-based deep learning model deployment method and device to partially solve the above-mentioned problems existing in the related technology.

[0008] This disclosure adopts the following technical solutions:

[0009] The present disclosure provides a search-based deep learning model deployment method, comprising:

[0010] Get the computational graph corresponding to the deep learning model;

[0011] Determine the operators included in the computation graph, and determine one or more hardware resources matched to each operator;

[0012] Constructing a search space based on one or more hardware resources matched by each operator, wherein for each sample included in the search space, the sample is used to represent an allocation scheme for allocating operators to hardware resources;

[0013] Selecting a target sample from the search space;

[0014] Determine the running time corresponding to the target sample;

[0015] Determine the neighborhood samples corresponding to the target sample, and determine the running time corresponding to the neighborhood samples;

[0016] If the runtime corresponding to the neighborhood sample is shorter than the runtime of the target sample, the neighborhood sample is used as a new target sample, and new neighborhood samples corresponding to the new target sample and the runtime corresponding to the new neighborhood samples are continuously determined until a preset iteration termination condition is met;

[0017] According to the allocation plan corresponding to the target sample finally determined, the hardware resources of the operators of the deep learning model are allocated to deploy the deep learning model.

[0018] In some embodiments, determining one or more hardware resources that each operator matches includes:

[0019] For each operator, determine whether the operator is supported by the preset hardware computing library;

[0020] If the operator is supported by the preset hardware computing library, the hardware resources matching the operator are determined based on the hardware computing library matching the operator. If the operator is not supported by the preset hardware computing library, the operator is determined to be assigned to the deep learning compiler for compilation, and the hardware resources matching the operator are the default hardware resources corresponding to the deep learning compiler.

[0021] In some embodiments, obtaining a computation graph corresponding to a deep learning model includes:

[0022] Determine, by a deep learning compiler, a high-level intermediate representation corresponding to the deep learning model;

[0023] Performing graph optimization on the high-level intermediate representation to obtain an optimized computational graph;

[0024] The optimized computational graph is used as the computational graph corresponding to the acquired deep learning model.

[0025] In some embodiments, selecting a target sample from the search space includes:

[0026] An initial sample is randomly selected from the search space as a target sample.

[0027] In some embodiments, for each sample, the neighborhood sample corresponding to the sample is obtained by randomly selecting an operator in the allocation scheme corresponding to the sample and randomly replacing the hardware resources allocated to the operator.

[0028] In some embodiments, determining the running time corresponding to the sample includes:

[0029] According to the allocation scheme corresponding to the sample, at least two adjacent operators that are assigned to the same hardware and conform to a preset logical relationship in the computation graph are merged to obtain a compilation unit; and / or the operators that are not merged are individually determined as a compilation unit;

[0030] For each compilation unit, determine the code corresponding to the compilation unit, and compile based on the code corresponding to the compilation unit to obtain an executable file corresponding to the compilation unit;

[0031] Link the executable files corresponding to each compilation unit to obtain a linked executable file;

[0032] Based on the linked executable file, the deep learning model is executed on the allocated hardware resources according to the allocation plan corresponding to the sample to obtain the running time corresponding to the sample.

[0033] In some embodiments, the preset iteration termination condition includes: reaching a preset number of iterations.

[0034] In some embodiments, satisfying a preset iteration termination condition includes:

[0035] If the running time corresponding to the target sample tends to be consistent in at least N iterations, it is determined that the preset iteration termination condition is met, where N is a positive integer.

[0036] The present disclosure provides a search-based deep learning model deployment device, comprising:

[0037] The acquisition module is used to obtain the computational graph corresponding to the deep learning model;

[0038] An operator determination module, configured to determine the operators included in the computation graph and determine one or more hardware resources that match each operator;

[0039] A construction module, configured to construct a search space according to one or more hardware resources matched by each operator, wherein for each sample included in the search space, the sample is used to represent an allocation scheme for allocating operators to hardware resources;

[0040] an iteration module, configured to select a target sample from the search space; determine a runtime corresponding to the target sample; determine a neighborhood sample corresponding to the target sample, and determine a runtime corresponding to the neighborhood sample; if the runtime corresponding to the neighborhood sample is shorter than the runtime of the target sample, determine the neighborhood sample as a new target sample, and continue to determine a new neighborhood sample corresponding to the new target sample and a runtime corresponding to the new neighborhood sample, until a preset iteration termination condition is met;

[0041] The deployment module is used to allocate hardware resources to the operators of the deep learning model according to the allocation plan corresponding to the target sample finally determined, so as to deploy the deep learning model.

[0042] In some embodiments, the operator determination module is specifically used to determine, for each operator, whether the operator is supported by a preset hardware computing library; if the operator is supported by the preset hardware computing library, determine the hardware resources matching the operator based on the hardware computing library matching the operator; if the operator is not supported by the preset hardware computing library, determine to assign the operator to a deep learning compiler for compilation, and the hardware resources matching the operator are the default hardware resources corresponding to the deep learning compiler.

[0043] In some embodiments, the acquisition module is specifically used to determine the high-level intermediate representation corresponding to the deep learning model through a deep learning compiler; perform graph optimization on the high-level intermediate representation to obtain an optimized computational graph; and use the optimized computational graph as the computational graph corresponding to the acquired deep learning model.

[0044] In some embodiments, the iteration module is specifically configured to randomly select an initial sample from the search space as a target sample.

[0045] In some embodiments, for each sample, the neighborhood sample corresponding to the sample is obtained by randomly selecting an operator in the allocation scheme corresponding to the sample and randomly replacing the hardware allocated to the operator.

[0046] The present disclosure provides a computer-readable storage medium storing a computer program, which implements the above-mentioned search-based deep learning model deployment method when executed by a processor.

[0047] The present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned search-based deep learning model deployment method is implemented.

[0048] At least one of the above technical solutions adopted in the present disclosure can achieve the following beneficial effects:

[0049] It can be seen from the above-mentioned search-based deep learning model deployment method that the computational graph corresponding to the deep learning model can be obtained, the operators included in the computational graph can be determined, and the hardware resources matching each operator can be determined. Then, according to the hardware resources matching each operator, a search space is constructed. For each sample included in the search space, the sample is used to represent an allocation scheme for allocating operators to each hardware resource. Then, a target sample is selected from the search space, and the running time corresponding to the target sample is determined, and the neighborhood sample corresponding to the target sample is determined, and the running time corresponding to the neighborhood sample is determined. If the running time corresponding to the neighborhood sample is shorter than the running time of the target sample, the neighborhood sample is used as the re-determined target sample, and the neighborhood sample corresponding to the target sample and the running time corresponding to the neighborhood sample are continued to be determined until the preset iteration termination condition is met. According to the allocation scheme corresponding to the target sample, the operator of the deep learning model is allocated hardware resources to deploy the deep learning model.

[0050] From the above content, it can be seen that the search-based deep learning model deployment method provided by the present disclosure can construct a search space composed of samples corresponding to various allocation schemes through hardware resources matched by operators, and search in the search space to continuously determine neighborhood samples, thereby continuously replacing neighborhood samples with shorter running times with target samples. Through continuous iteration, the optimal samples can be determined as much as possible, and according to the optimal samples, the hardware resources of the operators of the deep learning model are allocated, so that the deep learning model is deployed. Therefore, this method can improve the computational efficiency of the deep learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute improper limitations on the present disclosure.

[0052] FIG1 is a flow chart of a search-based deep learning model deployment method provided in the present disclosure.

[0053] FIG2 is a schematic diagram of a search space provided in the present disclosure.

[0054] FIG3 is a schematic diagram of a sample in the search space provided in the present disclosure.

[0055] FIG4 is a schematic structural diagram of a search-based deep learning model deployment device provided in the present disclosure.

[0056] FIG5 is a schematic diagram of an electronic device corresponding to FIG1 provided by the present disclosure. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with the specific embodiments of the present disclosure and the corresponding drawings. The described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0058] The technical solutions provided by various embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0059] FIG1 is a flow chart of a search-based deep learning model deployment method provided in the present disclosure, which includes the following steps S100 - S108 .

[0060] S100: Obtain a computational graph corresponding to a deep learning model.

[0061] S102: Determine the operators included in the computation graph, and determine one or more hardware resources matching each operator.

[0062] S104: Construct a search space according to one or more hardware resources matched by each operator. For each sample included in the search space, the sample is used to represent an allocation scheme for allocating operators to hardware resources.

[0063] In practical applications, deep learning models can be deployed on multiple hardware resources. That is, the deep learning model can be divided into multiple operators, and multiple hardware resources can execute the multiple operators.

[0064] Based on this, the server can obtain the computational graph corresponding to the deep learning model, determine the operators contained in the computational graph, and determine one or more hardware resources that match each operator.

[0065] Among them, the server can determine the high-level intermediate representation (high-level IR) corresponding to the above-mentioned deep learning model through the deep learning compiler, and perform graph optimization on the high-level intermediate representation to obtain an optimized computational graph, and then use the optimized computational graph as the computational graph corresponding to the acquired deep learning model. The graph optimization mentioned here may refer to the graph optimization method performed by a conventional deep learning compiler, which may include constant folding, dead code elimination, common subexpression elimination, etc.

[0066] The computational graph mentioned above can refer to a directed acyclic graph that represents the logical relationships between the operators included in a deep learning model. The operators mentioned here can refer to the basic computational units that constitute a deep learning model. For example, different activation functions can be used as different operators; different convolution operations can be used as different operators; summation, product, and modulus operations can also be used as separate operators. A deep learning model can be composed of multiple operators.

[0067] Each operator may correspond to one or more matching hardware resources. For example, there are hardware resources 1, hardware resources 2, and hardware resources 3. Operator 1 may be supported by hardware resources 1, hardware resources 2, and hardware resources 3, while operator 2 may only be supported by hardware resources 1 and hardware resources 2.

[0068] A search space can be constructed based on one or more hardware resources matched by each operator. For each sample included in the search space, the sample is used to represent an allocation scheme for allocating operators to hardware resources. The search space can include samples corresponding to all allocation schemes determined based on one or more hardware resources matched by each operator.

[0069] The following is a simple example to illustrate the computational graph of a deep learning model, the computing resources matched to each operator, and the search space constructed based on the former, as shown in Figures 2 and 3.

[0070] FIG2 is a schematic diagram of a search space provided in the present disclosure.

[0071] FIG3 is a schematic diagram of a sample in the search space provided in the present disclosure.

[0072] The deep learning model corresponding to the computation graph shown in Figure 2 contains five operators: Conv2D_01 (convolution operator #1) corresponds to three matching hardware resources (hardware resources 1, 2, and 3), Add_02 (summation operator #2) corresponds to two matching hardware resources (hardware resources 1 and 2), ReLu_03 (activation function operator #3) corresponds to one matching hardware resource (hardware resource 1), Conv2D_04 (convolution operator #4) corresponds to three matching hardware resources (hardware resources 1, 2, and 3), and ReLu_05 (activation function operator #5) corresponds to one matching hardware resource (hardware resource 1). As can be seen, the computation graph can illustrate the logical relationship (input-output relationship) between operators.

[0073] Therefore, the search space size formed by the five operators in the computation graph is 3*2*1*3*1=18 (the number of samples is 18), that is, there are 18 feasible allocation schemes for allocating the operators in the computation graph to the hardware resources.

[0074] The samples listed in FIG3 correspond to an allocation scheme in the search space shown in FIG2 , that is, Conv2D_01, Add_02, and ReLu_03 are allocated to hardware resource 1, Conv2D_04 is allocated to hardware resource 3, and ReLu_05 is allocated to hardware resource 2.

[0075] It should be noted that for the sake of ease of description, the server is used as the executor of this method to illustrate the overall process of this method, but there is no restriction on the executor of this method. The executor can be a server, desktop computer or other electronic device.

[0076] It should also be noted that since there may be multiple hardware resources, such as multiple neural network acceleration chips and CPUs, GPUs, etc., for example, a neural network acceleration chip may support certain operators, and CPUs and GPUs may also support certain operators. When determining one or more hardware resources that match an operator, it is possible to determine for each operator whether the operator is supported by a preset hardware computing library. If so, the hardware resources that match the operator can be determined based on the hardware computing library that matches the operator. If not, it is determined that the operator is assigned to the deep learning compiler for compilation, and the hardware resources that match the operator are: the default hardware resources corresponding to the deep learning compiler.

[0077] The default hardware resource can be a CPU or a GPU. When the hardware resource matched by the operator is the default hardware resource, it can be understood that in each sample in the search space, the hardware resource corresponding to the operator is fixed. If the operator can be supported by the hardware computing library of the hardware, then the operator can be supported by the hardware, and then the operator matches the hardware resources of the hardware. For example, a hardware computing library of a CPU or GPU includes oneDNN, and the operators supported by oneDNN include activation function operators, summation operators, etc.; a hardware computing library of a GPU includes cuDNN, and the operators supported by cuDNN include convolution operators, pooling operators, activation function operators, etc. The specific configuration of the hardware computing library can be obtained according to actual conditions, and this disclosure does not impose any restrictions on this.

[0078] S106: Select a target sample from the search space, and determine the running time corresponding to the target sample, and determine the neighborhood sample corresponding to the target sample, and determine the running time corresponding to the neighborhood sample. If the running time corresponding to the neighborhood sample is shorter than the running time of the target sample, use the neighborhood sample as a new target sample, and continue to determine the new neighborhood sample corresponding to the new target sample and the running time corresponding to the new neighborhood sample until the preset iteration termination condition is met.

[0079] S108: Allocate hardware resources to the operators of the deep learning model according to the allocation plan corresponding to the target sample finally determined, so as to deploy the deep learning model.

[0080] After the server determines the search space, it can select a target sample from the search space and determine the running time corresponding to the target sample, as well as determine the neighborhood sample corresponding to the target sample, and determine the running time corresponding to the neighborhood sample. If the running time corresponding to the neighborhood sample is shorter than the running time of the target sample, the neighborhood sample is used as the new target sample to be re-determined, and the new neighborhood sample corresponding to the new target sample and the running time corresponding to the new neighborhood sample are continued to be determined until the preset iteration termination condition is met. According to the allocation plan corresponding to the target sample finally determined, the hardware resources of the operators of the deep learning model are allocated to deploy the deep learning model.

[0081] In other words, by continuously searching samples in the search space, we can find the allocation scheme corresponding to the sample with the best possible running time. The above process is to search iteratively. Initially, a target sample is determined, and the running time corresponding to the target sample is determined. That is, the actual running time of running the deep learning model after allocating each operator to hardware resources according to the allocation scheme corresponding to the target sample is determined.

[0082] Then, the neighborhood samples of the target sample can be determined, and the running time corresponding to the neighborhood samples can be determined. For a sample, the neighborhood samples corresponding to the sample can be obtained by randomly selecting an operator in the allocation scheme corresponding to the sample and randomly replacing the hardware resources allocated to it (the operator).

[0083] Assume there are five operators in the computation graph, each supported by three types of hardware resources. A possible sample in the search space is encoded as (1,1,2,3,2): operators numbered 1 and 2 are assigned to hardware 1, operators numbered 3 and 5 are assigned to hardware 2, and operator numbered 4 is assigned to hardware 3. The neighborhood sample generation rule is as follows: an operator is randomly selected and its assigned hardware is randomly changed. For the above sample, (1,1,2,1,2) and (1,2,2,3,2) are both its neighborhood samples.

[0084] Of course, other methods can also be used to determine neighborhood samples, as long as the determined neighborhood samples are different from the samples in the previous iteration. It should be noted that each time the neighborhood samples corresponding to a sample are determined, it is equivalent to generating a new neighborhood sample. If the newly generated neighborhood samples have been tested on the hardware for a long time, the neighborhood samples can be regenerated until an unevaluated neighborhood sample is generated.

[0085] After determining the running time corresponding to the neighborhood sample, the running time between the neighborhood sample and the target sample can be compared. If the running time corresponding to the neighborhood sample is shorter than the running time of the target sample, the neighborhood sample is re-used as the target sample, and the neighborhood samples of this target sample and the corresponding running time are further determined.

[0086] If the runtime corresponding to the neighborhood sample is not shorter than the runtime of the target sample, then the other neighborhood samples of the original target sample and the runtime corresponding to other neighborhood samples can be determined. That is, the iteration is continued until a certain preset iteration termination condition is satisfied, and the iteration is stopped. The runtime corresponding to the target sample at the last iteration is the shortest runtime searched out. That is to say, each time the runtime corresponding to a sample is determined, it is considered an iteration. The neighborhood search can be continuously performed to continuously determine the runtime corresponding to the new sample. If there is a new sample whose runtime is shorter than the lowest runtime of all previously tested samples, this sample will be replaced with the target sample. In this way, in the end, the target sample is the sample with the best runtime searched out.

[0087] There may be multiple preset iteration termination conditions. For example, when a preset number of iterations is reached, it can be determined that the preset iteration termination condition is met. The preset number of iterations may be preset based on the number of operators. For example, if the number of operators is small, the number of iterations may be small. If the number of operators is large, the number of iterations may be large.

[0088] For another example, if the runtimes corresponding to the target samples converge over at least N iterations, then the predetermined iteration termination condition can be determined to be satisfied, where N is a positive integer. The "at least N times" mentioned here can mean that at the end of the iteration, if the runtimes corresponding to the target samples in each iteration are similar and do not decrease, then convergence is achieved, and the iterations can be terminated, with the target sample from the last iteration being the optimal sample.

[0089] In the above process, initially, there may be multiple target samples selected from the search space. For example, an initial sample may be randomly selected from the search space as the target sample. For another example, a basic priority sample may be selected from the search space as the target sample. The basic priority sample may be a more appropriate sample selected based on experience.

[0090] It should be noted that to determine the running time of a sample, it is necessary to conduct actual measurements on the actual hardware allocated to the operator according to the allocation plan corresponding to the sample to determine the running time.

[0091] Specifically, the server can merge at least two adjacent operators that are assigned to the same hardware and conform to a preset logical relationship in the calculation graph according to the allocation scheme corresponding to the sample to obtain a compilation unit, and the operators that are not merged are treated as separate compilation units, and each compilation unit contains at least one operator.

[0092] Then, for each compilation unit, the code corresponding to the compilation unit is determined, and the code corresponding to the compilation unit is compiled to obtain an executable file corresponding to the compilation unit. The executable files corresponding to the compilation units are then linked to obtain a linked executable file. Furthermore, based on the linked executable file, the deep learning model is executed on the allocated hardware resources according to the allocation scheme corresponding to the sample to obtain the runtime corresponding to the sample.

[0093] The adjacent operators mentioned above in the computation graph may refer to operators that are connected. For example, in FIG3 , the three operators Conv2D_01 , Add_02 , and ReLu_03 are adjacent, and the two operators Conv2D_04 and ReLu_05 are adjacent.

[0094] Complying with the preset logical relationship may mean that when adjacent operators are divided into the same hardware, the inputs of the remaining operators except the initial operator do not include the inputs of operators allocated to other hardware. The initial operator refers to the operator that ranks first among these adjacent operators according to the execution order in the computation graph.

[0095] For example, for the three adjacent operators Conv2D_01, Add_02, and ReLu_03 in the computational graph of Figure 3, these three adjacent operators are all assigned to the same hardware, and Add_02 and ReLu_03 (the initial operator is Conv2D_01) among the three adjacent operators have no input from operators of other hardware. Therefore, these three adjacent operators can form a compilation unit.

[0096] Some operators can be merged into compilation units, while some operators cannot be merged. In this case, one operator can be used as a compilation unit. Then, each compilation unit can be compiled according to the code, and their respective executable files can be linked to obtain the overall executable file.

[0097] The executable file can be run on a device where the required hardware resources are deployed, i.e., the deep learning model can be run. By running according to the executable file, each compilation unit can be assigned to the corresponding hardware resource for execution, thereby completing the overall deep learning model operation and obtaining the runtime corresponding to the sample. The device mentioned here can refer to the above-mentioned server or another electronic device.

[0098] It should be noted that the deep learning model in this disclosure does not limit its application scenarios. The deep learning model can be an image recognition model, an information recommendation model, etc.

[0099] It should be noted that after the iteration is completed, the allocation scheme corresponding to the target sample in the last iteration can be used as the allocation scheme for deploying the deep learning model. The above-mentioned hardware resources can be located in the same device (the device on which the deep learning model needs to be deployed). According to the allocation scheme corresponding to the target sample, the deep learning model is deployed on the device. Each hardware resource in the device can execute the operators in the deep learning model according to the allocation scheme corresponding to the target sample, so that the device can run the deep learning model. The device can be the above-mentioned server, or another separate device (the server is used to determine the allocation scheme).

[0100] From the above content, it can be seen that the search-based deep learning model deployment method provided by the present disclosure can construct a search space composed of samples corresponding to various allocation schemes through hardware resources matched by operators, and search in the search space to continuously determine neighborhood samples, thereby continuously replacing neighborhood samples with shorter running times with target samples. Through continuous iteration, the optimal samples can be determined as much as possible, and according to the optimal samples, the hardware resources of the operators of the deep learning model are allocated, so that the deep learning model is deployed. Therefore, this method can improve the computational efficiency of the deep learning model.

[0101] In general, the above technical solutions conceived by the present disclosure have the following beneficial effects compared with related technologies: First, the present disclosure combines the operator generation function of the deep learning compiler and the performance advantages of various hardware computing libraries (neural network chips); Second, in the case of multiple hardware deployments, the present disclosure introduces search technology to fully utilize the advantages of each hardware to achieve higher deployment performance.

[0102] Based on the above-mentioned search-based deep learning model deployment method, the present disclosure also provides a search-based deep learning model deployment device, as shown in Figure 4.

[0103] Acquisition module 401, used to obtain the calculation graph corresponding to the deep learning model;

[0104] An operator determination module 402 is configured to determine the operators included in the computation graph and determine one or more hardware resources that match each operator;

[0105] A construction module 403 is configured to construct a search space based on one or more hardware resources matched by each operator, wherein for each sample included in the search space, the sample is used to represent an allocation scheme for allocating operators to hardware resources;

[0106] Iterative module 404 is configured to select a target sample from the search space; determine the runtime corresponding to the target sample; determine a neighboring sample corresponding to the target sample, and determine the runtime corresponding to the neighboring sample; if the runtime corresponding to the neighboring sample is shorter than the runtime of the target sample, determine the neighboring sample as a new target sample, and continue to determine a new neighboring sample corresponding to the new target sample and the runtime corresponding to the new neighboring sample until a preset iteration termination condition is met;

[0107] The deployment module 405 is used to allocate hardware resources to the operators of the deep learning model according to the allocation plan corresponding to the target sample finally determined, so as to deploy the deep learning model.

[0108] In some embodiments, the operator determination module 402 is specifically used to determine, for each operator, whether the operator is supported by a preset hardware computing library; if the operator is supported by the preset hardware computing library, determine the hardware resources matching the operator based on the hardware computing library matching the operator; if the operator is not supported by the preset hardware computing library, determine to assign the operator to a deep learning compiler for compilation, and the hardware resources matching the operator are: the default hardware resources corresponding to the deep learning compiler.

[0109] In some embodiments, the acquisition module 401 is specifically used to determine the high-level intermediate representation corresponding to the deep learning model through a deep learning compiler, and perform graph optimization on the high-level intermediate representation to obtain an optimized computational graph; and use the optimized computational graph as the computational graph corresponding to the acquired deep learning model.

[0110] In some embodiments, the iteration module 404 is specifically configured to randomly select an initial sample from the search space as a target sample.

[0111] In some embodiments, for each sample, the neighborhood sample corresponding to the sample is obtained by randomly selecting an operator in the allocation scheme corresponding to the sample and randomly replacing the hardware resources allocated to the operator.

[0112] In some embodiments, the iteration module 404 is specifically used to, according to the allocation scheme corresponding to the sample, merge at least two adjacent operators that are assigned to the same hardware and conform to a preset logical relationship in the computation graph to obtain a compilation unit; and / or separately determine the operators that have not been merged as a compilation unit; for each compilation unit, determine the code corresponding to the compilation unit, and compile based on the code corresponding to the compilation unit to obtain an executable file corresponding to the compilation unit; link the executable files corresponding to each compilation unit to obtain a linked executable file; based on the linked executable file, execute the deep learning model on the allocated hardware resources according to the allocation scheme corresponding to the sample to obtain the running time corresponding to the sample.

[0113] In some embodiments, the preset iteration termination condition includes: reaching a preset number of iterations.

[0114] In some embodiments, the iteration module 404 is specifically configured to determine that the preset iteration termination condition is satisfied if the running time corresponding to the target sample tends to be consistent in at least N iterations, where N is a positive integer.

[0115] The present disclosure also provides a computer-readable storage medium storing a computer program, which can be used to execute the above-mentioned search-based deep learning model deployment method.

[0116] This disclosure also provides a schematic block diagram of an electronic device, shown in Figure 5. As shown in Figure 5, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile storage, and may also include other hardware required for its services. The processor reads the corresponding computer program from the non-volatile storage into the memory and then runs it to implement the above-mentioned search-based deep learning model deployment method.

[0117] Of course, in addition to software implementation, this disclosure does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0118] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0119] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0120] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0121] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0122] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0124] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0126] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0127] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0128] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0129] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0130] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] The present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0132] The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0133] The foregoing is merely an embodiment of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure are intended to be included within the scope of the claims of the present disclosure.

Claims

1. A search-based deep learning model deployment method, characterized in that: include: Get the computational graph corresponding to the deep learning model; Determine the operators included in the computation graph, and determine one or more hardware resources matched by each operator; Constructing a search space according to one or more hardware resources matched by each operator, wherein for each sample included in the search space, the sample is used to represent an allocation scheme for allocating operators to hardware resources; Selecting a target sample from the search space; Determine the running time corresponding to the target sample; Determine a neighborhood sample corresponding to the target sample, and determine a running time corresponding to the neighborhood sample; If the running time corresponding to the neighborhood sample is shorter than the running time of the target sample, the neighborhood sample is determined as a new target sample, and a new neighborhood sample corresponding to the new target sample and the running time corresponding to the new neighborhood sample are continuously determined until a preset iteration termination condition is met; Allocate hardware resources to operators of the deep learning model according to the allocation scheme corresponding to the target sample finally determined, so as to deploy the deep learning model; The determination of one or more hardware resources matching each operator includes: For each operator, determine whether the operator is supported by the preset hardware computing library; If the operator is supported by the preset hardware computing library, determine the hardware resources matched by the operator according to the hardware computing library matched with the operator; if the operator is not supported by the preset hardware computing library, determine to assign the operator to the deep learning compiler for compilation, and the hardware resources matched by the operator are the default hardware resources corresponding to the deep learning compiler; Among them, obtaining the calculation graph corresponding to the deep learning model includes: Determine, by a deep learning compiler, a high-level intermediate representation corresponding to the deep learning model; Optimizing the high-level intermediate representation to obtain an optimized computational graph; The optimized computational graph is used as the computational graph corresponding to the acquired deep learning model.

2. The method according to claim 1, characterized in that Selecting a target sample from the search space includes: An initial sample is randomly selected from the search space as a target sample.

3. The method according to claim 1, characterized in that For each sample, the neighborhood sample corresponding to the sample is obtained by randomly selecting an operator in the allocation scheme corresponding to the sample and randomly replacing the hardware resources allocated to the operator to obtain the neighborhood sample corresponding to the sample.

4. The method according to claim 1, characterized in that Determine the runtime corresponding to the sample, including: According to the allocation scheme corresponding to the sample, at least two adjacent operators that are allocated to the same hardware and conform to a preset logical relationship in the calculation graph are merged to obtain a compilation unit; and / or the operators that are not merged are separately determined as a compilation unit; For each compilation unit, determine the code corresponding to the compilation unit, and compile based on the code corresponding to the compilation unit to obtain an executable file corresponding to the compilation unit; Link the executable files corresponding to each compilation unit to obtain a linked executable file; Based on the linked executable file, the deep learning model is executed on the allocated hardware resources according to the allocation scheme corresponding to the sample to obtain the running time corresponding to the sample.

5. The method according to claim 1, characterized in that The preset iteration termination condition includes: reaching a preset number of iterations.

6. The method according to claim 1, characterized in that The preset iteration termination conditions are met, including: If the running time corresponding to the target sample tends to be consistent in at least N iterations, it is determined that the preset iteration termination condition is met, where N is a positive integer.

7. A search-based deep learning model deployment device, characterized in that: include: The acquisition module is used to obtain the computational graph corresponding to the deep learning model; An operator determination module, used to determine the operators included in the computation graph, and to determine one or more hardware resources matched by each operator; A construction module, configured to construct a search space according to one or more hardware resources matched by each operator, wherein for each sample included in the search space, the sample is used to represent an allocation scheme for allocating operators to hardware resources; An iteration module is used to select a target sample from the search space, determine the running time corresponding to the target sample, determine the neighborhood sample corresponding to the target sample, and determine the running time corresponding to the neighborhood sample. If the running time corresponding to the neighborhood sample is shorter than the running time of the target sample, determine the neighborhood sample as a new target sample, and continue to determine a new neighborhood sample corresponding to the new target sample and the running time corresponding to the new neighborhood sample until a preset iteration termination condition is met; The deployment module is used to allocate hardware resources to the operators of the deep learning model according to the allocation plan corresponding to the target sample finally determined, so as to deploy the deep learning model.

8. The device according to claim 7, characterized in that The operator determination module is specifically used to determine, for each operator, whether the operator is supported by a preset hardware computing library; if the operator is supported by the preset hardware computing library, determine the hardware resources matching the operator according to the hardware computing library matching the operator; if the operator is not supported by the preset hardware computing library, determine to assign the operator to a deep learning compiler for compilation, and the hardware resources matching the operator are: the default hardware resources corresponding to the deep learning compiler.

9. The device according to claim 7, characterized in that The acquisition module is specifically used to determine the high-level intermediate representation corresponding to the deep learning model through a deep learning compiler, and perform graph optimization on the high-level intermediate representation to obtain an optimized computational graph; and use the optimized computational graph as the computational graph corresponding to the acquired deep learning model.

10. The device according to claim 7, characterized in that The iteration module is specifically used to randomly select an initial sample from the search space as a target sample.

11. The device according to claim 7, characterized in that For each sample, the neighborhood sample corresponding to the sample is obtained by randomly selecting an operator in the allocation scheme corresponding to the sample and randomly replacing the hardware allocated to the operator to obtain the neighborhood sample corresponding to the sample.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Model using method, data processing method and device

    CN111752716A

  • Model deployment method, model deployment device, terminal equipment and readable storage medium

    CN113220457A

  • Optimization method for executing deep learning tasks in distributed mode and distributed system

    CN115543639A

  • Deep learning model deployment method and device based on search

    CN116306856A

  • Compiler-level general matrix multiplication configuration optimization

    US20210200521A1