Method of tailoring grouped neural execution considering heterogeneity on multiple edge and framework apparatus applying the same
The framework addresses the inefficiencies in determining fusion layer configurations by using a cost matrix and dynamic programming to reduce profiling costs and time, enabling efficient and cost-effective configuration search in heterogeneous computing environments.
Patent Information
- Application Number
- KR1020220045572
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-04-13
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2042-04-13
AI Technical Summary
Existing methods for determining optimal fusion layer configurations in heterogeneous computing environments are costly and time-consuming, requiring extensive profiling and are limited to single-device experiments, making it difficult to efficiently find the optimal configuration for deep learning models.
A framework and method using a cost matrix and dynamic programming to efficiently search for optimal fusion layer configurations by profiling at runtime, considering the performance and network performance of different devices, reducing the need for extensive upfront profiling.
Enables efficient and cost-effective determination of optimal fusion layer configurations with minimal profiling overhead, applicable to various model types and heterogeneous device combinations.
Smart Images

Figure R1020220045572_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a method for searching for a convergence layer configuration using convergence layer-wise parallelization in a heterogeneous computing environment and a framework for applying the same. More specifically, it relates to a method for searching for a convergence layer configuration using convergence layer-wise parallelization in a heterogeneous computing environment and a framework for applying the same, which enables searching for an optimal convergence layer configuration at a low cost by considering the performance and network performance of different devices in a heterogeneous computing environment. Background Technology
[0003] It was reported that a regression model was constructed to predict the time required for computation and data transmission when utilizing the fused layer-wise parallelization technique, but there are limitations in that the process of building the model requires a profiling process that generates a very large amount of prior data, and only one type of device was used with varying performance during the experiment.
[0004] Fusion layer-wise parallelization for deep learning models is a model parallelization-based method that prevents inter-device communication from occurring while performing multiple fusion-capable layers, namely fusion layers.
[0005] In this process, since the input data of the fusion layer becomes larger than that of the existing single layer and additional redundant operations are added between devices, it is more effective when the amount of computation within the fused layers exceeds the level that individual devices can handle.
[0006] However, since various cases must be considered to determine the optimal starting point and extent of merging between layers, it is difficult to determine this quickly and at low cost. The problem to be solved
[0008] The present invention aims to provide a method for searching for a fusion layer configuration using fusion layer-wise parallelization in a heterogeneous computing environment, which enables efficient searching for the configuration of an optimal fusion layer with only a small number of profilings, and a framework for applying the same.
[0009] In addition, the present invention aims to provide a method for searching for a convergence layer configuration using convergence layer-wise parallelization in a heterogeneous computing environment and a framework for applying the same, which enables searching for an optimal convergence layer configuration at low cost by considering the performance and network performance of different devices in a heterogeneous computing environment.
[0010] In addition, the present invention aims to provide a method for exploring a fusion layer configuration using fusion layer-wise parallelization in a heterogeneous computing environment and a framework for applying the same, which determines the optimal configuration by constructing a cost matrix for the current configuration through simple profiling performed at runtime and enables the method to be applied to various model types and heterogeneous device combinations.
[0012] The objects of the present invention are not limited to those mentioned above, and other unmentioned objects and advantages of the present invention may be understood from the following description and will be more clearly understood by the embodiments of the present invention. Furthermore, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims. means of solving the problem
[0014] A framework to which a method for searching for a fused layer configuration using fused layer-wise parallelization is applied in a heterogeneous computing environment to achieve this purpose includes: a preprocessing unit that specifies the number of an edge device to be used in an edge device pool among a number of devices and extracts layer functions required for partition execution for the device; a cost prediction unit that generates a fused layer configuration (FLC) using the layer functions required for partition execution received from the preprocessing unit, performs a performance evaluation on an execution graph corresponding to the FLC, stores the profile results of the performance evaluation in different matrices, generates an alpha matrix based on the layer functions required for partition execution, and generates a cost matrix using the different matrices and the alpha matrix; an FLC search unit that searches for a fused layer configuration using the cost matrix; and a parallelization unit that parallelizes the device based on the searched fused layer configuration to execute a model.
[0015] In addition, a method for searching for a fused layer configuration using fused layer-wise parallelization in a heterogeneous computing environment to achieve this purpose includes the steps of: a preprocessing unit specifying the number of an edge device to be used in an edge device pool among a plurality of devices and extracting layer functions required for partition execution for the device; a cost prediction unit generating a fused layer configuration (FLC) using the layer functions required for partition execution received from the preprocessing unit; the cost prediction unit performing a performance evaluation of an execution graph corresponding to the FLC and storing the profile results of the performance evaluation in different matrices; the cost prediction unit generating an alpha matrix based on the layer functions required for partition execution and generating a cost matrix using the different matrix and the alpha matrix; and an FLC search unit searching for and providing a fused layer configuration using the cost matrix. Effects of the invention
[0017] According to the present invention as described above, there is an advantage in that the configuration of the optimal fusion layer can be efficiently searched with only a small number of profilings.
[0018] In addition, according to the present invention, there is an advantage in that the optimal fusion layer configuration can be explored at a low cost by considering the performance and network performance of different devices in a heterogeneous computing environment.
[0019] In addition, according to the present invention, there is an advantage in that the optimal configuration is determined by constructing a cost matrix for the current configuration through simple profiling performed at runtime, and the method can be applied to various model types and heterogeneous device combinations. Brief explanation of the drawing
[0021] Figure 1 is an example of effectively minimizing response time in an execution flow considering multiple devices and a merging layer. FIG. 2 is an illustrative diagram for explaining a method for updating the cost of each element of a cost matrix according to one embodiment of the present invention. FIG. 3 is a diagram illustrating a framework to which a method for searching for a fusion layer configuration using fusion layer-wise parallelization in a heterogeneous computing environment according to an embodiment of the present invention is applied. FIG. 4 is a drawing for explaining an embodiment of evaluating the total cost when layers are fused according to an embodiment of the present invention. FIG. 5 is a diagram showing the predicted values of a cost matrix according to one embodiment of the present invention. Specific details for implementing the invention
[0022] The aforementioned objectives, features, and advantages are described in detail below with reference to the attached drawings, thereby enabling those skilled in the art to easily implement the technical concept of the present invention. In describing the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such descriptions would unnecessarily obscure the essence of the invention. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. In the drawings, the same reference numerals are used to indicate the same or similar components.
[0024] Figure 1 is an example of effectively minimizing response time in an execution flow considering multiple devices and a fusion layer.
[0025] FIG. 1(a) shows an example of execution on a single device, FIG. 1(b) shows an example of partition execution without layer fusion, and FIG. 1(c) shows an example of partition execution with layer fusion. In this case, the partition execution is known to utilize multiple partitioning and assignment model parallelization of the target neural network.
[0026] What can be executed across multiple networked devices is finding the optimal partition configuration. Layered parallelization is the simplest technique for partitioning each layer independently, but it incurs high network data transmission overhead between layers.
[0027] This is because all partial outputs from the splitting layer must be aggregated on the host side, and the converted inputs must be transmitted back to the target device executing the next splitting layer.
[0028] Layer-by-layer parallelization fusion can be utilized by identifying groups of layers. In this case, layers can be executed within the device without inter-layer data communication with the device host, and bulk partitioning is possible at the fusion layer level rather than at the single-layer level.
[0029] While fused layer-by-layer parallel execution minimizes data communication overhead between layers, partitioned fused layers may incur additional computational and communication overhead because layer-by-layer parallelization requires input data larger than that of the original layer. In this case, the overhead can outweigh the benefits of layer fusion, so it must be executed with caution.
[0030] The solution to this problem is to collect latency information for all possible layer fusion combinations by selecting the optimal fusion layer combination using a dynamic programming model that considers both computational and network transmission costs.
[0031] However, finding all information through profiling-based thorough search is nearly impossible because it requires a significant amount of computation and time overhead. Furthermore, as recent neural networks have become deeper, the time required for thorough search to find the optimal fusion layer configuration has increased significantly due to the large number of fusion layers. Regression model-based latency prediction is a common approach to drastically reduce profiling overhead.
[0032] However, building an accurate regression model that includes various features such as the input size, parameters, and shape of each layer still requires significant profiling overhead.
[0033] Therefore, to find the optimal parallel fusion layer-based execution strategy with minimal compilation and execution overhead for rapid deployment, an improved approach is required that goes beyond simple thorough search or regression-based techniques.
[0034] To address these difficulties, the present invention provides Legion, a lightweight workload offload framework, to minimize inference latency for multiple edge devices by finding the optimal fusion layer configuration with compilation and profiling execution overhead.
[0035] To find the optimal fusion layer configuration by specifying a specific number of computational units across multiple edge devices, the present invention first introduces the cost prediction unit (120) of FIG. 3 to fill in all elements of the cost matrix based on a linear model.
[0037] FIG. 2 is an illustrative diagram for explaining a method for updating the cost of each element of a cost matrix according to one embodiment of the present invention.
[0038] Referring to FIG. 2, FIG. 2(a) is a method for filling diagonal elements T[i, i] for FSS (full searching scheme) and then profiling each corresponding fusion layer group to fill other elements one by one, and FIG. 2(b) is a method according to an embodiment of the present invention in which only a single layer model is executed to fill diagonal elements, and the remaining elements are estimated based on profile information.
[0039] FSS is an approach that finds the optimal FLC using a pre-calculated cost matrix. FSS is used to find the optimal FLC by calculating costs per layer and per fusion layer and tabulating them. It consists of multiple fusion layer groups by minimizing C[0, n].
[0040] If k fusion layers exist, from l0 to l as in [Equation 1] n The layers up to f0∪f1...∪f k-1 It can be expressed as. The total cost C[0, n] can be calculated as [Equation 1].
[0042] [Mathematical Formula 1]
[0043]
[0045] In Figure 2(a), the cost T[i, j] is the value when the layers from the row index to the column index are fused. To accurately determine the optimal split point, all top triangular elements of the cost matrix must be pre-calculated using profiling runs. While it is desirable to reduce profiling run time by filling only a subset of elements before configuration search, some costs must be estimated based on missing elements during the search process.
[0046] A critical task in finding the optimal FLC is to update all elements in the upper triangle portion of the cost matrix. Assuming a target network of 5 layers as shown in Figure 2, the FSS must perform a total of 9 compilations and profilings.
[0047] For example, the target network can be decomposed into the configuration (f(0,0), f(1,1), f(2,2), f(3,3), f(4,4), and the diagonal elements of the matrix are updated through compilation and profiling execution as shown in Fig. 2(a). Also, when the network is decomposed into (f(0, 1), f(2, 3), f(4, 4), T[0, 1] and T[2, 3] are updated. T[4, 4] was already updated in the previous process.
[0048] By repeating these steps, a total of 9 compilation and profiling runs are required to update the cost values of all elements, and the number of compilation and profiling runs (#(C&E)) that fill the cost matrix of the target network into nl layers is calculated using [Equation 2].
[0050] [Mathematical Formula 2]
[0051]
[0053] As shown in [Equation 2], the complexity of calculating the (#(C&E)) value is O(). This makes the profiling cost of finding the optimal FLC expensive when the given network structure is quite deep. Therefore, the expensive profiling cost must be reduced by minimizing #(C&E). Submatrix construction and missing cost estimation can be an attractive alternative to the full matrix construction strategy to avoid significant profiling costs.
[0054] However, since the accuracy of this prediction cannot be guaranteed, it is an important task to obtain the optimal FLC while effectively reducing profiling costs by reducing #(C&E).
[0056] FIG. 3 is a diagram illustrating a framework to which a method for searching for a fusion layer configuration using fusion layer-wise parallelization in a heterogeneous computing environment according to an embodiment of the present invention is applied.
[0057] Referring to FIG. 3, a framework to which a fusion layer configuration search method using fusion layer-wise parallelization in a heterogeneous computing environment is applied includes a preprocessing unit (110), a cost prediction unit (120), and an FLC optimization unit (130).
[0058] The preprocessing unit (110) primarily handles the management of the entire edge device and the extraction of layer functions. First, the preprocessing unit (110) selects an edge device available for use from the edge device pool.
[0059] Here, the preprocessing unit (110) specifies the number of the device to be used. Then, the preprocessing unit (110) extracts the layer functions required for split execution and transmits them to the next module. At this time, the extracted layer functions include the kernel size (K), padding size (P), stride (S), and input and output sizes of each layer.
[0060] The cost prediction unit (120) includes a code compilation module (121), a runtime evaluation module (122), and a cost matrix generation module (123).
[0061] The code compilation module (121) compiles the code based on a specific layer function and a specific FLC.
[0062] First, the code compilation module (121) generates a default fused layer configuration (FLC) consisting only of a single layer that is not fused, and generates an execution graph corresponding to the FLC. At this time, the execution graph corresponding to the generated FLC is provided to the runtime evaluation module (122).
[0063] The runtime evaluation module (122) measures the performance of each device.
[0064] The runtime evaluation module (122) receives an execution graph corresponding to the FLC, performs a performance evaluation, and then stores the profile results for the performance evaluation in the Set Matrix (S), Run Matrix (E), and Get Matrix (R), respectively.
[0065] In one embodiment, the runtime evaluation module (122) receives an execution graph corresponding to the FLC and performs a performance evaluation of each device, and then stores the time of sending data among the profile results of the performance evaluation in the set matrix, the time of operation in the run matrix, and the time of receiving data in the get matrix.
[0066] As described above, profile results, such as the time required for data transmission, computation, and reception necessary for evaluating the performance of each device, are stored in different matrices.
[0067] Additionally, the runtime evaluation module (122) sets appropriate values in the alpha matrix based on the layer function. The values updated in the alpha matrix are the index i-th layer (l i In ) the j-th layer (l j It is the rate at which input data increases when fused with ), and is calculated using [Equation 3].
[0069] [Mathematical Formula 3]
[0070]
[0072] OLSd[i, j]: the i-th layer (l) on device d i Input size of )
[0073] OLSFd[i, j]: f on device d (i,j) Increased input size of
[0074] Ad[i, j]: Layer at index i of the alpha matrix on device d (l i ) and layer of index j(l j Overhead for connecting ),
[0075] n d : Number of devices in use
[0077] At this time, the i-th layer (l i In ) the j-th layer (l j When fusing with ), the increased size ratio of the input data affects the cost. Therefore, since the amount of computational overhead increases as the amount of OLSF increases, the cost of the fused layer can be predicted by multiplying the cost of a single layer by an alpha value.
[0078] When the set matrix, run matrix, get matrix, and alpha matrix are generated through the runtime evaluation module (122) above, the cost matrix generation module (123) begins to estimate the elements of the cost matrix using the set matrix, run matrix, get matrix, and alpha matrix.
[0079] The cost matrix generation module (123) uses [Equation 4] to predict T[i, j] of the cost matrix and considers the increasing input data size when fusing layers. Since all devices start running simultaneously, the total cost is determined based on the end time of the last device to complete execution.
[0081] [Mathematical Formula 4]
[0082]
[0084] T[i, j]: Total cost of fusion layer group f(i, j),
[0085] S d [i]: The i-th layer of the set matrix on device d (l i ) Data transmission costs,
[0086] R d [j]: the j-th layer of the run matrix on device d (l j Data reception cost of ),
[0087] A d[i, j]: the i-th layer of the alpha matrix on device d (l i ) and the j-th layer(l j The overhead incurred in combining ),
[0088] Ed[k, k]: l on device d k from l k Fusion block execution cost up to,
[0089] l i : Layer at index i,
[0090] l j : Layer at index i,
[0092] The FLC optimization unit (130) includes a search module (131) and a wild card module (132).
[0093] When all essential elements in the cost matrix are updated by the cost prediction unit (120), the search module (131) uses a dynamic programming approach to find the optimal FLC. The following [Algorithm 1] is pseudocode describing the process of determining the optimal FLC.
[0095] [Algorithm 1]
[0096]
[0098] [Algorithm 1] calculates the minimum cost using a specified number of layers N. The optimal FLC is constructed by determining effective fusion points and counting the number of layers for each fusion group.
[0099] However, the FLC obtained through the predicted cost matrix is not always guaranteed to be the optimal FLC. To compensate for this error, the present invention uses a wild card module (132) that provides additional fusion opportunities. Sometimes, additional fusion is impossible due to small differences between the total cost values.
[0100] For example, if an optimal FLC obtained through the cost matrix (f0, f1, f2) exists, F0 and f1 of C[0, n] = T[f0] + T[f1] + T[f2] cannot be fused because the value of T[f0] + T[f1] is smaller than T[f0∪ f1]. If the difference between F0 and f1 is very small, the present invention provides an opportunity to fuse once again, considering the possibility of further performance improvement.
[0101] The present invention refers to a group of fusion layers that have the potential to be further fused as a wild card W, which is calculated by [Equation 5].
[0103] [Mathematical Formula 5]
[0104]
[0106] f i : Fusion layer group of index i,
[0107] f i+1 : Fusion layer group of index i+1,
[0108] T[f i ]: Fusion layer group f i The total cost of,
[0109] n f : Number of fusion layer groups in a specific FLC
[0112] Accordingly, the present invention uses a wildcard module (132) corresponding to the minimum difference value obtained by [Equation 5] f i and f i+1 Select the best FLC and the next best FLC with wildcards applied, and select the more effective FLC as the optimal FLC.
[0114] FIG. 4 is a diagram illustrating an example of evaluating the total cost when layers are fused according to an embodiment of the present invention. The embodiment of FIG. 4 shows part of the process of constructing and filling in a cost matrix to search for an optimal fusion layer configuration. That is, FIG. 4 is an example showing the total cost evaluation when fusing a layer (l1) corresponding to index 1 and a layer (l2) corresponding to index 2.
[0115] Referring to FIG. 4, the cost matrix generation module (123) records the cost value required to perform fusion from layer n to layer m when sequentially numbering each layer of the target deep learning model from 0, at the n row and m column position of the cost matrix. The cost matrix generation module (123) repeats the above process to fill all the values on the upper downward diagonal of the cost matrix.
[0116] Previously, to fill in the values within the cost matrix, a large amount of data had to be prepared in advance and used for regression models or profiled individually. However, in the present invention, only the process of filling in the diagonal values was actually profiled, and the values above it were entered as predicted results using an appropriate formula.
[0117] For example, the cost matrix generation module (123) uses [Equation 6] to fill the values in row 1, column 2 (T[1,2]) highlighted in dark color in the cost matrix of FIG. 3.
[0119] [Mathematical Formula 6]
[0120]
[0122] Sending cost: Time taken to send data from the main device
[0123] execution cost: time taken to perform operations on the device,
[0124] receiving cost: the time to retrieve the calculation result from the device back to the main device,
[0125] n d : Number of devices in use
[0126] A d [1, 2]: Values listed in row 1, column 2 of the alpha matrix in device d, overhead for connecting the layer (l1) of index 1 and the layer (l2) of index 2,
[0127] A d [2, 2]: A value listed in row 2, column 2 of the alpha matrix in device d, an overhead for connecting the layer (l2) of index 2 and the layer (l2) of index 2,
[0128] E d [1, 1]: A value listed in row 1, column 1 of the run matrix on device d, representing the execution cost of the fusion block from layer (l1) of index 1 to layer (l1) of index 1,
[0129] E d [2, 2]: The value listed in row 2, column 2 of the run matrix on device d, the execution cost of the fusion block from l2 to l2,
[0130] R d [2]: Data reception cost of l2 on device d,
[0132] That is, to fill the value (T[1,2]) of the first row and second column of the cost matrix in Fig. 3, the data consists of the sum of the time sent from the main device (sending cost), the time to perform the operation at the device (execution cost), and the time to bring the result of the operation back from the device to the main device (receiving cost).
[0133] At this time, the cost matrix generation module (123) calculates each value of the diagonal position in the cost matrix and calculates the three time required at each device recorded during profiling by appropriately taking them from the set matrix, run matrix, and get matrix, multiplying them by the ratio of the data size increased by layer fusion (i.e., the value stored in the alpha matrix) and summing them.
[0134] At this time, the sending cost from the main device is the value recorded in the first column (Sd[1]) of the set matrix and the value in the first row and second column (A) of the alpha matrix. d It is calculated based on the product of the values listed in [1, 2]).
[0135] In addition, the execution cost on the device is the value recorded in row 1, column 1 (Ed[1, 1]) of the run matrix and row 1, column 2 (A d The product of the values listed in [1, 2]) and the 2nd row and 2nd column of the run matrix (E d The value listed in [2, 2]) and the 2nd row and 2nd column of the alpha matrix (A d It is calculated based on the sum of the products of the values listed in [2, 2]).
[0136] In addition, the time to bring the calculation result back to the main machine (receiving cost) is the 2nd row and 2nd column of the alpha matrix (A d The value listed in [2, 2]) and the second (R) of the run matrix d It is calculated based on the product of the values stored in [2]).
[0137] As described above, once the cost matrix is fully completed in this manner, the combination expected to have the lowest sum of the estimated time required for the fused layers from the beginning to the last layer is adopted as the optimal configuration of the fused layer.
[0139] FIG. 5 is a diagram showing the predicted values of a cost matrix according to one embodiment of the present invention.
[0140] Figure 5(a) is a layer-by-layer operation and can be calculated as the cost T[i, i] for index i of each layer. However, as shown in Figure 5(b), when merging layers, the amount of computation and data communication costs increase because the OLSF increases when merging multiple layers.
[0141] Accordingly, the cost matrix generation module (123) can roughly predict the cost generated by the fusion of layers by multiplying the values of S and E by the corresponding alpha value of A. If the cost matrix generation module (123) updates the cost of all elements of the cost matrix, it can find the path where C[i, j] is minimized. At this time, the path where C[i, j] is minimized becomes the optimal FLC.
[0143] Although the present invention has been described by limited embodiments and drawings, it is not limited to the above embodiments, and various modifications and variations are possible from this description by those skilled in the art. Accordingly, the concept of the present invention should be understood only by the claims set forth below, and all equivalent or analogous variations thereof shall be considered to be within the scope of the concept of the present invention. Explanation of the symbols
[0145] 110: Preprocessing section, 120: Cost forecasting unit, 121: Code compilation module, 122: Runtime evaluation module, 123: Cost Matrix Generation Module, 130: FLC Optimization Unit, 131: Search module, 132: Wild Card Module
Claims
Claim 1 An apparatus for performing a method for searching for a fused layer configuration using fused layer-wise parallelization in a heterogeneous computing environment, characterized by comprising: a preprocessing unit that specifies the number of an edge device to be used in an edge device pool among a plurality of devices and extracts layer functions required for partition execution for the device; a cost prediction unit that generates a fused layer configuration (FLC) using layer functions required for partition execution received from the preprocessing unit, performs a performance evaluation on an execution graph corresponding to the FLC, stores the profile results of the performance evaluation in different matrices, generates an alpha matrix based on the layer functions required for partition execution, and generates a cost matrix using the different matrices and the alpha matrix; and an FLC search unit that searches for and provides a fused layer configuration using the cost matrix. Claim 2 An apparatus for performing a method for searching for a fusion layer configuration using fusion layer-wise parallelization in a heterogeneous computing environment, wherein the cost prediction unit comprises: a code compilation module that generates a default FLC consisting of only a single layer and generates an execution graph corresponding to the FLC; a runtime evaluation module that receives the execution graph corresponding to the FLC, performs a performance evaluation of each device, and then stores the time of sending data in a set matrix, the time of operation in a run matrix, and the time of receiving data in a get matrix among the profile results of the performance evaluation; and a cost matrix generation module that generates a cost matrix using the set matrix, run matrix, get matrix, and alpha matrix. Claim 3 An apparatus for performing a method for searching for a fusion layer configuration using fusion layer-wise parallelization in a heterogeneous computing environment, characterized in that, in paragraph 2, the runtime evaluation module determines the value of the corresponding position as 1 if the row index and the column index of the alpha matrix are the same, and if the row index and the column index are different, determines the value of the corresponding position using the size of the input data increased when fusing from the row index layer to the column index layer and the existing input data size of the single layer to generate the alpha matrix. Claim 4 A device for performing a method for searching for a fusion layer configuration using fusion layer-wise parallelization in a heterogeneous computing environment, wherein, in paragraph 2, the cost matrix generation module calculates each value at a diagonal position in the cost matrix by multiplying the time for sending data from the main device in the set matrix, the time for performing operations at the corresponding device in the run matrix, and the time for bringing the operation result value to the main device in the get matrix by the value stored in the alpha matrix. Claim 5 A method for searching for a fused layer configuration using fused layer-wise parallelization in a heterogeneous computing environment, comprising: a step in which a preprocessor specifies the number of an edge device to be used in an edge device pool among a plurality of devices and extracts a layer function required for partition execution for the device; a step in which a cost prediction unit generates a fused layer configuration (FLC) using the layer function required for partition execution received from the preprocessor; a step in which the cost prediction unit performs a performance evaluation of an execution graph corresponding to the FLC and stores the profile result of the performance evaluation in different matrices; a step in which the cost prediction unit generates an alpha matrix based on the layer function required for partition execution and generates a cost matrix using the different matrix and the alpha matrix; and a step in which an FLC search unit searches for and provides a fused layer configuration using the cost matrix. Claim 6 In claim 5, the step of the cost matrix generation module of the cost prediction unit performing a performance evaluation of an execution graph corresponding to the FLC and then storing the profile results for the performance evaluation in different matrices comprises: a step in which, when an execution graph corresponding to a default FLC consisting of only a single layer is generated by the code compilation module of the cost prediction unit, the runtime evaluation module of the cost prediction unit performs a performance evaluation of the execution graph corresponding to the FLC and generates profile results for the performance evaluation; and a step in which the runtime evaluation module of the cost prediction unit stores the time of sending data among the profile results in a set matrix, stores the operation time in a run matrix, and stores the time of receiving data in a get matrix, respectively. This characterizes a method for searching for a fusion layer configuration using fusion layer-wise parallelization in a heterogeneous computing environment. Claim 7 In claim 6, the step in which the runtime evaluation module of the cost prediction unit stores the time of sending data among the profile results in a set matrix, the time of operation in a run matrix, and the time of receiving data in each get matrix is characterized by including the step of the cost matrix generation module of the cost prediction unit generating the alpha matrix by determining the value at the corresponding position as 1 if the row index and the column index of the alpha matrix are the same, and determining the value at the corresponding position using the size of the increased input data when fusing from the layer of the row index to the layer of the column index and the existing input data size of the single layer when fusing from the layer of the row index to the layer of the column index, thereby generating the alpha matrix. Claim 8 A method for searching for a fusion layer configuration using fusion layer-wise parallelization in a heterogeneous computing environment, wherein, in claim 6, the step of the cost matrix generation module of the cost prediction unit performing a performance evaluation of an execution graph corresponding to the FLC and then storing the profile results of the performance evaluation in different matrices includes the step of the cost matrix generation module of the cost prediction unit extracting the time for sending data from the main device in the set matrix, the time for performing operations at the corresponding device in the run matrix, and the time for bringing the operation result value to the main device in the get matrix, and then multiplying by the value stored in the alpha matrix.
Citation Information
Patent Citations
Method and system for opportunistic load balancing in neural networks using metadata.
KR1020210013707A
Multilayer artificial neural network performance evaluation apparatus including multiple sub models and method therefor
KR1020210027590A
Distributed processing method and apparatus of neural network model
KR1020210091947A
Method of dividing plurality of layers included in machine learning model and determining processor that performs divided layers, and device performing method
KR1020220025452A
System, Method, and Accelerator to Process Convolutional Neural Network Layers
US20190220734A1
