Layer integration determination device, method, and program

The described device and method automate layer integration in deep learning models by generating calculation graphs and extracting subgraphs that match predefined patterns, addressing the inefficiencies of conventional methods and enabling flexible support for various models, thus optimizing computational resources and reducing processing time.

WO2024247053A1PCT designated stage expired Publication Date: 2024-12-05NT T INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/019959
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-29
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Conventional methods for layer integration of deep learning models with multiple convolutional layers are time-consuming and inflexible, as they require detailed analysis and individual handling, failing to automate the integration process effectively.

Method used

A device and method that generate a calculation graph for deep learning models with multiple convolutional layers, integrating operations corresponding to each layer and extracting subgraphs that match predefined patterns, determining optimal layer integration intervals based on hardware specifications to automate the layer integration process.

Benefits of technology

This approach reduces the time required for layer integration determination and allows flexible support for various deep learning models, enabling efficient inference processing while optimizing computational resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023019959_05122024_PF_FP_ABST
    Figure JP2023019959_05122024_PF_FP_ABST
Patent Text Reader

Abstract

A layer integration determination device (10), wherein: a generation unit (21) generates a computation-integrated model by integrating computations corresponding to each of a convolution layer that can be processed by dedicated hardware that executes inference processing of a deep learning model and a peripheral layer accompanying the convolution layer, among computations corresponding to layers of a deep learning model which includes a plurality of convolution layers and which is input in the form of a calculation graph in which the layers are represented by nodes; an extraction unit (26) extracts a partial graph matching a pattern registered in advance in a pattern description unit (42) as a combination of convolution layers for which layer integration is possible, on the basis of the layer configuration of each of the convolution layers in the computation-integrated model; and a determination unit (28) determines, as a layer integration section, a section corresponding to a partial graph that satisfies a condition registered in advance in a condition description unit (44) as a condition for layer integration based on the specifications of the AI chip.
Need to check novelty before this filing date? Find Prior Art

Description

Layer integration determination device, method, and program

[0001] The disclosed technology relates to a layer merging determination device, a layer merging determination method, and a layer merging determination program.

[0002] Deep learning is a machine learning method that uses a neural network that replicates the mechanisms of human neurons, and is applied to fields such as video processing and natural language processing. For example, in the field of video processing, an image recognition method has been proposed that uses a deep learning model called a convolutional neural network (CNN) to determine the class of an object in an image. Many other methods have also been proposed using deep learning models, such as object detection that determines the position and class of an object in an image and segmentation that infers the class of an object for each pixel. Furthermore, in the field of natural language processing, methods using models equipped with attention mechanisms, such as transformers, have been applied to tasks such as machine translation and summarization. Such deep learning models have achieved performance that exceeds that of conventional machine learning models, and there is a movement to utilize deep learning in various fields such as medicine and industry.

[0003] Factors contributing to the improvement in the performance of such deep learning models include improvements in the computing power of computers and the development of cloud technology. For example, large-scale parallel computing has become possible by repurposing GPUs (Graphical Processing Units). Furthermore, the emergence of cloud services equipped with a large number of GPUs, such as GCP (Google Cloud Platform) and AWS (Amazon Web Service), has made it easier to train large-scale deep learning models.

[0004] When using a trained model to perform inference processing on data acquired by a device such as an in-vehicle camera or a smartphone, there are two methods: a method of performing inference on the cloud, and a method of performing inference on the device that acquired the data. In the former method, real-time performance is impaired due to network delays, and transmitting data to the cloud via the Internet may pose security risks or privacy violations. Therefore, in recent years, attention has been focused on edge AI (Artificial Intelligence), which performs inference processing on the device that acquired the data.

[0005] While edge AI enables real-time data processing while protecting privacy, it has faced the challenge of securing power and computing resources. In particular, when performing inference processing on mobile devices such as drones and smartphones, weight restrictions make it difficult to equip them with devices that consume large amounts of power, such as GPUs. Therefore, by installing specialized hardware for inference processing, known as an AI chip, on the device, it is possible to reduce power consumption while securing the computing resources required for edge AI.

[0006] Furthermore, to efficiently utilize limited computing resources, a technique called layer merging has been introduced, which integrates the operations of multiple layers included in a deep learning model (Non-Patent Document 1). Layer merging integrates multiple convolutional layers in a deep learning model, rather than processing each convolutional operation independently. This enables computational processing using a small cache, thereby reducing access to external memory, shortening processing time, and improving power efficiency. A method for automating this type of computational integration has also been proposed (Non-Patent Document 2).

[0007] F. Indirli, A. Erdem and C. Silvano, "A Tile-based Fused-layer CNN Accelerator for FPGAs," 2020 27th IEEE International Conference on Electronics, Circuits and Systems (ICECS), Glasgow, UK, 2020, pp. 1-4. Zhi Chen and Cody Yu, "How to Bring Your Own Codegen to TVM," [online], July 15, 2020, Amazon Web Services, Inc. [Retrieved April 12, 2023], Internet <URL: https: / / tvm.apache.org / 2020 / 07 / 15 / how-to-bring-your-own-codegen-to-tvm>

[0008] While conventional methods can automate the integration of primitive operations in deep learning models, they do not consider layer integration for deep learning models that contain multiple convolutional layers. Therefore, when integrating layers of deep learning models that contain multiple convolutional layers using conventional methods, it is necessary to analyze the target deep learning model in detail and make layer integration decisions for each deep learning model individually, which poses the issue of time-consuming layer integration decisions.

[0009] The disclosed technology has been developed in consideration of the above points, and aims to flexibly support various deep learning models while reducing the time required to determine layer integration for deep learning models that include multiple convolutional layers.

[0010] A first aspect of the present disclosure is a layer-integration determination device that includes: a generation unit that generates a computation-integrated model by integrating computations corresponding to each of the convolutional layers and peripheral layers associated with the convolutional layers that can be processed by dedicated hardware that executes inference processing of the deep learning model, out of computations corresponding to each layer of a deep learning model including a plurality of convolutional layers that are input in the form of a computation graph in which each layer is represented by a node; an extraction unit that extracts, from the computation-integrated model, a subgraph that matches a pattern that is registered in advance in a pattern description unit as a combination of convolutional layers that can be layer-integrated based on the layer configuration of each of the convolutional layers; and a determination unit that determines, as a layer-integration section, a section corresponding to the subgraph that satisfies a condition that is registered in advance in a condition description unit as a condition for layer integration based on the specifications of the dedicated hardware.

[0011] A second aspect of the present disclosure is a layer-integration determination method executed by a layer-integration determination device including a generation unit, an extraction unit, and a determination unit, in which the generation unit integrates, among operations corresponding to each layer of a deep learning model including a plurality of convolutional layers, input in the form of a computation graph in which each layer is represented by a node, operations corresponding to each of the convolutional layers that can be processed by dedicated hardware that executes inference processing of the deep learning model and surrounding layers associated with the convolutional layers, to generate a computation-integrated model, the extraction unit extracts, from the computation-integrated model, a subgraph that matches a pattern that is registered in advance in a pattern description unit as a combination of convolutional layers that can be layer-integrated based on the layer configuration of each of the convolutional layers, and the determination unit determines, as a layer-integration section, a section corresponding to the subgraph that satisfies a condition that is registered in advance in a condition description unit as a condition for layer integrating based on the specifications of the dedicated hardware.

[0012] A third aspect of the present disclosure is a layer integration determination program, which is a program for causing a computer to function as each part of the layer integration determination device described above.

[0013] The disclosed technology can flexibly accommodate a variety of deep learning models while reducing the time required to determine layer merging for deep learning models that include multiple convolutional layers.

[0014] 1 is a block diagram showing the configuration of a computation merging device according to a conventional method; FIG. 2 is a diagram for explaining layers included in a deep learning model; FIG. 3 is a flowchart showing the flow of computation merging processing according to a conventional method; FIG. 4 is a block diagram showing the hardware configuration of a layer merging determination device according to the first and second embodiments; FIG. 5 is a block diagram showing an example of the functional configuration of a layer merging determination device according to the first embodiment; FIG. 6 is a diagram showing an example of a pattern registered in a pattern description unit; FIG. 7 is a diagram showing another example of a pattern registered in a pattern description unit; FIG. 8 is a flowchart showing the flow of layer merging determination processing according to the first embodiment; FIG. 9 is a flowchart showing the flow of layer-merged model candidate generation processing; FIG. 10 is a block diagram showing an example of the functional configuration of a layer merging determination device according to the second embodiment;

[0015] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0016] <Regarding Conventional Methods> First, before describing the details of each embodiment, conventional methods will be described.

[0017] FIG. 1 shows the configuration of a computation integration device 1000 of a conventional method for automating the integration of computations in a deep learning model, such as the method described in Non-Patent Document 2.

[0018] A deep learning model is input to the computation integration device 1000 in the form of a computation graph in which each layer of the deep learning model is represented by a node. The computation integration device 1000 scans the input computation graph, performs processing to integrate layers corresponding to integrable operations, and outputs a computation-integrated model. As shown in FIG. 1 , the computation integration device 1000 functionally includes one description unit called a correspondence description unit 1040, and two processing units, a labeling unit 1022 and a computation integration unit 1024.

[0019] Generally, when inference processing of a machine learning model is performed using hardware, computational efficiency is improved by integrating and executing the computation of a convolutional (Conv) layer and the computation of each of the surrounding layers associated with the convolutional layer, as shown in Figure 2. In the example of Figure 2, the surrounding layers associated with the convolutional layer are a padding (Pad) layer, a batch normalization (BN) layer, and an activation function (ReLU) layer.

[0020] Therefore, the correspondence description unit 1040 registers, among the layers of the deep learning model, operations that can be processed by dedicated hardware (hereinafter, also referred to as an "AI chip" as an example) that executes the inference processing of the deep learning model, and combinations of operations that can be integrated. Operations that can be processed by an AI chip are operations for which a dedicated circuit required to execute the processing of that operation is implemented on the AI ​​chip. Combinations of operations that can be integrated are combinations of primitive operations that can be integrated, such as the combination of an operation in a convolution layer and an operation in each of the surrounding layers associated with that operation, as described above.

[0021] The labeling unit 1022 individually determines whether the calculations in each layer of the deep learning model can be processed on an AI chip and labels the layers of calculations to be processed on the AI ​​chip so that they can be identified. Specifically, the labeling unit 1022 determines whether each calculation matches the calculations that can be processed on the AI ​​chip described in the correspondence description unit 1040. The labeling unit 1022 labels the layers of calculations that can be processed on the AI ​​chip with a label indicating that they will be processed on the AI ​​chip, and labels the other layers with a label indicating that they will be processed on general-purpose hardware (e.g., a CPU, a GPU, etc.) for controlling the AI ​​chip, etc. The labeling unit 1022 passes the labeled model, in which each layer of the deep learning model (each node of the computation graph) has been labeled, to the calculation integration unit 1024.

[0022] The operation integration unit 1024 integrates into one layer layers in the labeled model that correspond to combinations of operations in layers labeled to be processed by an AI chip and that match combinations of operations described in the correspondence description unit 1040. The operation integration unit 1024 thereby reconstructs a computation graph representing the deep learning model, and generates and outputs a computation-integrated model.

[0023] FIG. 3 is a flowchart showing the flow of the operation integration process executed by the operation integration device 1000 of the conventional method.

[0024] When a deep learning model represented by a computation graph is input to the computation integration device 1000, the deep learning model is scanned and the loop processing of step S1000 is executed, with each layer (each node of the computation graph) included in the deep learning model as the processing target. Specifically, in step S1002, the labeling unit 1022 determines whether the computation of the processing target layer can be processed on the AI ​​chip based on the computations that can be processed by the AI ​​chip and are registered in the correspondence description unit 1040. If the computation can be processed, the process proceeds to step S1004, and if the computation cannot be processed, the process proceeds to step S1006.

[0025] In step S1004, the labeling unit 1022 attaches a label to the layer to be processed indicating that it will be processed by an AI chip, while in step S1006, the labeling unit 1022 attaches a label to the layer to be processed indicating that it will be processed by general-purpose hardware.

[0026] Next, in step S1008, the labeling unit 1022 determines whether scanning of the deep learning model has finished, i.e., whether labeling processing has finished for all layers included in the deep learning model. If scanning has finished, the process proceeds to step S1010. If scanning has not finished, the loop processing of step S1000 is repeated. In step S1010, the computation integration unit 1024 reconstructs the computation graph by grouping together sections of consecutive layers labeled to be processed by the AI ​​chip.

[0027] Then, while scanning the deep learning model, the loop processing of step S1012 is executed with each section as the processing target. Specifically, in step S1014, the operation integrator 1024 performs pattern matching between the combination of operations of each layer in the processing target section and the combination of primitive operations that can be integrated and are registered in the correspondence description unit 1040. The operation integrator 1024 determines whether the layers in the processing target section can be integrated based on whether the combination of operations matches through pattern matching. If integration is possible, the process proceeds to step S1016; if integration is not possible, the process proceeds to step S1018.

[0028] In step S1016, layer combinations corresponding to operation combinations that match through pattern matching are integrated by replacing them with one layer, and the computation graph is reconstructed. Next, in step S1018, the computation integration unit 1024 determines whether scanning of the deep learning model is complete, i.e., whether the process of determining whether to integrate layers within all sections included in the deep learning model is complete. If scanning is complete, the process proceeds to step S1020. If scanning is not complete, the loop process of step S1012 is repeated until there are no more sections that can be integrated. In step S1020, the computation integration unit 1024 outputs the computation graph finally reconstructed in step S1016 as a computation-integrated model, and the computation integration process ends.

[0029] As mentioned above, conventional methods can automate the integration of primitive operations in deep learning models, but do not consider layer integration of deep learning models that include multiple convolutional layers. When multiple convolutional layers are included, the network structure differs for each deep learning model. Therefore, in order to integrate deep learning models that include multiple convolutional layers using conventional methods, it was necessary to analyze the deep learning model to be integrated in detail and address each model individually.

[0030] Specifically, conventional methods require determining the combination of convolutional layers to be integrated based on the AI ​​chip's specifications and the constraint of the maximum number of convolutional layers that can be integrated on the AI ​​chip. Furthermore, conventional methods require checking whether the kernel size, input / output data size, etc., within the layer integration interval for each combination of convolutional layers to be integrated are equal to or smaller than the cache capacity of the AI ​​chip. Therefore, conventional methods have the problem of being unable to flexibly accommodate layer integration for various deep learning models.

[0031] Therefore, in the following embodiments, a method is proposed for automating the extraction of layer integration sections that satisfy constraints based on the specifications of an AI chip and the determination of the optimal layer integration section for any input deep learning model.

[0032] 4 is a block diagram showing the hardware configuration of a layer integration determination device 10 according to the first embodiment. As shown in Fig. 4, the layer integration determination device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication I / F (Interface) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0033] The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a layer integration determination program, which will be described later.

[0034] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured by a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

[0035] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information. The display unit 16 is, for example, a liquid crystal display, and displays various types of information. The display unit 16 may also function as the input unit 15 by employing a touch panel system.

[0036] The communication I / F 17 is an interface for communicating with other devices, and the communication may use a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark).

[0037] Next, the functional configuration of the layer integration determination device 10 according to the first embodiment will be described. FIG. 5 is a block diagram showing an example of the functional configuration of the layer integration determination device 10. As shown in FIG. 5, the layer integration determination device 10 includes, as its functional configuration, a generation unit 21, an extraction unit 26, a determination unit 28, and an optimization unit 30. In addition, a predetermined storage area of ​​the layer integration determination device 10 is provided with a correspondence description unit 40, a pattern description unit 42, and a condition description unit 44. Each functional configuration is realized when the CPU 11 reads out a layer integration determination program stored in the ROM 12 or the storage 14, expands it in the RAM 13, and executes it.

[0038] The layer merging determination device 10 receives a deep learning model to be processed by layer merging, the deep learning model including multiple convolution layers, in the form of a computation graph in which each layer, such as a convolution (Conv) layer and an activation function (Activation) layer, is represented by a node. Each node holds parameter information related to the layer corresponding to the node. Examples of parameters include the size of the input and output feature maps, the kernel size and number of channels used in the convolution operation, the number of multiplications of the convolution matrix operation and activation function, and the number of additions in the bias addition.

[0039] The correspondence description unit 40 is similar to the correspondence description unit 1040 of the conventional method described in Fig. 1. That is, the correspondence description unit 40 registers operations that can be processed by an AI chip and combinations of primitive operations that can be integrated.

[0040] The generation unit 21 integrates, among the operations corresponding to each layer of the input deep learning model, operations corresponding to the convolutional layer that can be processed by the AI ​​chip and the surrounding layers associated with the convolutional layer, to generate a computation-integrated model. Specifically, the generation unit 21 includes a labeling unit 22 and a computation integration unit 24.

[0041] The labeling unit 22 identifiably labels, among the layers of the input deep learning model, layers of operations that match operations that can be processed by the AI ​​chip and that are pre-registered in the correspondence description unit 40. Other specific details of the labeling unit 22 are similar to those of the labeling unit 1022 in Figure 1, so detailed description will be omitted. The labeling unit 22 passes the labeled model (computation graph) to the computation integration unit 24.

[0042] Based on the labeled model received from the labeling unit 22, the operation integration unit 24 identifies combinations of operations corresponding to layers labeled as layers of operations processable by the AI ​​chip. The operation integration unit 24 then aggregates, as processing blocks, combinations of operations that match combinations of primitive operations that can be integrated and are pre-registered in the correspondence description unit 40, among the identified combinations of operations, to reconstruct a computation graph. In this way, the operation integration unit 24 generates a operation-integrated model. Other specific details of the operation integration unit 24 are similar to those of the operation integration unit 1024 in FIG. 1 , and therefore detailed description thereof will be omitted. The operation integration unit 24 passes the generated operation-integrated model to the extraction unit 26.

[0043] Patterns of combinations of convolutional layers that can be merged based on the layer configuration of each convolutional layer are registered in the pattern description unit 42. Specifically, the pattern description unit 42 comprehensively registers in advance subgraph patterns of deep learning models that satisfy constraints that take into account the "number of convolutional layers that can be merged" and the "connections between layers" determined by the specifications of the AI ​​chip.

[0044] FIG. 6 shows an example of a subgraph pattern. FIG. 6( a) is a computation graph representing a combination of layers of integrable operations registered in the correspondence description unit 40 shown in FIG. 2. This is treated as one processing block (Conv Block), and a pattern combining multiple Conv Blocks, as shown in FIG. 6( b), is registered in the pattern description unit 42. As shown in FIG. 6( b), the pattern may also include other operators such as an upsampling layer. As shown in FIG. 7( c), the pattern may also be a more complex pattern including a layer such as a residual layer (Res) that combines a skip connection and an addition layer.

[0045] Conditions for layer integration based on the specifications of the AI ​​chip are registered in the condition description unit 44. For example, a conditional expression for the cache usage on the AI ​​chip required to read the kernels of the convolution layer is defined in advance based on the parameters of each node, as in the following equation (1):

[0046]

[0047] In equation (1), the layer integration section is the lth s Layer 1 e layer, k a is the kernel size of the a-th convolution layer, iCH a is the number of input channels of the a-th layer, oCH a is the number of channels of the output of the a-th layer. cache represents the cache capacity, and is a value determined according to the specifications of the AI ​​chip.

[0048] The extraction unit 26 extracts, as a subgraph, a portion of the computation-integrated model (computation graph) passed from the computation integration unit 24 that matches a pattern registered in advance in the pattern description unit 42. Specifically, the extraction unit 26 scans the computation graph indicating the computation-integrated model, matches it with the pattern registered in the pattern description unit 42, and extracts an interval where the pattern matches as a subgraph that is a candidate for a layer-integration interval. The extraction unit 26 passes the extracted subgraph to the determination unit 28.

[0049] The determination unit 28 determines, from among the subgraphs passed from the extraction unit 26, a section corresponding to a subgraph that satisfies the conditions previously registered in the condition description unit 44 as a layer-integrated section. Specifically, the determination unit 28 extracts parameters of each node in the subgraph and reads out the conditional expression registered in the condition description unit 44. The determination unit 28 then calculates the read-out conditional expression using the extracted parameters and determines whether the subgraph satisfies the conditional expression. The determination unit 28 generates a layer-integrated model candidate by integrating each layer in the layer-integrated section that satisfies the condition and reconstructing the computationally integrated model. The determination unit 28 passes the generated layer-integrated model candidate to the optimization unit 30.

[0050] The optimization unit 30 selects and outputs the optimal candidate as the layer-merged model based on an optimization index from among multiple layer-merged model candidates, each with a different layer-merged interval, generated by multiple executions of the extraction unit 26 and the determination unit 28. The optimization index may be any, but is preferably a numerical value representing the processing time of each layer-merged interval, specifically, an index using at least one of the computational complexity of the deep learning model and the amount of data exchanged between the external memory and the AI ​​chip. For example, the optimization unit 30 calculates the sum of the amount of product-sum computations for each layer-merged interval, the variance of the read / write time of the convolution kernel and input / output features, etc., as the optimization index. The optimization unit 30 selects and outputs the layer-merged model candidate with the smallest variance as the final layer-merged model. This selects the candidate with the most uniform processing time for each layer-merged interval included in the layer-merged model candidate. This reduces processing delays due to waiting for the completion of previous processing when configuring pipeline processing, thereby enabling efficient inference processing.

[0051] Next, the operation of the layer integration determination device 10 according to the first embodiment will be described. Fig. 8 is a flowchart showing the flow of the layer integration determination process performed by the layer integration determination device 10. The layer integration determination process is performed by the CPU 11 reading out a layer integration determination program from the ROM 12 or the storage 14, expanding it into the RAM 13, and executing it. The layer integration determination process is an example of a layer integration determination method according to the present disclosure.

[0052] In step S10, the CPU 11, functioning as the generation unit 21, executes a computation integration process. The computation integration process is similar to the computation integration process of the conventional method shown in Fig. 3. Next, in step S12, a loop process is executed in which the process is repeated a specified number of times. The specified number of times of the repetition process is externally specified as a hyperparameter.

[0053] Specifically, in step S14, the CPU 11 functions as the extraction unit 26 to change the order of patterns to be matched for the patterns registered in the pattern description unit 42. The order of the patterns can be set arbitrarily; for example, the order may be changed randomly, or the order may be changed regularly by grouping the patterns by the number of convolutional layers in the patterns. This is processing for generating different layer-integrated model candidates for each iteration in the processing of step S16 described below.

[0054] Next, in step S16, the CPU 11 executes a layer-integrated model candidate generation process. Next, in step S18, the CPU 11, functioning as the optimization unit 30, determines whether the repetition process has reached a designated number of times. If the repetition process has reached the designated number of times, the process proceeds to step S20. If the repetition process has not reached the designated number of times, the process repeats the loop process of step S12.

[0055] Here, the layer-integrated model candidate generation process executed in step S16 will be described with reference to FIG.

[0056] The CPU 11 sets the patterns registered in the pattern description unit 42 as patterns to be processed in the order changed in step S14, and executes the loop process of step S160. Specifically, the CPU 11 scans the computation-integrated model and executes the loop process of step S162.

[0057] In step S164, the CPU 11, functioning as the extraction unit 26, determines whether or not there are any unintegrated sections, i.e., sections other than the layer-integrated sections extracted based on patterns other than the pattern to be processed. If there are any unintegrated sections remaining, the process proceeds to step S166; if there are no unintegrated sections remaining, the process proceeds to step S176.

[0058] In step S166, the CPU 11, functioning as the extraction unit 26, searches for a subgraph that matches the pattern to be processed in the unintegrated section of the computationally integrated model (computation graph). The CPU 11, functioning as the extraction unit 26, determines whether or not a subgraph that matches the pattern to be processed exists. If a subgraph exists, the process proceeds to step S168; if not, the loop process of step S162 ends.

[0059] In step S168, the CPU 11, functioning as the extraction unit 26, extracts a subgraph that matches the pattern to be processed. Then, the CPU 11, functioning as the determination unit 28, extracts parameters for each node in the subgraph and reads out a conditional expression registered in the condition description unit 44. Furthermore, the CPU 11, functioning as the determination unit 28, calculates the read-out conditional expression using the extracted parameters and determines whether the subgraph satisfies the corresponding conditional expression. If the subgraph satisfies the condition, the process proceeds to step S170; if the subgraph does not satisfy the condition, the loop process of step S162 ends.

[0060] In step S170, the CPU 11, functioning as the extraction unit 26, determines whether scanning of the entire computation-integrated model has been completed. If scanning has not been completed, the loop processing of step S162 is repeated, and if scanning has been completed, the process proceeds to step S174. In step S174, the CPU 11, functioning as the extraction unit 26, determines whether matching processing with the computation-integrated model has been completed for all patterns registered in the pattern description unit 42. If unprocessed patterns exist, the loop processing of step S160 is repeated for the unprocessed patterns, and if matching has been completed for all patterns, the process proceeds to step S176.

[0061] In step S176, there are no unintegrated sections remaining in the computationally integrated model, or the matching process for all patterns has been completed. The CPU 11, as the determination unit 28, generates a layer-integrated model candidate by integrating each layer in the layer-integrated section extracted in step S170 to reconstruct the computationally integrated model. Note that layers labeled to be processed by the AI ​​chip but not included in any layer-integrated section are treated as single layers and are output together with each layer-integrated section as part of the layer-integrated model candidate. The layer-integrated model candidate generation process then ends, and the process returns to the layer-integration determination process ( FIG. 8 ). The loop process (step S12) including the layer-integrated model candidate generation process in step S16 is repeated a specified number of times, thereby generating a specified number of layer-integrated model candidates.

[0062] Next, in step S20, the CPU 11 executes the loop process of step S20 for each generated layer-integrated model candidate as the optimization unit 30. Specifically, in step S22, the CPU 11 executes the loop process of step S20 for each generated layer-integrated model candidate as the optimization unit 30. Specifically, in step S22, the CPU 11 calculates, as an optimization index, the variance of an index indicating the processing time of each layer-integrated section included in the layer-integrated model candidate to be processed.

[0063] Next, in step S24, the CPU 11 determines whether or not the process of calculating the optimization indexes for all layer-integrated model candidates has been completed as the optimization unit 30. If there are any unprocessed layer-integrated model candidates, the loop process of step S20 is repeated, and if all have been completed, the process proceeds to step S26.

[0064] In step S26, the CPU 11, as the optimization unit 30, selects an optimal layer-integrated model from the layer-integrated model candidates based on the optimization index. For example, the CPU 11, as the optimization unit 30, selects and outputs the layer-integrated model candidate that has the smallest variance of the index representing the processing time for each layer-integrated section calculated as the optimization index, as the final layer-integrated model. Then, the layer-integration determination process ends.

[0065] As described above, the layer-merging determination device according to the first embodiment integrates, among the operations corresponding to each layer of a deep learning model including multiple convolutional layers, operations corresponding to convolutional layers and surrounding layers associated with the convolutional layers that can be processed by dedicated hardware that executes the inference processing of the deep learning model, to generate a computationally integrated model. From the computationally integrated model, the device extracts subgraphs that match patterns pre-registered in a pattern description unit as combinations of convolutional layers that can be merged based on the layer configuration of each of the convolutional layers. The device determines, as a layer-merging section, a section corresponding to a subgraph that satisfies a condition pre-registered in a condition description unit as a condition for layer merging based on the specifications of the dedicated hardware. The device then selects and outputs, based on a predetermined index, an optimal layer-merged model from among multiple candidate layer-merged models with different layer-merged sections. This allows the device to flexibly accommodate various deep learning models while shortening the time required to determine whether a deep learning model including multiple convolutional layers should be merged, using only the deep learning model represented as a computational graph as input.

[0066] Second Embodiment In the second embodiment, a deep learning model is scanned in advance to generate conditions to be used in optimization as additional conditional expressions, which are then passed to the determination unit, thereby enabling the extraction unit and the determination unit to perform their processing in a single step. In the layer integration determination device according to the second embodiment, components similar to those of the layer integration determination device 10 according to the first embodiment are designated by the same reference numerals, and detailed descriptions thereof will be omitted. Furthermore, the hardware configuration of the layer integration determination device according to the second embodiment is similar to the hardware configuration of the layer integration determination device 10 according to the first embodiment shown in FIG. 4 , and therefore description thereof will be omitted.

[0067] The functional configuration of a layer integration determination device 210 according to the second embodiment will be described. FIG. 10 is a block diagram showing an example of the functional configuration of the layer integration determination device 210. As shown in FIG. 10, the layer integration determination device 210 includes, as its functional configuration, a generation unit 21 including a labeling unit 22 and a calculation integration unit 24, an extraction unit 26, a determination unit 228, and an addition unit 232. In addition, a predetermined storage area of ​​the layer integration determination device 210 is provided with a correspondence description unit 40, a pattern description unit 42, and a condition description unit 44. Each functional configuration is realized when the CPU 11 reads out a layer integration determination program stored in the ROM 12 or the storage 14, expands it in the RAM 13, and executes it.

[0068] The adding unit 232 adds an optimization condition based on the predetermined number of layer-integrated sections to the conditions used by the determining unit 228. For example, the adding unit 232 adds, as a condition, a range of the amount of calculation for each layer-integrated section that equalizes the processing time for each layer-integrated section included in the layer-integrated model, using at least one of the amount of calculation for the deep learning model and the amount of data exchanged between the external memory and the AI ​​chip.

[0069] More specifically, the adding unit 232 acquires the deep learning model (computation graph) input to the layer integration determination device 210, and scans the computation graph to estimate the computational complexity of the entire deep learning model. The adding unit 232 estimates the computational complexity from parameters of the deep learning model, including the sizes of the input and output feature maps, the kernel size used in the convolution computation, the number of channels, the convolution matrix computation and the number of multiplications of the activation function, the number of additions for bias addition, etc. As described above, these parameters are held by each node in the computation graph.

[0070] The adding unit 232 sets upper and lower limits on the amount of computation per layer-integrated section from the estimated amount of computation for the entire learning model. Specifically, the adding unit 232 calculates the amount of computation per layer-integrated section by dividing the amount of computation for the entire deep learning model by the expected number of layer-integrated sections. The expected number of layer-integrated sections may be manually provided as a hyperparameter, or may be estimated mechanically. When estimating mechanically, for example, a layer-integrated model is generated without providing any additional conditional expressions from the adding unit 232, and the number of layer-integrated sections included in the generated layer-integrated model is used. Then, the adding unit 232 sets the upper and lower limits based on the amount of computation per layer-integrated section.

[0071] The adding unit 232 passes the formula used to calculate the amount of calculation and an additional condition formula indicating the range of the amount of calculation per one layer integration section to the determining unit 228. The additional condition formula is written, for example, as in the following formula (2).

[0072]

[0073] In equation (2), the layer integration section is the lth s Layer 1 e layer, m a is the number of multiplications of the a-th layer, which is a convolutional layer or an activation function layer (type = conv, activation), C min is the lower limit of the amount of computation, C max is the upper limit of the amount of calculation. Equation (2) is a conditional decision equation that determines whether the sum of the number of multiplications in the convolution layer and the activation function layer is within the range from the upper limit to the lower limit.

[0074] The additional condition is not limited to a condition related to the amount of computation of the deep learning model, but may be a condition using the amount of data exchanged with an external memory, a combination of these, or the like. Furthermore, in equation (2), an example of an additional condition formula is shown in which an upper and lower limit are set, but an additional condition formula in which only an upper limit is set may also be used. For example, an upper limit on the amount of computation per layer integration section may be set based on the computing power of the AI ​​chip and the processing time of one stage of pipeline processing.

[0075] In addition, if a conditional expression having the same calculation method as the created additional conditional expression is registered in the condition description section 44, the adding section 232 may pass the additional conditional expression to the judgment section 228 by overwriting the conditional expression registered in the condition description section 44 with the additional conditional expression.

[0076] The determination unit 228 determines whether the subgraph extracted by the extraction unit 26 satisfies a condition, similar to the determination unit 28 in the first embodiment. However, the determination unit 228 in the second embodiment determines whether the subgraph satisfies an optimization condition based on the number of layer integration sections passed from the adding unit 232, along with a layer integration condition based on the AI ​​chip specifications registered in the condition description unit 44. The determination unit 228 generates and outputs a layer-integrated model obtained by reconstructing the computationally integrated model by integrating the layers in the layer integration sections that satisfy the condition.

[0077] Next, a description will be given of the operation of the layer merging determination device 210 according to the second embodiment. Fig. 11 is a flowchart showing the flow of the layer merging determination process performed by the layer merging determination device 210. The layer merging determination process is performed by the CPU 11 reading out a layer merging determination program from the ROM 12 or the storage 14, loading it into the RAM 13, and executing it.

[0078] In step S210, the CPU 11, functioning as the adding unit 232, determines the expected number of layer-integration sections, for example, by acquiring manually assigned hyperparameters or by mechanically estimating them. Next, in step S212, the CPU 11, functioning as the adding unit 232, acquires the deep learning model (computation graph) input to the layer-integration determination device 210, scans the computation graph, and estimates the amount of computation for the entire deep learning model from the parameters of the deep learning model.

[0079] Next, in step S214, the CPU 11, as the adding unit 232, calculates the amount of computation per layer integration section by dividing the amount of computation for the entire deep learning model by the expected number of layer integration sections. Then, the CPU 11, as the adding unit 232, sets a range (e.g., upper and lower limits) of the amount of computation per layer integration section based on the calculated amount of computation per layer integration section, and creates an additional conditional expression. Next, in step S216, the CPU 11, as the adding unit 232, passes the expression used to calculate the amount of computation and the created additional conditional expression to the determining unit 228.

[0080] Next, in step S10, the CPU 11, functioning as the generation unit 21, executes a computation integration process. The computation integration process is similar to the computation integration process of the conventional method shown in FIG. 3 . Next, in step S218, the CPU 11, functioning as the extraction unit 26 and the determination unit 228, executes a layer-integrated model generation process. The layer-integrated model generation process is similar to the layer-integrated model candidate generation process shown in FIG. 9 . However, when determining in step S168 whether the subgraph satisfies the condition, the CPU 11, functioning as the determination unit 228, determines whether the subgraph satisfies the condition registered in the condition description unit 44 as well as the condition passed from the adding unit 232. Furthermore, the model generated in step S176 is not a layer-integrated model candidate but a final layer-integrated model.

[0081] Next, in step S220, the determination unit 228 outputs the generated layer-merged model, and the layer-merging determination process ends.

[0082] As described above, the layer integration determination device according to the second embodiment generates a layer-integrated model by integrating layers within a layer integration interval that satisfies optimization conditions based on a predetermined number of layer integration intervals, such as the range of the amount of calculation per layer integration interval, along with layer integration conditions based on the AI ​​chip specifications registered in the condition description unit. This eliminates the need for the repeated processing of the extraction unit and determination unit in the first embodiment, and makes it possible to reduce the processing time for the layer integration determination process compared to the layer integration determination device according to the first embodiment.

[0083] In addition, the layer integration determination process executed by the CPU by reading software (programs) in each of the above embodiments may be executed by various processors other than the CPU. Examples of such processors include a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacture, such as an FPGA (Field-Programmable Gate Array), and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to execute specific processing. Furthermore, the layer integration determination process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0084] In addition, in each of the above embodiments, the layer integration determination program is described as being pre-stored (installed) in storage, but this is not limiting. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0085] The following additional notes are provided regarding the above-described embodiments.

[0086] (Supplementary Item 1) A layer integration determination device including: a memory; and at least one processor connected to the memory, wherein the processor is configured to: integrate operations corresponding to each layer of a deep learning model including a plurality of convolutional layers, the operations being input in the form of a computation graph in which each layer is represented by a node, and operations corresponding to each of the convolutional layers and peripheral layers associated with the convolutional layers that can be processed by dedicated hardware that executes inference processing of the deep learning model; extract, from the computation-integrated model, a subgraph that matches a pattern pre-registered in a pattern description unit as a combination of convolutional layers that can be layer-integrated based on the layer configuration of each of the convolutional layers; and determine, as a layer-integration section, a section corresponding to the subgraph that satisfies a condition pre-registered in a condition description unit as a condition for layer integration based on the specifications of the dedicated hardware.

[0087] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to execute a layer integration determination process, wherein the layer integration determination process: integrates, among operations corresponding to each layer of a deep learning model including a plurality of convolutional layers, input in the form of a computation graph in which each layer is represented by a node, operations corresponding to each of the convolutional layers and peripheral layers associated with the convolutional layers that can be processed by dedicated hardware that executes the inference process of the deep learning model, to generate a computation-integrated model; extracts, from the computation-integrated model, a subgraph that matches a pattern pre-registered in a pattern description section as a combination of convolutional layers that can be layer-integrated based on the layer configuration of each of the convolutional layers; and determines, as a layer-integration section, a section corresponding to the subgraph that satisfies a condition pre-registered in a condition description section as a condition for layer integration based on the specifications of the dedicated hardware.

[0088] REFERENCE SIGNS LIST 10, 210 Layer integration determination device 11 CPU 12 ROM 13 RAM 14 Storage 15 Input unit 16 Display unit 17 Communication I / F 19 Bus 21 Generation unit 22 Labeling unit 24 Operation integration unit 26 Extraction unit 28, 228 Determination unit 232 Addition unit 30 Optimization unit 40 Correspondence relationship description unit 42 Pattern description unit 44 Condition description unit

Claims

1. A layer integration determination device including: a generation unit that generates a computation-integrated model by integrating computations corresponding to each of the convolutional layers and surrounding layers associated with the convolutional layers that can be processed by dedicated hardware that executes inference processing of the deep learning model, among computations corresponding to each layer of a deep learning model including multiple convolutional layers, input in the form of a computation graph in which each layer is represented by a node; an extraction unit that extracts, from the computation-integrated model, a subgraph that matches a pattern that is pre-registered in a pattern description unit as a combination of convolutional layers that can be layer-integrated based on the layer configuration of each of the convolutional layers; and a determination unit that determines, as a layer integration section, a section corresponding to the subgraph that satisfies a condition that is pre-registered in a condition description unit as a condition for layer integration based on the specifications of the dedicated hardware.

2. The layer integration judgment device of claim 1, further comprising an optimization unit that selects an optimal layer integration model based on a predetermined index from among a plurality of layer integration model candidates each having a different layer integration section, which are generated by executing the processing of the extraction unit and the judgment unit multiple times.

3. The layer-merging determination device according to claim 2, wherein the optimization unit uses, as the specified index, at least one of the amount of calculation of the deep learning model and the amount of data exchanged between an external memory and the dedicated hardware, and selects, as the optimal layer-merged model, the layer-merged model in which the processing time of each layer-merged section included in the layer-merged model is most uniform.

4. The layer integration judgment device according to claim 1, further comprising an addition section which adds an optimization condition based on a predetermined number of layer integration sections to the conditions used by the judgment section.

5. The layer merging determination device of claim 4, wherein the adding unit uses at least one of the amount of calculation of the deep learning model and the amount of data exchanged between an external memory and the dedicated hardware as a condition for adding a range of the amount of calculation of the layer merging section such that the processing time of each layer merging section included in the computationally integrated model is uniform.

6. A layer integration determination device as claimed in any one of claims 1 to 5, wherein the generation unit includes: a labeling unit that identifiably labels, among the operations of the input deep learning model, operations that match operations that can be processed by the dedicated hardware and are pre-registered in a correspondence description unit; and an operation integration unit that generates the operation-integrated model by combining, as processing blocks, combinations of operations that can be labeled and are processable by the dedicated hardware and that match combinations of primitive operations that can be integrated and are pre-registered in the correspondence description unit, to reconstruct a computation graph.

7. A layer integration determination method executed by a layer integration determination device including a generation unit, an extraction unit, and a determination unit, wherein the generation unit integrates operations corresponding to each layer of a deep learning model including a plurality of convolution layers, the operations being input in the form of a computation graph in which each layer is represented by a node, and operations corresponding to each of the convolution layers that can be processed by dedicated hardware that executes inference processing of the deep learning model and surrounding layers associated with the convolution layers, to generate a computation-integrated model; the extraction unit extracts a subgraph from the computation-integrated model that matches a pattern registered in advance in a pattern description unit as a combination of convolution layers that can be layer-integrated based on the layer configuration of each of the convolution layers; and the determination unit determines, as a layer integration section, a section corresponding to the subgraph that satisfies a condition registered in advance in a condition description unit as a condition for layer integration based on the specifications of the dedicated hardware.

8. A layer integration judgment program for causing a computer to function as each part of the layer integration judgment device according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Methods of optimization of computational graphs of neural networks

    US20190303762A1