System, method, and program

Abstract

To execute processing more appropriately even in the case of a network structure in which a processing result in a certain layer in a convolutional neural network is input to a next layer and a further subsequent layer.SOLUTION: A sub-network included in a convolutional neural network includes a first layer, a second layer, a third layer, and a fourth layer, and has a network structure in which an output of the first layer is an input to the second layer, an output of the second layer is an input to the third layer, and an output of the first layer and an output of the third layer are inputs to the fourth layer. The acquisition unit divides and acquires data to be processed so that a total size of input data to each layer is equal to or less than a storage capacity of the internal memory.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a system, a method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technology for performing calculations for multiple intermediate layers that make up a convolutional neural network using a memory with multiple banks that can switch the read / write state on a bank-by-bank basis. In Patent Document 1, the allocation of the read / write state of banks that store the input data or output data of the intermediate layers is switched depending on the transfer amount and transfer speed of the input data and output data of the intermediate layers that make up the convolutional neural network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-207458 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the prior art does not consider the problem of a network structure in which the processing result of a certain layer in a convolutional neural network (CNN) is input to the next layer and further layers. Other problems and novel features will become apparent from the description of this specification and the accompanying drawings. [Means for solving the problem]

[0005] In one embodiment, a system is provided that includes a processing circuit that performs operations on a sub-network included in a convolutional neural network and an external memory, the processing circuit having an internal memory, an acquisition unit, and a control unit, the sub-network including a first layer, a second layer, a third layer, and a fourth layer, and having a network structure in which an output of the first layer is input to the second layer, an output of the second layer is input to the third layer, and an output of the first layer and an output of the third layer are input to the fourth layer, the acquisition unit divides and acquires data to be processed so that the total size of the output of the first layer and the output of the third layer is equal to or less than the storage capacity of the internal memory, and the control unit executes processing of each layer included in the sub-network based on the data acquired by the acquisition unit. [Effects of the Invention]

[0006] According to the embodiment, even in the case of a network structure in which the processing result in a certain layer in a convolutional neural network is input to the next layer and further subsequent layers, processing can be executed more appropriately. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of a control unit according to the embodiment. [Figure 3] FIG. 3 is a flowchart showing an example of the system 1 according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a network structure of a convolutional neural network according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of information indicating the network structure of a convolutional neural network according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of input / output destinations of each layer included in the sub-network according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] The principles of the present disclosure will be described with reference to some exemplary embodiments. It should be understood that these embodiments are set forth for illustrative purposes only, to aid those skilled in the art in understanding and practicing the present disclosure, without implying any limitation on the scope of the disclosure. The disclosure described herein may be implemented in various ways other than those described below.

[0009] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessarily required to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0011] <Configuration> The configuration of a system 1 according to an embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram illustrating an example of the configuration of the system 1 according to the embodiment. The system 1 may be realized by, for example, a semiconductor device. The technology of the present disclosure can be applied to, for example, an image processing device such as a neural network processing accelerator for image recognition, and an image processing system that performs calculations related to image recognition such as convolution processing. The technology of the present disclosure can also be applied to, for example, automatic driving and driving assistance of moving objects such as automobiles, automatic navigation of moving objects, and object identification in surveillance cameras.

[0012] The system 1 includes accelerators 10-1, ..., accelerator 10-N (N is an integer of 2 or more), a main control unit 20, and an external memory 30. In the following, when there is no need to distinguish between the accelerators 10-1, ..., accelerator 10-N, they will be simply referred to as "accelerator 10" as appropriate. The accelerator 10 is an example of a "processing circuit."

[0013] The main control unit 20 controls each unit of the system 1. The external memory 30 is provided outside the accelerators 10 and is a memory that can be read and written by each accelerator 10.

[0014] The accelerator 10 may be, for example, hardware for realizing acceleration of processing in a neural network. Note that circuit information of the accelerator 10 may be provided as, for example, an IP (Intellectual Property) core.

[0015] In the example of FIG. 1, accelerator 10-1 has internal memory 11-1, acquisition unit 12-1, and control unit 13-1. Internal memory 11-1 is a memory provided inside accelerator 10-1. Note that the configurations of accelerators 10 other than accelerator 10-1 are the same as accelerator 10-1. Hereinafter, when it is not necessary to distinguish between control units 13-1, ..., 13-N of each accelerator 10, they will be simply referred to as "control unit 13" as appropriate.

[0016] Acquisition unit 12-1 divides the data to be processed and acquires it from main control unit 20 so that the total size of input data to each layer of each sub-network included in the convolutional neural network is equal to or less than the storage capacity of internal memory 11-1. Control unit 13-1 executes processing of each layer included in the sub-network based on the data acquired by acquisition unit 12-1.

[0017] <Hardware configuration> Fig. 2 is a diagram showing an example of the hardware configuration of the control unit 13 according to the embodiment. In the example of Fig. 2, the control unit 13 (computer 100) includes a processor 101, a memory 102, and a communication interface 103. These units may be connected by a bus or the like. The memory 102 stores at least a part of a program 104. The communication interface 103 includes an interface required for communication with other network elements.

[0018] When the program 104 is executed by the processor 101, memory 102, and the like in cooperation with each other, the computer 100 performs at least some of the processing of the embodiments of the present disclosure. The memory 102 may be of any type. As a non-limiting example, the memory 102 may be a non-transitory computer-readable storage medium. The memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. Although only one memory 102 is shown in the computer 100, several physically different memory modules may exist in the computer 100. The processor 101 may be of any type. The processor 101 may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture, as a non-limiting example. The computer 100 may have multiple processors, such as application-specific integrated circuit chips that are time-slaved to a clock that synchronizes the main processor.

[0019] Embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.

[0020] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, that execute on a target real or virtual processor or device to perform the processes or methods of the present disclosure. Program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or divided among program modules as desired in various embodiments. The machine-executable instructions of the program modules may be executed in local or distributed devices. In a distributed device, the program modules may be located in both local and remote storage media.

[0021] The program code for executing the methods of the present disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus. When the program code is executed by the processor or controller, the functions / acts in the flowcharts and / or implementing block diagrams are performed. The program code may be executed entirely on the machine, partly on the machine, as a standalone software package, partly on the machine and partly on a remote machine, or entirely on a remote machine or server.

[0022] The program can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media, magneto-optical recording media, optical disk media, and semiconductor memory. Magnetic recording media include, for example, flexible disks, magnetic tapes, and hard disk drives. Magneto-optical recording media include, for example, magneto-optical disks. Optical disk media include, for example, Blu-ray discs, CD (Compact Disc)-ROMs (Read Only Memory), CD-Rs (Recordable), and CD-RWs (Rewritable). Semiconductor memory includes, for example, solid-state drives, mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory). The program may also be provided to a computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire or an optical fiber, or via a wireless communication path.

[0023] <Processing> Next, an example of processing of the system 1 according to the embodiment will be described with reference to Fig. 3 to Fig. 6. Fig. 3 is a flowchart showing an example of the system 1 according to the embodiment. Fig. 4 is a diagram showing an example of the network structure of a convolutional neural network according to the embodiment. Fig. 5 is a diagram showing an example of information 501 indicating the network structure of a convolutional neural network according to an embodiment. Fig. 6 is a diagram showing an example of input / output destinations of each layer included in a sub-network according to an embodiment. Note that the processing in Fig. 3 may be executed at a timing corresponding to an operation by an operator (administrator), for example.

[0024] In step S101, the acquisition unit 12-1 divides a convolutional neural network to be processed into multiple subnetworks for extracting image features. Here, the acquisition unit 12-1 may divide the convolutional neural network to be processed into each subnetwork including multiple layers, for example, based on information indicating the network structure of the convolutional neural network to be processed. Note that the convolutional neural network to be processed may also be divided into multiple subnetworks by an operator or the like. In this case, information about each divided subnetwork may be specified (set) in advance in the system 1 by, for example, an operator or the like.

[0025] 4 shows an example of a network structure of a convolutional neural network according to an embodiment. In the example of FIG. 4, the convolutional neural network according to the embodiment has a backbone part 401, which is a main feature extraction part of the model, and a head part 402. The backbone part 401 extracts features from an input image. The head part 402 uses the features extracted by the backbone part 401 to generate an output suitable for a specific task (classification, detection, segmentation, etc.).

[0026] In the example of Fig. 4, in the convolutional neural network according to the embodiment, in the backbone part 401, the output of a certain layer 411 is input to the immediately succeeding layer 412 and another layer 413 several stages later. Note that, hereinafter, when the output of a certain layer is also input to a layer other than the immediately succeeding layer, the input / output from the certain layer to the other layer is also referred to as a "skip connection."

[0027] Fig. 5 shows an example of information 501 indicating the network structure of a convolutional neural network according to the embodiment. Fig. 6 shows an example of input / output destinations of each layer included in a sub-network according to the embodiment.

[0028] In the example of FIG. 5, data on a combination of an operation type, an operation parameter, the number of inputs, the number of outputs, and input connection information is recorded for each layer. The operation type is the type of operation performed in each layer. The operation type may include, for example, a convolution process (Conv) for extracting features, and a process (Pooling) for reducing the resolution of convolved data (feature map). The operation parameters are, for example, parameters for the operation performed by each layer. The operation parameters may also specify the size of the output data. The input connection information is information on the layer that inputs data to each layer.

[0029] In the examples of Figures 5 and 6, the output of Layer 1 is input to the immediately succeeding Layer 2. The output of Layer 2 (an example of a "first layer") is input to the immediately succeeding Layer 3 and to Layer 6, which is several stages later. Therefore, Layer 2 is skip-connected to Layer 6. The output of Layer 3 (an example of a "second layer") is input to the immediately succeeding Layer 4. The output of Layer 4 (an example of a "third layer") is input to the immediately succeeding Layer 5. The output of Layer 5 (another example of a "third layer") is input to the immediately succeeding Layer 6 (an example of a "fourth layer").

[0030] The following steps S102 to S106 are performed for each sub-network included in the specific convolutional neural network to be processed. Also, the output of one sub-network may be used as the input to another sub-network.

[0031] Next, the acquisition unit 12-1 divides the data (e.g., an image) to be processed by the convolutional neural network and acquires the divided data from the main control unit 20 to be used as input to the sub-network to be processed (step S102). Here, the acquisition unit 12-1 may divide and acquire the data to be processed so that the total size of the output of the layer from which the output branches and the output of the layer immediately before the layer to which the output of the layer joins is equal to or less than the storage capacity of the internal memory 11-1. This allows, for example, when processing the layer where the output branches, to store the output of the layer from which the output branches and the output of the layer immediately before the layer, thereby improving the processing speed.

[0032] The subnetwork to be processed includes a first layer (e.g., layer 2 in FIG. 6), a second layer subsequent to the first layer (e.g., layer 3 in FIG. 6), a third layer subsequent to the second layer (e.g., layers 4 and 5 in FIG. 6), and a fourth layer subsequent to the third layer (e.g., layer 6 in FIG. 6). The subnetwork to be processed has a network structure in which the output of the first layer is input to the second layer, the output of the second layer is input to the third layer, and the output of the first layer and the output of the third layer are input to the fourth layer. In this case, the acquiring unit 12-1 divides and acquires the data to be processed so that the total size of the output of the first layer and the output of the third layer is equal to or less than the storage capacity of the internal memory 11-1.

[0033] Furthermore, the acquisition unit 12-1 may divide and acquire the data to be processed so that the total size of the first layer output and the third layer output is the maximum possible size within the storage capacity of the internal memory 11-1. This allows, for example, when processing the fourth layer, the first layer output and the third layer output to be processed together as the maximum possible data within the range that can be stored in the internal memory 11-1, thereby further improving the processing speed. In this case, the acquisition unit 12-1 may, for example, determine multiple candidate sizes for dividing the data to be processed and select one candidate that meets the conditions.

[0034] Furthermore, the acquiring unit 12-1 may divide and acquire the data to be processed so that the size of the output data of the first layer is equal to or less than a threshold value according to the storage capacity of the external memory 30. This reduces the occurrence of a situation where, for example, when the accelerators 10 perform processing in parallel, the processing of other accelerators 10 is delayed due to a relatively large amount of data recorded in the external memory 30 by one accelerator 10. In this case, the threshold value according to the storage capacity of the external memory 30 may be determined by multiplying the storage capacity of the external memory 30 by a specific coefficient. In this case, the specific coefficient may be set in the system 1 in advance by an operator or the like.

[0035] Next, the control unit 13-1 uses the data acquired by the acquisition unit 12-1 as input to execute processing of the first layer of the sub-network to be processed, and records the output of the first layer in the internal memory 11-1 and the external memory 30 (step S103).

[0036] Next, the control unit 13-1 executes the processing of the second layer using the output of the first layer recorded in the internal memory 11-1 as the input to the second layer, and records (overwrites) the output of the second layer in the internal memory 11-1 (step S104).

[0037] Next, the control unit 13-1 executes processing of the third layer using the output of the second layer recorded in the internal memory 11-1 as input to the third layer, and records (overwrites) the output of the third layer in the internal memory 11-1 (step S105).

[0038] Next, the control unit 13-1 moves the first layer output recorded in the external memory 30 to a storage area (empty storage area) other than the storage area in the internal memory 11-1 where the third layer output is recorded (step S106).

[0039] Next, the control unit 13-1 executes processing of the fourth layer using the third layer output and the first layer output recorded in the internal memory 11-1 as input to the fourth layer, records (overwrites) the fourth layer output in the internal memory 11-1 or the external memory 30 (step S107), and terminates the processing.

[0040] When the accelerators 10 perform parallel processing, the control unit 13-1 may input data passed from one accelerator 10 to another accelerator 10 via the external memory 30.

[0041] <Modification> System 1 may be a device mounted on a single substrate (chip) or a device contained in a single housing, but the system 1 of the present disclosure is not limited to this. Each unit of system 1 may be realized, for example, by cloud computing configured with one or more computers. Furthermore, at least a portion of the processing of each functional unit of control unit 13 may be executed by main control unit 20. Such a system 1 is also included as an example of the "system" of the present disclosure.

[0042] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate. [Explanation of symbols]

[0043] 1 System 10. Accelerators 11 Internal memory 12 Acquisition Department 13 Control Unit 20 Main control unit 30 External Memory

Claims

1. A processing circuit for performing operations on a sub-network included in a convolutional neural network and an external memory are included, the processing circuit has an internal memory, an acquisition unit, and a control unit; the sub-network has a network structure including a first layer, a second layer, a third layer, and a fourth layer, in which an output of the first layer is an input to the second layer, an output of the second layer is an input to the third layer, and an output of the first layer and an output of the third layer are an input to the fourth layer; the acquisition unit divides and acquires data to be processed so that a total size of the output of the first layer and the output of the third layer, which is to be input to the fourth layer, is equal to or less than a storage capacity of the internal memory; The control unit executes processing of each layer included in the sub-network based on the data acquired by the acquisition unit. system.

2. The control unit executing the first layer processing using the data acquired by the acquisition unit as input, and recording the output of the first layer in the internal memory and the external memory; Executing the processing of the second layer using the output of the first layer recorded in the internal memory as an input to the second layer, and recording the output of the second layer in the internal memory; Executing the processing of the third layer using the output of the second layer recorded in the internal memory as an input to the third layer, and recording the output of the third layer in the internal memory; and executing processing of the fourth layer using the output of the third layer recorded in the internal memory and the output of the first layer recorded in the internal memory from the external memory as inputs to the fourth layer. The system of claim 1 .

3. the acquisition unit divides and acquires the data to be processed so that the size of the output data of the first layer is equal to or smaller than a threshold value according to the storage capacity of the external memory.

3. The system according to claim 1 or 2.

4. the acquisition unit divides and acquires the data to be processed so that a total size of the output of the first layer and the output of the third layer is a maximum that is equal to or less than a storage capacity of the internal memory.

3. The system according to claim 1 or 2.

5. The sub-network extracts image features.

3. The system according to claim 1 or 2.

6. the processing circuit includes a first processing circuit and a second processing circuit that perform parallel processing; the control unit records data input from the first processing circuit to the second processing circuit in the external memory.

3. The system according to claim 1 or 2.

7. the acquisition unit divides the convolutional neural network into sub-networks each including a plurality of layers based on information indicating a network structure of the convolutional neural network; 3. The system according to claim 1 or 2.

8. A processing circuit for performing operations on a sub-network included in a convolutional neural network, the sub-network having a network structure including a first layer, a second layer, a third layer, and a fourth layer, wherein an output of the first layer is input to the second layer, an output of the second layer is input to the third layer, and an output of the first layer and an output of the third layer are input to the fourth layer, Dividing and acquiring data to be processed so that the total size of the output of the first layer and the output of the third layer, which is to be input to the fourth layer, is equal to or less than the storage capacity of an internal memory; Execute processing of each layer included in the sub-network based on the acquired data. method.

9. A processing circuit for performing operations on a sub-network included in a convolutional neural network, the sub-network having a network structure including a first layer, a second layer, a third layer, and a fourth layer, wherein an output of the first layer is input to the second layer, an output of the second layer is input to the third layer, and an output of the first layer and an output of the third layer are input to the fourth layer, Dividing and acquiring data to be processed so that the total size of the output of the first layer and the output of the third layer, which is to be input to the fourth layer, is equal to or less than the storage capacity of an internal memory; Execute processing of each layer included in the sub-network based on the acquired data. A program that causes the processing circuit to execute processing.

Images