Processing apparatus and image processing apparatus

Abstract

To efficiently perform arithmetic processing in a neural network using a processor.SOLUTION: A processing apparatus that performs arithmetic processing in a neural network in which a plurality of feature planes are hierarchically connected. Convolution operation processing using the weight coefficient is performed for the feature plane data of the feature plane. The feature plane data is held and supplied. The weight coefficient is held and supplied. A common control parameter is stored for each feature plane group obtained by grouping a plurality of feature planes having a common feature plane to be referred to in arithmetic processing for calculating feature plane data based on commonality of the arithmetic processing. When feature plane data of a feature plane is calculated, control is performed to perform an operation according to a control parameter corresponding to a feature plane group to which the feature plane belongs.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a processing device and an image processing device, and more particularly to a processing device that performs processing in a neural network. [Background technology]

[0002] Neural networks, including convolutional neural networks (CNNs), are used in deep learning. Neural network processing often involves a variety of computational processes. For example, neural network processing can involve convolutional processing using feature surface data of various sizes and kernels of various sizes. To perform various computational processes using hardware (accelerators) that perform neural network processing, it is necessary to configure the accelerator registers according to the processing content.

[0003] For example, Patent Document 1 discloses that for each processing node (each convolution operation), an offset address for memory access and the number of operation execution threshold lines for execution control are stored in a register (setting unit and storage unit) provided in a unit operation execution unit. Also, Patent Document 2 discloses that for each processing layer, information for controlling convolution operation processing, such as the kernel size and the number of feature planes, is stored in a register (storage unit). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2008-310524 [Patent Document 2] Patent Publication No. 2020-201883 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, neural networks have become increasingly complex. For example, the same layer may contain feature planes of different sizes. Also, convolution operations in the same layer may have different kernel sizes or pooling processes. Furthermore, there may be connections between layers to skip processing.

[0006] When applying the method described in Patent Document 2 to such a complex neural network, registers are configured so that all convolution operations in one layer can be executed. For example, registers are configured so that processing is performed according to the maximum feature surface size and the maximum kernel size for multiple convolution operations in one layer. In this case, an excessive amount of memory is required to store feature surface data, and unnecessary product-sum operations are performed. On the other hand, if registers are configured for each convolution operation, as in the method described in Patent Document 1, the time required for register configuration reduces the operation speed.

[0007] An object of the present invention is to efficiently perform arithmetic processing in a neural network using a processing device. [Means for solving the problem]

[0008] A processing device according to one embodiment of the present invention has the following configuration: That is, a processing device that performs arithmetic processing in a neural network in which a plurality of feature planes are hierarchically connected, comprising: arithmetic means that performs convolution arithmetic processing on feature plane data of the feature planes using weighting coefficients, feature plane storage means that holds the feature plane data and supplies it to the arithmetic means, coefficient storage means that holds the weighting coefficients and supplies them to the arithmetic means, storage means that stores common control parameters for each feature plane group obtained by grouping a plurality of feature planes that share a common feature plane referenced in arithmetic processing to calculate feature plane data, based on the commonality of the arithmetic processing, and control means that controls the arithmetic means, the feature plane storage means, and the coefficient storage means so that, when calculating the feature plane data of a feature plane, the processing device performs an operation in accordance with the control parameters corresponding to the feature plane group to which the feature plane belongs. [Effects of the Invention]

[0009] It is possible to efficiently perform arithmetic processing in a neural network using a processing device. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of the arrangement of a processing apparatus according to an embodiment. [Figure 2] FIG. 1 is a diagram showing an example of a neural network used in an embodiment. [Figure 3] FIG. 10 is a diagram showing an example of control parameters for each channel. [Figure 4] FIG. 10 is a diagram showing an example of control parameters according to an embodiment. [Figure 5] FIG. 4 is a diagram showing an example of a data structure of a control parameter. [Figure 6] FIG. 4 is a diagram showing an example of a data structure of a control parameter. [Figure 7] FIG. 10 is a diagram showing an example of control parameters for each layer. [Figure 8] FIG. 8 is a diagram showing the flow of processing corresponding to the control parameters shown in FIG. 7. [Figure 9] FIG. 10 is a diagram showing an example of the amount of resources used by a processing device according to an embodiment. [Figure 10] 10 is a flowchart of a process performed by a processing device according to an embodiment. [Figure 11] FIG. 1 is a diagram showing an example of a neural network used in an embodiment. [Figure 12] FIG. 10 is a diagram showing the flow of processing using control parameters for each layer. [Figure 13] FIG. 1 is a diagram showing an example of the arrangement of an image processing apparatus according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] (Configuration of processing device) A processing device 100 according to one embodiment will be described with reference to FIG. 1. The processing device 100 performs arithmetic processing in a neural network in which multiple feature planes are hierarchically connected. An example of the neural network will be described later. FIG. 1 shows an example configuration of the processing device 100. The processing device 100 includes a control unit 110, a feature plane storage unit 120, a coefficient storage unit 130, a calculation unit 140, and a distribution unit 150. The processing device 100 may include multiple calculation units 140 for parallel processing. The number of other processing units is not limited to one. Each of these units may be implemented by a hardware circuit such as a sequencer, an ASIC, or an FPGA. A processing unit such as the control unit 110 may also be implemented by a CPU. The processing device 100 further includes a control parameter memory 111.

[0013] The control unit 110 controls the overall operation of the control unit 110. Specifically, the control unit 110 can control the feature plane storage unit 120, the coefficient storage unit 130, and the calculation unit 140, which will be described later. When calculating feature plane data for a feature plane, the control unit 110 controls these processing units so that they perform operations in accordance with control parameters corresponding to the feature plane group to which the feature plane belongs.

[0014] The control parameter memory 111 is a memory that stores control parameters. The control parameter memory 111 can temporarily hold control parameters input from outside the processing device 100. In this embodiment, the control parameter memory 111 stores common control parameters for each feature plane group. This feature plane group is obtained by grouping multiple feature planes that share a common feature plane referenced in the arithmetic processing for calculating feature plane data, based on the commonality of the arithmetic processing. Details of the feature plane group will be described later. In this embodiment, the control unit 110 has the control parameter memory 111. The control parameter memory 111 can be a memory such as a DRAM or an SRAM.

[0015] The feature plane storage unit 120 holds feature plane data. The feature plane storage unit 120 can temporarily store image data input from outside the processing device 100 and feature plane data obtained by convolution calculation processing by the calculation unit 140. In this embodiment, the feature plane storage unit 120 has a feature plane memory 121, which is a memory such as a DRAM or SRAM, that stores feature plane data. In addition, the feature plane storage unit 120 supplies the feature plane data to the calculation unit 140.

[0016] The coefficient storage unit 130 stores weighting coefficients. The coefficient storage unit 130 can temporarily store weighting coefficients used in the convolution operation process that are input from outside the processing device 100. The coefficient storage unit 130 may have a memory such as a DRAM or an SRAM for storing the weighting coefficients. The coefficient storage unit 130 also supplies the weighting coefficients to the operation unit 140.

[0017] The calculation unit 140 performs convolution calculation processing on the feature plane data using weighting coefficients. That is, the calculation unit 140 can perform convolution calculation processing on the feature plane data supplied from the feature plane storage unit 120 using weighting coefficients supplied from the coefficient storage unit 130. The calculation unit 140 can have, for example, a product-sum calculation circuit for the convolution calculation processing.

[0018] The calculation unit 140 calculates feature plane data of the feature plane to be processed by performing this convolution operation on each pixel of the feature plane to be processed. Furthermore, the feature plane storage unit 120 and the coefficient storage unit 130 sequentially supply feature plane data and weighting coefficients to the calculation unit 140 in accordance with the processing order of the convolution operation performed by the calculation unit 140. In this embodiment, the control unit 110 controls the calculation unit 140 so that the calculation unit 140 performs the convolution operation in accordance with the control parameters stored in the control parameter memory 111. Furthermore, the control unit 110 controls the supply of feature plane data and weighting coefficients to the calculation unit 140 by the feature plane storage unit 120 and the coefficient storage unit 130 in accordance with the control parameters stored in the control parameter memory 111. The feature plane storage unit 120 can hold the feature plane data of the feature plane to be processed calculated in this manner.

[0019] It should be noted that the processing performed by the calculation unit 140 is not limited to convolution calculation processing. For example, the calculation unit 140 can perform other calculation processing such as pooling processing and activation processing. The control unit 110 can control the type of calculation processing performed by the calculation unit 140. For example, the control unit 110 can control the calculation unit 140 to perform a specific calculation processing in order to calculate feature plane data for a specific feature plane, in accordance with control parameters stored in the control parameter memory 111.

[0020] The distribution unit 150 distributes data input from outside the processing device 100 to each unit of the processing device 100. For example, the distribution unit 150 can supply the input control parameters to the control parameter memory 111. The distribution unit 150 can also supply the input image data to the feature plane storage unit 120 as feature plane data. The distribution unit 150 can also supply the input weighting coefficients to the coefficient storage unit 130. Furthermore, the distribution unit 150 can output the feature plane data obtained by the convolution operation process and held in the feature plane storage unit 120 to the outside of the processing device 100.

[0021] (Neural Networks) The processing device according to this embodiment is capable of performing arithmetic processing in a neural network. An example of a neural network will be described with reference to FIG. 2(A). A neural network includes a plurality of feature planes connected hierarchically. For example, the neural network shown in FIG. 2(A) has feature planes CH0 to CH8. The feature planes are connected as shown by solid lines. Here, feature plane CH0 is an input feature plane, which corresponds to the image data to be processed that is input to the processing device 100. Furthermore, feature plane CH8 is an output feature plane, which corresponds to the result of arithmetic processing in the neural network that is output from the processing device 100.

[0022] In addition, in a neural network, weighting coefficients (also referred to as filter coefficients or kernels) used in convolution processing (also referred to as filtering processing) are also hierarchically connected. In processing in a neural network, convolution processing is performed for each spatial locality (window) using weighting coefficients determined by learning and feature data. The convolution processing is a product-sum operation, including multiple multiplication operations and cumulative addition operations. Then, feature data of the connected feature planes is calculated by performing a convolution operation on the feature data of the feature planes. In the example of FIG. 2(A), feature data of feature plane CH1 is calculated by performing a convolution operation on the feature data of feature plane CH0 using a 3×3×1ch weighting coefficient. Furthermore, feature data of feature plane CH4 is calculated by performing a convolution operation on the feature data of multiple feature planes CH1 to CH3 using a 7×7×3ch weighting coefficient. The convolution operation can be performed, for example, according to equation (1) described below.

[0023] Furthermore, in neural network processing, pooling processing can also be performed. Pooling processing is processing that outputs a representative value (maximum value, minimum value, average value, etc.) for each spatial local area (window). The stride is a parameter for pooling processing and indicates the width of the window movement. When the stride is 2, the feature image is reduced to half its size in both the vertical and horizontal directions by pooling processing. In the example shown in Figure 2(A), feature data for feature plane CH7 is calculated by performing a convolution operation and pooling processing on the feature data for feature plane CH4.

[0024] The multiple feature planes are classified into multiple layers (hierarchies) according to their connection relationships. In the example of FIG. 2(A), feature plane CH0 is classified into layer 0, feature planes CH1 to CH3 into layer 1, feature planes CH4 to CH6 into layer 2, feature plane CH7 into layer 3, and feature plane CH8 into layer 4. In the example of FIG. 2(A), multiple feature planes (e.g., feature planes CH1 to CH3) may correspond to multiple channels. In the following description, the feature plane data of each feature plane is one channel of data. A neural network may include connections between feature planes that skip layers. For example, in the example of FIG. 2(A), the connection between feature plane CH5 and feature plane CH8 skips layer 3.

[0025] The network structure of a neural network such as that shown in Figure 2 can be represented by network information, which can include the connection relationships between feature planes, filter size, bit width of weighting coefficients, feature plane size, and bit width of feature plane data.

[0026] Equation (1) shows an example of a calculation formula for the convolution process.

number

[0027] The feature plane storage unit 120 and the coefficient storage unit 130 sequentially supply the feature plane data and weighting coefficients used in the convolution operation to the calculation unit 140 in accordance with the processing order of the convolution operation performed by the calculation unit 140. As described above, the control unit 110 controls the operations of the feature plane storage unit 120, the coefficient storage unit 130, and the calculation unit 140 in accordance with the control parameters.

[0028] (control parameters) Fig. 3 shows an example of control parameters. Fig. 3 shows an example of setting control parameters for each feature surface to be processed. The control parameters can include a read buffer start address, a read buffer size, a write buffer start address, a write buffer size, and a kernel size for each feature surface to be processed. The control parameters can also include identification information for the feature surface to be processed, identification information for the reference feature surface, and information indicating whether or not pooling processing is to be performed.

[0029] The identification information of the processing-target feature surface is information that identifies the processing-target feature surface. The identification information of the reference feature surface is information that identifies the feature surface that is referenced to calculate the feature surface data of the processing-target feature surface. The read buffer start address indicates the start address of the memory area in feature plane memory 121 where the feature surface data of the reference feature surface is stored. The read buffer size indicates the size of the memory area in feature plane memory 121 where the feature surface data of the reference feature surface is stored. In this way, the read buffer start address and read buffer size identify the memory area in feature plane memory 121 where the feature surface data of the reference feature surface is stored. The control unit 110 can set the read buffer start address and read buffer size in a register included in the feature plane storage unit 120. Then, in order to calculate the feature plane data of the processing-target feature surface, the feature plane storage unit 120 can sequentially supply the feature plane data of the reference feature surface stored in feature plane memory 121 to the calculation unit 140 according to the information set in the register.

[0030] The register in the feature plane storage unit 120 can store a read counter value for each reference feature plane. For reference, the read counter values ​​corresponding to each reference feature plane are shown in Fig. 3. The feature plane storage unit 120 can identify the read position of the feature plane data to be supplied to the calculation unit 140 according to the read buffer start address and the read counter value. In other words, the feature plane storage unit 120 can sequentially supply the feature plane data of the reference feature planes stored in the feature plane memory 121 to the calculation unit 140 while counting up the read counter value.

[0031] The write buffer start address indicates the start address of a memory area in feature plane memory 121 that stores feature plane data of the feature plane to be processed. The write buffer size indicates the size of the memory area in feature plane memory 121 that stores feature plane data of the feature plane to be processed. In this way, the write buffer start address and write buffer size specify the memory area in feature plane memory 121 that stores the feature plane data of the feature plane to be processed. Control unit 110 can set the write buffer start address and write buffer size in a register included in feature plane storage unit 120. Then, feature plane storage unit 120 can sequentially store the feature plane data of the feature plane to be processed, which is calculated by control unit 110 through arithmetic processing and supplied to feature plane storage unit 120 from control unit 110, in feature plane memory 121 according to the information set in the register.

[0032] The register of the feature plane storage unit 120 can store a write counter value for each feature plane to be processed. For reference, FIG. 3 shows the write counter values ​​corresponding to each feature plane to be processed. The feature plane storage unit 120 can identify the write position of the feature plane data supplied from the calculation unit 140 according to the write buffer start address and the write counter value. That is, the feature plane storage unit 120 can sequentially store the feature plane data of the feature plane to be processed supplied from the calculation unit 140 in the feature plane memory 121 while counting up the write counter value. In the example of FIG. 3, the write buffer start address for the reference feature plane matches the read buffer start address for the feature plane to be processed. Therefore, when performing calculation processing on the feature plane to be processed, the feature plane storage unit 120 can supply the calculation unit 140 with the feature plane data of the reference feature plane that was previously calculated.

[0033] The kernel size indicates the size of the window in the convolution process and also indicates the number of weighting coefficients used in the convolution process. The control unit 110 can set information for identifying the feature surface to be processed and the kernel size in a register included in the coefficient storage unit 130. Then, the feature surface storage unit 120 can sequentially supply the weighting coefficients stored in the coefficient storage unit 130 to the calculation unit 140 in accordance with the information set in the register in order to calculate feature surface data for the feature surface to be processed.

[0034] The information indicating whether or not pooling is performed specifies whether the control unit 110 will perform pooling in addition to the convolution operation. The control unit 110 can specify information indicating the kernel size and whether or not pooling is performed in a register included in the operation unit 140. Then, the operation unit 140 performs convolution operation using the feature plane data of the processing target feature plane supplied from the feature plane storage unit 120 and the filter coefficients supplied from the coefficient storage unit 130, according to the information set in the register. Furthermore, the operation unit 140 further performs or does not perform pooling, according to the information set in the register. In this way, the operation unit 140 calculates the feature plane data of the processing target feature plane and sequentially supplies the calculated feature plane data to the feature plane storage unit 120. Note that the feature plane data supply operation by the feature plane storage unit 120 may be controlled according to the information indicating whether or not pooling is performed, so that the window moves according to the stride.

[0035] When operating in accordance with the control parameters shown in FIG. 3, when changing the feature surface to be processed, the control unit 110 can set the registers of the feature surface storage unit 120, the coefficient storage unit 130, and the calculation unit 140 in accordance with the control parameters corresponding to the feature surface to be processed.

[0036] On the other hand, in this embodiment, the control parameter memory 111 stores common control parameters for each feature plane group. This embodiment will be described with reference to FIG. 4. FIG. 4 shows an example of control parameters according to this embodiment. The control parameters can include, for each feature plane group to be processed, a read buffer start address, a read buffer size, a write buffer start address, a write buffer size, and a kernel size. Furthermore, the control parameters can include, for each feature plane group to be processed, identification information for the feature plane group to be processed, identification information for the reference feature plane group, and information indicating whether or not pooling processing is performed. Thus, the control parameters corresponding to the feature plane group to which a feature plane belongs can include information specifying the feature plane to be referenced in calculating feature plane data for the feature plane. The significance of each control parameter is as described above. Here, the feature plane group to be processed refers to the feature plane group for which feature plane data is to be calculated. Furthermore, the reference feature plane group refers to the feature plane group referenced in calculating feature plane data for the feature plane to be processed.

[0037] The control parameters corresponding to the feature plane group to which the feature plane belongs can indicate a memory area for storing the feature plane data that the feature plane storage unit 120 supplies to the control unit 110 in order to calculate the feature plane data for the feature plane. In the example shown in Fig. 4, as information indicating such a memory area, the read buffer start address and read buffer size are set for each feature plane group to be processed.

[0038] Furthermore, the control parameters corresponding to the feature plane group to which the feature plane belongs can indicate a memory area in which the control unit 110 stores the feature plane data for the feature plane calculated by the control unit 110. In the example shown in FIG. 4, the write buffer start address and write buffer size are also set for each feature plane group to be processed as information indicating such a memory area. That is, in this example, the feature plane storage unit 120 stores the feature plane data for two or more feature planes belonging to a feature plane group in consecutive memory areas in the control parameter memory 111. Therefore, as shown in the rows of the feature plane groups CG2 and CG3 to be processed in FIG. 4, the feature plane storage unit 120 can use one read counter and one write counter when performing arithmetic processing for one feature plane group to be processed. In this way, the registers included in the feature plane storage unit 120 can store a read counter value for each reference feature plane group. Furthermore, the registers included in the feature plane storage unit 120 can store a write counter value for each feature plane group to be processed.

[0039] Furthermore, the control parameters corresponding to the feature plane group to which the feature plane belongs can indicate the processing to be performed by the feature plane storage unit 120 to calculate feature plane data for the feature plane. Furthermore, the control parameters corresponding to the feature plane group to which the feature plane belongs can indicate the filter size of the convolution operation to be performed by the feature plane storage unit 120 to calculate feature plane data for the feature plane. In the example shown in Fig. 4, as information indicating such processing, information indicating whether or not pooling processing is to be performed and a kernel size are also set for each feature plane group to be processed.

[0040] As described above, a feature plane group is obtained by grouping multiple feature planes that share a common feature plane referenced in the arithmetic processing for calculating feature plane data, based on the commonality of the arithmetic processing. For example, in the example shown in FIG. 2(A), a common feature plane CH0 is referenced in the arithmetic processing for calculating the feature planes CH1 to CH3. Furthermore, the arithmetic processing for calculating the feature planes CH1 to CH3 is a convolution operation using a 3×3×1ch weighting coefficient, and does not include pooling processing. In this way, the arithmetic processing for calculating the feature planes CH1 to CH3 is common. For this reason, the feature planes CH1 to CH3 are grouped into the feature plane group CG1.

[0041] On the other hand, in the arithmetic processing for calculating the feature planes CH4 to CH6, the common feature planes CH1 to CH3 are referenced. However, the arithmetic processing for calculating the feature plane of feature plane CH4 is a convolution operation using a weighting factor of 7×7×3ch and does not include pooling. On the other hand, the arithmetic processing for calculating the feature planes CH5 to CH6 is a convolution operation using a weighting factor of 3×3×3ch and includes pooling. Thus, while the arithmetic processing for calculating the feature planes of feature planes CH5 to CH6 is common, it is different from the arithmetic processing for calculating the feature plane of feature plane CH4. For this reason, feature planes CH5 to CH6 are grouped into feature plane group CG3. On the other hand, feature plane CH4 is not grouped with feature planes CH5 to CH6 and independently constitutes feature plane group CG2.

[0042] Furthermore, there are no other feature surfaces that share the same reference feature surface in the calculation process for calculating the feature surface data as feature surface CH7, and therefore feature surface CH7 is not grouped with other feature surfaces, but rather forms a separate feature surface group CG4.

[0043] Figure 2(B) shows a neural network including such grouped feature planes. In this way, at least one feature plane group can include two or more feature planes. Furthermore, at least two feature planes in at least one layer may belong to different feature plane groups.

[0044] The method of grouping feature surfaces based on the commonality of arithmetic processing is not particularly limited. For example, feature surfaces may be grouped so that the arithmetic processing performed to calculate the feature surface data of each feature surface included in one feature surface group is common. For example, in FIG. 2(B), the arithmetic processing performed to calculate the feature surface data of feature surface group CG1 is common as described above. Therefore, as shown in FIG. 4, common control parameters are set for feature surface group CG1. In such a configuration, as will be described later, it is possible to minimize the time required to set the control parameters and the amount of data required for the control parameters.

[0045] On the other hand, the arithmetic processing performed to calculate the feature plane data of each feature plane included in a feature plane group does not have to be completely common. For example, feature planes may be grouped so that the widths and heights of each feature plane included in a feature plane group are the same. Having the same widths and heights of each feature plane indicates that the number of convolution operations required to calculate the feature plane data of each feature plane is the same. Furthermore, feature planes may be grouped so that at least one of the multiple arithmetic processing operations performed to calculate the feature plane data of each feature plane included in a feature plane group is common. For example, multiple feature planes that undergo pooling processing to calculate feature plane data may be grouped. In this case, the control parameters for one feature plane group may include control parameters common to each feature plane and control parameters for each feature plane. Even with this configuration, the control parameter setting time and the control parameter data volume can be reduced.

[0046] In the example shown in FIG. 2(B), to calculate the feature plane data of the feature plane groups CG1 to CG4, one feature plane group is referenced for each. Therefore, as shown in FIG. 4, the control parameters for each of the processing target feature plane groups CG1 to CG4 specify one reference feature plane group and specify the memory area in which the feature plane data of this reference feature plane group is stored. On the other hand, to calculate the feature plane data of the feature plane group CG5, the feature plane data of the feature plane groups CG3 and CG4 are referenced. As shown in FIG. 4, the feature plane data of the feature plane groups CG3 and CG4 are stored in different memory areas of the feature plane memory 121. In this way, the feature plane data of the feature plane group CG5 is obtained by combining the feature plane data of the feature plane groups CG3 and CG4 in the channel direction. Therefore, as shown in FIG. 4, the control parameters for the feature plane group CG5 specify the memory areas in which the feature plane data of the feature plane groups CG3 and CG4 are stored.

[0047] An example of the data structure of such control parameters will be described with reference to FIG. 5 and FIGS. 6(A)-(B). The control parameters shown in FIG. 5 include information specifying up to four reference feature surface groups for each feature surface group to be processed. For example, in FIG. 5, the control parameters for feature surface group CG5 to be processed include identification information for reference feature surface groups CG3 and CG4 that specifies reference feature surface groups CG3 and CG4. The control parameters for feature surface group CG5 to be processed also include information indicating memory areas in which the feature surface data of these reference feature surface groups CG3 and CG4 are stored. In this example, in the data structure shown in FIG. 5, the information indicating the memory areas in which the feature surface data of each feature surface group to be processed and the information indicating the processing content to be performed by feature surface storage unit 120 are common to each feature surface group to be processed. This allows the amount of data in the control parameters to be reduced.

[0048] On the other hand, the control parameters shown in FIG. 6(A) include, for each feature surface group to be processed, information identifying a reference feature surface group to be referenced in order to calculate feature surface data for the feature surfaces belonging to the feature surface group to be processed. For example, in FIG. 6(A), the control parameters for the feature surface group to be processed CG5 include identification information for the reference feature surface group CG3, which identifies the reference feature surface group CG3. The control parameters for the feature surface group to be processed CG5 also include the read buffer start address and read buffer size for the reference feature surface group CG3. Information indicating the memory area for storing this data also identifies the reference feature surface group CG3. The control parameters shown in FIG. 6(A) also include, for each feature surface group to be processed, information indicating the location of information identifying an additional reference feature surface group to be referenced in order to calculate feature surface data for the feature surfaces belonging to the feature surface group to be processed. For example, the control parameters for the feature surface group to be processed CG5 include a link number ACG1 indicating the location of information identifying the additional reference feature surface group CG4.

[0049] In addition to the control parameters having such a data structure, the control parameter memory 111 can store additional control parameters including information for identifying an additional reference feature surface group. Fig. 6(B) shows an example of the data structure of such additional control parameters. The control parameters shown in Fig. 6(B) include the read buffer start address and read buffer size of the additional reference feature surface group CG4, associated with the link number ACG1 indicating the additional reference feature surface group. Such additional control parameters may correspond to the link number ACG1 indicating the additional reference feature surface group and include identification information of the reference feature surface group CG4 that identifies the additional reference feature surface group CG4.

[0050] Furthermore, as shown in FIG. 6(A), the control parameters may include, for each target feature surface group, the number of additional reference feature surface groups to be referenced in order to calculate feature surface data for the feature surfaces belonging to the target feature surface group. In the example of FIG. 6(A), the control parameters for target feature surface group CG5 indicate that the number of additional reference feature surface groups is 1. Therefore, a record in one row starting with link number ACG1 is referenced as information identifying the additional reference feature surface group CG4. Furthermore, if the number of additional reference feature surface groups is 2, a record in two rows starting with link number ACG1 is referenced as information identifying the additional reference feature surface groups. This data structure allows for a further reduction in the amount of data in the control parameters compared to the data structure shown in FIG. 5, while also enabling any number of feature surface groups to be merged.

[0051] (Effects of this embodiment) By using the control parameters according to this embodiment, it is possible to efficiently perform arithmetic processing in a neural network using a processing device. For example, the control parameters according to this embodiment shown in FIG. 4 have a smaller amount of data than the control parameters for each feature plane shown in FIG. 3. Therefore, according to this embodiment, the amount of data for the control parameters can be reduced. Furthermore, the control parameters according to this embodiment shown in FIG. 3 have a smaller number of records (number of rows) than the control parameters shown in FIG. 4. This means that the control unit 110 can reduce the number of times it sets the registers of the feature plane storage unit 120, the coefficient storage unit 130, and the calculation unit 140, i.e., the calculation speed can be improved.

[0052] Alternatively, it is also possible to use control parameters for each layer. FIG. 7 shows an example of control parameters for each layer corresponding to a neural network. FIG. 8 shows the processing flow in the neural network according to the control parameters shown in FIG. 7. To represent the skip connection between feature planes CH5 and CH6 and feature plane CH8 using the control parameters for each layer, two dummy feature planes, dummy1 and dummy2, are inserted into layer 3, as shown in FIG. 8. The feature plane data of dummy feature planes dummy1 and dummy2 is the same as that of feature planes CH5 and CH6. In this example, because control parameters for each layer are used, the feature plane data of dummy feature planes dummy1 and dummy2 are calculated by a 3×3×3ch convolution operation, similar to that of feature plane CH7. Here, weighting coefficients are selected so that the feature plane data of dummy feature planes dummy1 and dummy2 is the same as that of feature planes CH5 and CH6. Similarly, the feature plane data of feature planes CH5 and CH6 is calculated by a 7×7×3ch convolution operation, similar to that of feature plane CH4. In this way, when control parameters for each layer are used, the amount of convolution calculation increases compared to when feature surfaces are grouped as shown in FIG.

[0053] 9 shows examples of the amount of resources used by the processing device 100 when using control parameters for each feature plane group, when using control parameters for each feature plane, and when using control parameters for each layer. Fig. 9 compares the amount of memory required to store feature plane data of the feature planes, the amount of convolution calculations, and the amount of control parameters in neural network processing for each case. By using control parameters for each feature plane group as in this embodiment, the amount of convolution calculations and the amount of control parameters can be reduced simultaneously, and the amount of feature plane memory can also be suppressed, thereby improving the efficiency of computational processing in the neural network.

[0054] (Processing method) An example of a method of operation of the processing device 100 will now be described with reference to the flow chart of FIG.

[0055] In S1010, the distribution unit 150 acquires control parameters from an external source and stores them in the control parameter memory 111. The control parameters are set manually or automatically in advance depending on the structure of the neural network. For example, the distribution unit 150 may acquire the control parameters from the memory 1320 included in the image processing device 1300 shown in FIG. 13.

[0056] In S1020, the control unit 110 selects a feature surface group to be processed in accordance with the control parameters. The control unit 110 may select the feature surface groups to be processed in the order of the feature surface group numbers indicated in the control parameters.

[0057] In S1030, control unit 110 sets the registers of feature plane storage unit 120, coefficient storage unit 130, and calculation unit 140 as described above, in accordance with the control parameters for the feature plane group to be processed. As described above, in this embodiment, control unit 110 can set the registers each time the feature plane group to be processed changes. Note that, as shown in FIG. 6, when an additional reference feature plane group is set, control unit 110 may set the registers of feature plane storage unit 120 so that feature plane data for both the reference feature plane group and the additional reference feature plane group can be read. On the other hand, control unit 110 may sequentially set the registers of feature plane storage unit 120 to read feature plane data for the reference feature plane group and to read feature plane data for the additional reference feature plane group as the calculation process progresses.

[0058] In S1040, the feature plane storage unit 120, the coefficient storage unit 130, and the calculation unit 140 perform processing in accordance with the information set in the registers, as described above. In this way, the feature plane data of the feature plane group to be processed is calculated. The results of the calculation processing are stored in the feature plane memory 121 in accordance with the information set in the registers, as described above. Note that when the feature plane group to be processed is CG0, the feature plane storage unit 120 stores the image data transmitted from the distribution unit 150 in the feature plane memory 121 as feature plane data of the feature plane group CG0. The distribution unit 150 can acquire image data from an external source and transmit it to the feature plane storage unit 120. For example, the distribution unit 150 may acquire image data from the memory 1320 included in the image processing device 1300 shown in FIG. 13 .

[0059] In S1050, the control unit 110 determines whether or not the processing for all feature surface groups to be processed has been completed. If the control unit 110 determines that the processing has been completed, the process proceeds to S1060. If not, the process returns to S1020, and arithmetic processing is performed for the next feature surface group to be processed.

[0060] In S1060, the distribution unit 150 acquires the results of the arithmetic processing stored in the feature plane memory 121. For example, the distribution unit 150 can acquire the data in the memory area indicated by the write buffer start address and write buffer size for feature plane group CG5 as the results of processing the input image in the neural network. The distribution unit 150 then outputs the results of the arithmetic processing to the outside. For example, the distribution unit 150 can store the results of the arithmetic processing in the memory 1320 included in the image processing device 1300 shown in FIG. 13.

[0061] Here, the description has been given of a case where the calculation processing is performed sequentially in units of feature plane groups. However, the order of the calculation processing is not particularly limited. For example, the calculation processing may be performed line by line. In this case, in S1020, the control unit 110 can select a processing target line of the processing target feature plane group. Furthermore, in S1020 after the processing of S1030 to S1050, the control unit 110 can select another processing target line of the same processing target feature plane group or a processing target line of another processing target feature plane group. The selection order of the processing target lines can be set manually or automatically in advance so that calculation of the feature plane data referenced for calculating the feature plane data of the processing target line is completed before calculation of the feature plane data of the processing target line is started. When the calculation processing is performed line by line in this way, the number of times register setting processing according to the control parameters is performed in S1030 is increased compared to when the calculation processing is performed in units of feature plane groups. Therefore, using control parameters for each feature plane group instead of using control parameters for each feature plane, as in this embodiment, is more effective in reducing the control parameter setting time.

[0062] (Feature surface memory constraints) The feature plane storage unit 120 may have multiple feature plane memories. Furthermore, memory input / output may be restricted so that, when performing convolution processing, feature plane data is read from one feature plane memory of the feature plane storage unit 120 while feature plane data is written to the other feature plane memory of the feature plane storage unit 120. In other words, there may be a restriction that the memory that supplies the feature plane data of the reference feature plane group is different from the memory that writes the feature plane data of the processing target feature plane group. Such a configuration makes it easy to speed up memory input / output.

[0063] A case will be described in which processing in the neural network shown in Fig. 11(A) is performed using processing device 100 having such a configuration. In this example, feature plane storage unit 120 has a first feature plane memory mem0 and a second feature plane memory mem1. To satisfy the above constraints, the feature plane data of feature plane groups CG0, CG2, and CG4 is stored in mem0. Furthermore, the feature plane data of feature plane groups CG1 and CG3 is stored in mem1.

[0064] On the other hand, feature plane group CG4 is connected to feature plane group CG0 in addition to feature plane group CG3. That is, to calculate the feature plane data of feature plane group CG4, the feature plane data of feature plane groups CG3 and CG0 are referenced. However, because the feature plane data of feature plane groups CG4 and CG0 are both stored in mem0, the above constraint cannot be met.

[0065] Therefore, in one embodiment, the multiple feature surfaces can include a first feature surface and a second feature surface connected to the first feature surface. Here, the feature surface data of the first feature surface and the feature surface data of the second feature surface are identical. In the example of FIG. 11(A), a feature surface group CG0', which is a copy of the feature surface group CG0, is inserted. The feature surface group CG0' is connected to the feature surface group CG0, and is further connected to the feature surface group CG4. In this case, the feature surface data of the feature surface group CG0' is stored in mem1, so the above constraints can be satisfied when calculating the feature surface data of the feature surface group CG4.

[0066] Figure 11(B) shows an example of control parameters for each feature plane group used when performing processing in the neural network shown in Figure 11(A). As shown in Figure 11(B), the feature plane group CG0' is calculated by convolution processing using weighting coefficients of 1x1x2ch. The amount of calculation required for this convolution processing is not very large. Furthermore, the area size of the additional feature plane memory used to store the feature plane group CG0' is limited to 96x96x2.

[0067] On the other hand, Figure 12 shows the processing flow in a neural network when a copy hierarchy is introduced to satisfy memory input / output constraints and control parameters for each layer are used. In this case, to use common control parameters for each layer, it is necessary to insert a copy feature plane CH0' and a copy feature plane CH1' into each of layers 1 to 3. Furthermore, to calculate the feature plane data for each copy feature plane CH0' and copy feature plane CH1', convolution processing is performed using a weighting coefficient of 3x3x2ch or 3x3x4ch. Therefore, the amount of additional convolution calculations and the area size of the additional feature plane memory due to the insertion of a copy feature plane become quite large.

[0068] Furthermore, as described above with reference to Fig. 9, when control parameters are used for each feature surface, the amount of control parameter data becomes considerably large. This also applies to the case where a copy feature surface is inserted. In this way, using control parameters for each feature surface group as in this embodiment is particularly useful when there are memory input / output constraints.

[0069] (Example of use of processing equipment) Fig. 13 is a block diagram showing an example of the configuration of an image processing device 1300 according to an embodiment. In Fig. 13, a processor 1310 is, for example, a CPU, and controls the operation of the entire computer. A memory 1320 is, for example, a RAM, and temporarily stores programs, data, etc. A computer-readable storage medium 1330 is, for example, a hard disk or a CD-ROM, and stores programs, data, etc. long-term. In this embodiment, a program stored in the storage medium 1330 is read into the memory 1320. Then, the processor 1310 operates according to the program on the memory 1320.

[0070] The input interface 1340 is an interface for acquiring information from an external device. The output interface 1350 is an interface for outputting information to an external device. The image processing device 1300 also includes the processing device 100 described above. The processing device 100 performs arithmetic processing on the image in a neural network. The bus 1360 connects the above-mentioned components and enables data exchange.

[0071] In this embodiment, the processor 1310 operates in accordance with a program on the memory 1320 to generate an image processing result for an image based on the processing result output by the processing device 100. For example, the processor 1310 can generate an image processing or image recognition result based on the processing result by the processing device 100. In one embodiment, the processing device 100 outputs a confidence map indicating the likelihood that a detection target object exists for each position or region of the input image. In this case, the processor 1310 can generate and output information indicating the position of a specific object in the image according to the confidence map. For example, the processor 1310 can determine that the object exists at the position of a peak value in the confidence map. The processor 1310 can then superimpose information indicating the determined position of the object on the input image.

[0072] Such an image processing device 1300 can be realized using a computer to which the processing device 100 is connected. Examples of the computer include a general-purpose desktop computer, a laptop computer, a tablet PC, and a smartphone. However, at least a part of the processing units of the image processing device 1300 may be realized by dedicated hardware. Furthermore, the image processing device 1300 may be configured by a plurality of information processing devices connected via a network, for example.

[0073] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0074] The disclosure of this specification includes the following processing devices and image processing devices. (Item 1) A processing device that performs arithmetic processing in a neural network in which a plurality of feature planes are hierarchically connected, a calculation means for performing a convolution calculation process on the feature plane data of the feature plane using a weighting coefficient; a feature surface storage means for storing the feature surface data and supplying the data to the calculation means; a coefficient storage means for storing the weighting coefficients and supplying the weighting coefficients to the calculation means; a storage means for storing common control parameters for each feature surface group obtained by grouping a plurality of feature surfaces that share a common feature surface referenced in a calculation process for calculating feature surface data based on the commonality of the calculation process; a control means for controlling the calculation means, the feature surface storage means, and the coefficient storage means so that, when calculating feature surface data of a feature surface, the operation follows a control parameter corresponding to the feature surface group to which the feature surface belongs; A processing device comprising: (Item 2) Item 2. The processing device according to item 1, characterized in that at least one of the feature surface groups includes two or more feature surfaces. (Item 3) 3. The processing device according to any one of items 1 to 2, wherein at least two feature surfaces in at least one layer belong to different feature surface groups. (Item 4) 4. The processing device according to any one of items 1 to 3, wherein the width and height of each of the feature surfaces included in one of the feature surface groups are the same. (Item 5) 5. The processing device according to any one of items 1 to 4, characterized in that at least one of a plurality of arithmetic processes performed to calculate the feature surface data of each feature surface included in one of the feature surface groups is common. (Item 6) 6. The processing device according to any one of items 1 to 5, characterized in that the arithmetic processing performed to calculate the feature surface data of each feature surface included in one of the feature surface groups is common. (Item 7) 7. The processing device according to any one of items 1 to 6, wherein the control parameters corresponding to the feature plane group to which the feature plane belongs include information specifying a feature plane to be referenced in order to calculate feature plane data for the feature plane. (Item 8) 8. The processing device according to any one of items 1 to 7, wherein the control parameter corresponding to the feature plane group to which the feature plane belongs indicates a memory area for storing feature plane data that the feature plane storage means supplies to the calculation means in order to calculate feature plane data for the feature plane. (Item 9) 9. The processing device according to any one of items 1 to 8, wherein the control parameter corresponding to the feature surface group to which the feature surface belongs indicates a memory area into which the feature surface data for the feature surface calculated by the calculation means is written. (Item 10) 10. The processing device according to any one of items 1 to 9, wherein the control parameters corresponding to the feature surface group to which the feature surface belongs indicate processing to be performed by the calculation means to calculate feature surface data for the feature surface. (Item 11) 11. The processing device according to any one of items 1 to 10, wherein the control parameter corresponding to the feature plane group to which the feature plane belongs indicates a filter size of a convolution operation performed by the calculation means to calculate feature plane data for the feature plane. (Item 12) 12. The processing device according to any one of items 1 to 11, wherein the feature surface storage means stores feature surface data for two or more feature surfaces belonging to the feature surface group in a continuous memory area. (Item 13) 13. The processing device according to any one of items 1 to 12, wherein the control parameters corresponding to the feature plane group to which the feature plane belongs include information indicating a first feature plane to be referenced in order to calculate feature plane data for the feature plane, and information indicating the location of information indicating a second feature plane to be referenced in order to calculate feature plane data for the feature plane. (Item 14) Item 14. The processing device according to item 13, wherein the storage means stores, in addition to the control parameters, additional control parameters including information indicating a memory area for storing feature surface data of the second feature surface. (Item 15) the feature plane storage means has a first feature plane memory and a second feature plane memory for storing the feature plane data; the plurality of feature surfaces include a first feature surface and a second feature surface connected to the first feature surface, and feature surface data of the first feature surface and feature surface data of the second feature surface are identical; 15. The processing device according to any one of items 1 to 14, wherein feature plane data of the first feature plane is stored in the first feature plane memory, and feature plane data of the second feature plane is stored in the second feature plane memory. (Item 16) A processing device according to any one of items 1 to 15, which performs arithmetic processing on an image using a neural network; generating means for generating a result of image processing for the image based on the processing result output by the processing device; An image processing device comprising:

[0075] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0076] 100: Processing device, 110: Control unit, 111: Control parameter memory, 120: Feature surface storage unit, 121: Feature surface memory, 130: Coefficient storage unit, 140: Calculation unit, 150: Distribution unit

Claims

1. A processing device that performs arithmetic processing in a neural network in which a plurality of feature planes are hierarchically connected, a calculation means for performing a convolution calculation process on the feature plane data of the feature plane using a weighting coefficient; a feature surface storage means for storing the feature surface data and supplying the data to the calculation means; a coefficient storage means for storing the weighting coefficients and supplying the weighting coefficients to the calculation means; a storage means for storing common control parameters for each feature surface group obtained by grouping a plurality of feature surfaces that share a common feature surface referenced in a calculation process for calculating feature surface data based on the commonality of the calculation process; a control means for controlling the calculation means, the feature surface storage means, and the coefficient storage means so that, when calculating feature surface data of a feature surface, the operation follows a control parameter corresponding to the feature surface group to which the feature surface belongs; A processing device comprising:

2. 2. The processing apparatus of claim 1, wherein at least one of said feature surface groups includes two or more feature surfaces.

3. 2. The processing device according to claim 1, wherein at least two feature surfaces in at least one hierarchy belong to different feature surface groups.

4. 2. The processing device according to claim 1, wherein the width and height of each of the feature surfaces included in one of said feature surface groups are the same.

5. 2. The processing device according to claim 1, wherein at least one of a plurality of arithmetic processes performed to calculate the feature surface data of each of the feature surfaces included in one of said feature surface groups is common.

6. 2. The processing device according to claim 1, wherein a common arithmetic process is performed to calculate the feature surface data of each of the feature surfaces included in one of said feature surface groups.

7. 2. The processing device according to claim 1, wherein the control parameters corresponding to the feature plane group to which the feature plane belongs include information specifying a feature plane to be referenced for calculating feature plane data for the feature plane.

8. 2. The processing device according to claim 1, wherein a control parameter corresponding to the feature plane group to which the feature plane belongs indicates a memory area for storing feature plane data that the feature plane storage means supplies to the calculation means in order to calculate feature plane data for the feature plane.

9. 2. The processing device according to claim 1, wherein the control parameter corresponding to the feature surface group to which the feature surface belongs indicates a memory area into which the feature surface data for the feature surface calculated by the calculation means is written.

10. 2. The processing device according to claim 1, wherein the control parameter corresponding to the feature surface group to which the feature surface belongs indicates a process to be performed by the calculation means to calculate feature surface data for the feature surface.

11. 2. The processing device according to claim 1, wherein the control parameter corresponding to the feature plane group to which the feature plane belongs indicates a filter size for a convolution operation performed by the calculation means to calculate feature plane data for the feature plane.

12. 2. The processing device according to claim 1, wherein said feature plane storage means stores the feature plane data for two or more feature planes belonging to said feature plane group in a continuous memory area.

13. 2. The processing device according to claim 1, wherein the control parameters corresponding to the feature plane group to which the feature plane belongs include information indicating a first feature plane to be referenced in order to calculate feature plane data for the feature plane, and information indicating the location of information indicating a second feature plane to be referenced in order to calculate feature plane data for the feature plane.

14. 14. The processing device according to claim 13, wherein said storage means stores, in addition to said control parameters, additional control parameters including information indicating a memory area for storing feature surface data of said second feature surface.

15. the feature plane storage means has a first feature plane memory and a second feature plane memory for storing the feature plane data; the plurality of feature surfaces include a first feature surface and a second feature surface connected to the first feature surface, and feature surface data of the first feature surface and feature surface data of the second feature surface are identical; 2. The processing device according to claim 1, wherein the feature plane data of the first feature plane is stored in the first feature plane memory, and the feature plane data of the second feature plane is stored in the second feature plane memory.

16. The processing device according to any one of claims 1 to 15, which performs arithmetic processing on an image using a neural network; generating means for generating a result of image processing for the image based on the processing result output by the processing device; An image processing device comprising:

Images