Forward propagation device, learning device, processing method, and program

Abstract

Provided is a novel technology with which it is possible to achieve high-speed processing by omitting some of the multiply-accumulate operations in convolutional layers. A forward propagation device (2) for a neural network comprises a mask generation unit (4) that generates a binary mask, and a layer execution unit (5) that performs a sparse convolutional layer operation according to the value at each coordinate in the binary mask, wherein the mask generation unit (4): performs convolutional layer operations on an input feature map to generate a number of heat maps equal to the number of types of detection targets of the neural network; combines a number of heat maps equal to the number of types of detection targets into a single heat map to generate a combined heat map, by totaling the values of the heat maps at each coordinate; and binarizes the value at each coordinate in the combined heat map using a prescribed threshold value to generate said binary mask.

Description

[Technical field]

[0001] The present disclosure relates to a forward propagation device, a learning device, an information processing system, a processing method, and a non-transitory computer-readable medium having a program stored thereon. [Background technology]

[0002] Technological developments related to video recognition and natural language processing using deep neural networks (DNNs) are actively underway. In particular, in the field of image recognition, convolutional neural networks (CNNs), a type of DNN, have achieved high recognition accuracy. Examples of the image recognition field mentioned above include object recognition, which estimates the type and coordinate position of an object, semantic region segmentation, which estimates the type of an object on a pixel-by-pixel basis, and human pose estimation, which estimates the coordinates of human joint points.

[0003] CNN can achieve high recognition by training on a large-scale server using a large amount of learning data. On the other hand, CNN inference needs to be performed in real time in power-limited environments such as edge devices and mobile terminals. CNN is mainly composed of convolutional layers. In the convolutional layer, many multiplication and accumulation operations need to be repeated.

[0004] Various studies have been conducted to process CNNs, which require a huge number of product-sum operations, at high speed. For example, Non-Patent Document 1 proposed a method to speed up 3D object detection by using two CNNs. This document proposed a method in which the first CNN, a semantic region segmentation CNN, infers a mask indicating the image area around the object, and the second CNN, an object detection CNN, performs convolution operations only on the image area around the object. This method achieved a 2.7-fold speed increase, although it was accompanied by a degradation of recognition accuracy of about 1%, compared to a normal object detector that performs product-sum operations the same number of times for all pixels. In addition, Non-Patent Document 2 proposed a mask unit that infers the image area around the object every two to three layers. In this technology, in the evaluation of pose estimation, it was realized to be 1.6 times faster without a degradation of recognition accuracy compared to when this technology is not used. In addition, Patent Documents 1 to 3 proposed a convolution hardware circuit that reduces the number of product-sum operations by omitting product-sum operations when the pixel value of the input feature map is 0. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Mengye Ren et al., “SBNet: Sparse Blocks Network for Fast Inference”, Computer Vision and Pattern Recognition Conference (CVPR) 2018. [Non-Patent Document 2] Thomas Verelst et al., “Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference”, CVPR 2020. [Patent documents]

[0006] [Patent Document 1] US Patent Application Publication No. 2019 / 0205740 [Patent Document 2] US Patent Application Publication No. 2020 / 0082254 [Patent Document 3] US Patent Application Publication No. 2020 / 0012608 Summary of the Invention [Problem to be solved by the invention]

[0007] Thus, there is a demand for a technique for processing neural networks at high speed. Therefore, one of the objectives of the embodiments disclosed in this specification is to provide a novel technique that can realize high-speed processing by omitting part of the product-sum calculations in the convolution layer. [Means for solving the problem]

[0008] A forward propagation device according to a first aspect of the present disclosure includes: A forward propagation device for a neural network, comprising: a mask generating means for generating a binary mask; A layer execution means for performing an operation of a sparse convolution layer according to the value of each coordinate of the binary mask; having The mask generating means A convolutional layer is operated on the input feature map to generate a heat map of the number of types of detection targets of the neural network; The heat map values ​​are summed for each coordinate to generate a composite heat map, which is a composite of the heat maps for the number of detection target types. The binary mask is generated by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold value.

[0009] A learning device according to a second aspect of the present disclosure, A learning device for a forward propagation device of a neural network, comprising: a first heat map acquisition means for acquiring a first heat map of the number of types of detection targets of the neural network, the first heat map being generated by the forward propagation device; A second heat map acquisition means for acquiring a second heat map of the number of types of the detection objects generated based on the correct labels of the detection objects; a loss calculation means for calculating a difference between the first heat map and the second heat map; an update means for updating weight values ​​of a convolutional layer for generating the first heat map in the forward propagation device based on the calculated difference; having The forward propagation device is a device that generates a binary mask and performs an operation of a sparse convolution layer according to the values ​​of each coordinate of the generated binary mask; The forward propagation device, in the process of generating the binary mask, A convolutional layer is operated on the input feature map to generate the first heat map of the number of types of detection targets of the neural network; generating a composite heat map by combining the first heat maps for the number of types of detection targets into one heat map by summing the values ​​of the first heat maps for the number of types of detection targets for each coordinate; The binary mask is generated by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold value.

[0010] An information processing system according to a third aspect of the present disclosure includes: A neural network forward propagation device; a learning device for the forward propagation device; Equipped with The forward propagation device comprises: a mask generating means for generating a binary mask; A layer execution means for performing an operation of a sparse convolution layer according to the value of each coordinate of the binary mask; having The mask generating means A convolutional layer is operated on the input feature map to generate a first heat map of the number of types of detection targets of the neural network; generating a composite heat map by summing the values ​​of the first heat maps for the number of types of detection targets for each coordinate, and synthesizing the first heat maps for the number of types of detection targets into a single heat map; generating the binary mask by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold; The learning device includes: a first heat map acquisition means for acquiring the first heat map of the number of types of detection targets of the neural network, the first heat map being generated by the forward propagation device; A second heat map acquisition means for acquiring a second heat map of the number of types of the detection objects generated based on the correct labels of the detection objects; a loss calculation means for calculating a difference between the first heat map and the second heat map; an update means for updating weight values ​​of the convolutional layer for generating the first heat map in the forward propagation device based on the calculated difference; has.

[0011] A processing method according to a fourth aspect of the present disclosure includes: A processing method for a neural network forward propagation device, comprising: Generate a binary mask, Performing calculations on a sparse convolutional layer according to the values ​​of each coordinate of the binary mask; In the process of generating the binary mask, A convolutional layer is operated on the input feature map to generate a heat map of the number of types of detection targets of the neural network; The heat map values ​​are summed for each coordinate to generate a composite heat map, which is a composite of the heat maps for the number of detection target types. The binary mask is generated by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold value.

[0012] A program according to the fifth aspect of the present disclosure includes: The neural network forward propagation device computer a mask generation step for generating a binary mask; A layer execution step of performing an operation of a sparse convolution layer according to the value of each coordinate of the binary mask; Run the command, In the mask generating step, A convolutional layer is operated on the input feature map to generate a heat map of the number of types of detection targets of the neural network; The heat map values ​​are summed for each coordinate to generate a composite heat map, which is a composite of the heat maps for the number of detection target types. The binary mask is generated by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold value. Effect of the Invention

[0013] According to the present disclosure, it is possible to provide a novel technique that can achieve high-speed processing by omitting some of the product-sum operations in the convolution layer. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an example of a configuration of a forward propagation device according to a first embodiment. [Diagram 2] FIG. 1 is a schematic diagram showing the computation of a single sparse convolutional layer by a layer execution unit. [Diagram 3] FIG. 1 is a schematic diagram showing the computation of multiple sparse convolutional layers by a layer execution unit. [Figure 4] 11 is a block diagram showing an example of the configuration of a binary mask generation unit and a layer execution unit. FIG. [Diagram 5] 11 is a block diagram showing an example of the configuration of a binary mask generation unit and a layer execution unit. FIG. [Figure 6] FIG. 2 is a block diagram showing a specific configuration example of a mask unit. [Figure 7] 1 is a block diagram showing an example of a configuration of a learning device according to a first embodiment. [Figure 8] FIG. 13 is a schematic diagram for explaining calculation of loss. [Figure 9] 11 is a schematic diagram showing an example of generation of a correct heat map by a heat map generating unit; FIG. [Figure 10] 4 is a flowchart showing an example of the operation of the learning device according to the first embodiment. [Figure 11] 2 is a block diagram showing an example of a hardware configuration of a forward propagation device and a learning device according to the first embodiment. FIG. [Figure 12] FIG. 11 is a block diagram showing an example of a configuration of an information processing system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Before describing the embodiment, we will first explain the calculation in the convolution layer. As mentioned above, CNN is mainly composed of a convolution layer. Here, X(x,y,z) is the pixel value of the input feature map X at the (x,y,z) coordinate, F(x,y,z,w) is the value of the weight F, B(z) is the value of the bias B, f act is the activation function, and Z(x,y,z) is the pixel value of the output feature map Z. In addition, in the (x,y,z) coordinates of the input feature map X(x,y,z) and the output feature map Z(x,y,z), x indicates the position in the width direction, y indicates the position in the height direction, and z indicates the position in the channel direction. In this case, a convolution layer with a kernel whose size in the width direction and the size in the height direction are both K, that is, a convolution layer with a kernel size of K×K, is defined by the following equation.

[0016] <Formula (1)> TIFF0007670142000001.tif21162

[0017] Here, K', X, Z, F, and B satisfy the following: TIFF0007670142000002.tif6157W is the width of the feature map, and H is the height of the feature map. in is the number of input feature maps, C outis the number of output feature maps. In the convolutional layer, the bias is added to the result of the multiply-accumulate (MAC: Multiply Accumulation) operation between the input feature map and the weights, and then the activation function is applied. The activation function f act Generally, the ReLU (Rectified Linear Unit) function is used.

[0018] In a normal convolutional layer, if we count the multiplications and additions by Σ separately, it takes 2KKC for one output feature map pixel value Z(x,y,z). in In the convolution layer, each coordinate position requires multiplication and accumulation operations. The same number of multiply-and-accumulate operations (specifically 2KKC) are performed on TIFF0007670142000003.tif433. in Since the multiplication and accumulation operations are repeated 2KKC times, in HWC out multiply-and-accumulate operations are required.

[0019] In CNN, the huge number of product-sum operations included in the convolution layer is the cause of the long execution time. In this embodiment, the calculation speed of the convolution layer is increased by omitting some of these product-sum operations. Specifically, in order to process CNN at high speed, product-sum operations corresponding to the background part of the feature map are omitted.

[0020] <Embodiment 1> First, a description will be given of a forward propagation apparatus 100 according to an embodiment. The forward propagation apparatus 100 is a forward propagation apparatus for a neural network, and is an apparatus for performing image recognition such as object recognition, semantic region segmentation, and human pose estimation using the neural network.

[0021] 1, the forward propagation device 100 has a binary mask generation unit 101 and a layer execution unit 102. The forward propagation device 100 sets parameters of the binary mask generation unit 101 and the layer execution unit 102 using sparse CNN weights 90, and then performs forward propagation calculation of the sparse CNN using an input image 91, and outputs an output result 92. Here, in the inference phase, weights learned by a learning device 200 (described later) are used as the sparse CNN weights 90 (i.e., weights of each layer of a mask unit 110 and the layer execution unit 102 (described later)).

[0022] The binary mask generator 101 executes a network called a mask unit 110 (see FIG. 4, FIG. 5, and FIG. 6) to generate a binary mask. The layer execution unit 102 executes calculations for each layer of the neural network, including a sparse convolutional layer. When the layer execution unit 102 executes a sparse convolutional layer, it uses the binary mask to perform a multiply-and-accumulate operation only on the foreground part of the image.

[0023] Here, a method of calculating a sparse convolutional layer by the layer execution unit 102 will be described with reference to FIG. 2 and FIG. 3. FIG. 2 shows an example in which the layer execution unit 102 executes only one layer of a sparse convolutional layer (SpConv) using a binary mask created by the binary mask generation unit 101. The binary mask is a two-dimensional image in which each coordinate takes a value of either 0 or 1, and in this embodiment, 1 (non-zero value) is a pixel for which calculation is performed, and 0 (zero value) is a pixel for which calculation is omitted. The size of the binary mask M is the same as the size of the output feature map Z. Note that, since the size of the convolution stride is set to 1 here (because the input feature map X and the output feature map Z are the same size), the size of the binary mask M is also the same as the size of the input feature map X.

[0024] One layer of the sparse convolution using a binary mask performed in the layer execution unit 102 can be defined by the following equation:

[0025] <Formula (2)> TIFF0007670142000004.tif38161

[0026] where Y is a temporary calculation value in the calculation process, and M is a binary mask. The other symbols in equation (2) are the same as in equation (1). In the sparse convolution layer, M(x,y) is the output feature map Z(x,y,z) of the coordinates (x,y) with non-zero values, and the 2KKC in times. On the other hand, for the output feature map of coordinates where M(x,y) has a value of zero, the product-sum operation is not performed, and the value of the output feature map is 0. In this way, the layer execution unit 102 performs the operation of the sparse convolution layer according to the value of each coordinate of the binary mask. That is, the layer execution unit 102 performs the operation of the sparse convolution layer by performing the product-sum operation for coordinates having a value other than a predetermined value (zero value) in the binary mask, and omitting the product-sum operation for coordinates having a predetermined value (zero value) in the binary mask.

[0027] Note that it takes real values ​​(e.g., floating point numbers). Using TIFF0007670142000005.tif527, M f (x,y) may be multiplied by Y(x,y,z) to calculate an output feature map for coordinates (x,y) where M(x,y) has a non-zero value.

[0028] <Formula (3-1)> TIFF0007670142000006.tif13113

[0029] Here, for example, M f The value of M is set to a large value for relatively important image regions among the foreground regions (image regions where M is not zero), and set to a small value for relatively unimportant image regions. f Specifically, the value of M may be set using the map value (the map value before binarization) output by the resolution adjustment unit 113 described later. f Multiplying by has the role of highlighting more noteworthy areas in the foreground region, leading to improved accuracy.

[0030] Also, take real values Using TIFF0007670142000007.tif523, M f '(z) may be multiplied by Y(x,y,z) to calculate the output feature map for the coordinates (x,y) where M(x,y) has a non-zero value.

[0031] <Formula (3-2)> TIFF0007670142000008.tif13110

[0032] Here, for example, M f The value of M' is set to a large value for a relatively important channel and a small value for a relatively unimportant channel. f Multiplying by ' serves to highlight channels that are of more interest in the foreground region, leading to improved accuracy. f and M f ' may be calculated by performing calculations of a convolutional layer and a pooling layer on the mask heat map described later.

[0033] In the example shown in FIG. 2, the binary mask generating unit 101 supplies a binary mask generated by a predetermined series of processes from the convolution layer executing unit 111 to the binarization unit 114, which will be described later, for each layer of the convolution layer. However, in order to reduce the time required for generating a binary mask, as shown in FIG. 3, a binary mask generated by the above-mentioned predetermined series of processes may be supplied for each of a plurality of layers of the convolution layer (for example, every two or three layers). That is, as shown in FIG. 3, a binary mask generated by the above-mentioned predetermined series of processes may be supplied for each group (block) consisting of a plurality of consecutive convolution layers. Note that FIG. 3 shows an example in which a mask generated by a predetermined series of processes is supplied for every two layers of the convolution layer.

[0034] In the example shown in FIG. 3, the first convolution layer performs convolution using 38 input pixels and weights in each (x, y) plane, and 16 output pixels are calculated. Then, in the second convolution layer, 16 input pixels and weights are used to calculate two output pixels. That is, in the example shown in FIG. 3, the final layer only needs to calculate two pixels, but in order to do so, 16 pixels must be calculated in the previous layer. In this way, the closer to the input layer, the larger the image area must be calculated. For this reason, a binary mask generated by a predetermined series of processes and a binary mask obtained by expanding the non-zero image area in this binary mask are used as described below. In the following description, the binary mask obtained by expanding the non-zero image area in the binary mask is referred to as the expanded binary mask.

[0035] When a binary mask M generated by a series of predetermined processes is supplied for each of a plurality of layers, this binary mask M is used for the sparse convolutional layer that is the final layer in the group (the second layer in the example of FIG. 3). As described above, in the convolution of the layer before the final layer, a multiplication and accumulation operation between an image area larger than the non-zero image area in the binary mask M used in the final layer and weights must be calculated, and the binary mask M cannot be used. Therefore, the binary mask generation unit 101 generates a binary mask M' by expanding the binary mask M of the final layer according to the kernel size and stride of the convolution. However, the expanded binary mask is generated by performing an expansion process on the binary mask to be expanded without performing a series of predetermined processes from the convolution layer execution unit 111 to the binarization unit 114, which will be described later. In this expansion process, the binary mask generation unit 101 generates a binary mask as the expanded binary mask M' in which the coordinates (x, y) of the pixel of the input feature map required for calculating the output feature map corresponding to the binary mask M to be expanded are non-zero values.

[0036] When the number of layers in the group is three, the binary mask generation unit 101 uses the expanded binary mask M' as the binary mask to be expanded and performs further expansion processing to generate an expanded binary mask M''. Similarly, when the number of layers in the group is four, the binary mask generation unit 101 uses the expanded binary mask M'' as the binary mask to be expanded and performs further expansion processing to generate an expanded binary mask M''. In this way, the binary mask generation unit 101 repeats the expansion processing a number of times according to the number of layers in the group, and generates expanded binary masks that are one less than the number of layers.

[0037] The binary mask generation unit 101 will be described with reference to Fig. 4 to Fig. 6. Fig. 4 and Fig. 5 are block diagrams showing configuration examples of the binary mask generation unit 101 and the layer execution unit 102. Fig. 4 and Fig. 5 show the supply of an input feature map to the mask unit 110 and the sparse convolution layer (SpConv) 120, and the supply of a binary mask generated by the mask unit 110 to the sparse convolution layer (SpConv) 120. Fig. 6 is a block diagram showing a specific configuration example of the mask unit 110.

[0038] As shown in FIG. 4, the binary mask generating unit 101 includes m mask units 110 (m is an integer equal to or greater than 1), and performs calculations for each mask unit 110. An input to the mask unit 110 is an input feature map to be input to a sparse convolutional layer 120 that is executed first in a group of sparse convolutional layers 120 that uses the binary mask supplied by the mask unit 110. Note that a group does not necessarily have to include multiple sparse convolutional layers 120, and the group may include only one sparse convolutional layer 120. The mask unit 110 outputs a binary mask M for a group of sparse convolutional layers 120 according to the resolution (resolution defined by height H and width W) of the output feature map of the sparse convolutional layer 120 of the group. The mask unit 110 also outputs J mask heat maps each having a height H and a width W. The binary mask generation unit 101 outputs TIFF0007670142000009.tif422. The mask heat map is a two-dimensional image with height H and width W generated by a convolution layer execution unit 111 (described later) of the binary mask generation unit 101, and is used to calculate a loss function in a mask loss calculation unit 206 (described later) of the learning device 200.

[0039] Fig. 6 is a block diagram showing an example of the internal structure of the mask unit 110. As shown in Fig. 6, each mask unit 110 includes a convolution layer execution unit 111, a summation processing unit 112, a resolution adjustment unit 113, a binarization unit 114, and an extended mask generation unit 115.

[0040] The convolution layer execution unit 111 executes convolution on the input feature map and outputs J feature maps (mask heat maps). That is, the convolution layer execution unit 111 executes a convolution layer designed to generate J binary images from the input feature map. In the mask loss calculation unit 206 of the learning device 200 described later, it is necessary to match the number of feature maps between the mask heat map output by the convolution layer execution unit 111 and the correct heat map. For this reason, the number of feature maps J of the mask heat map output by the convolution layer execution unit 111 is equal to the number of types of detection targets of the neural network of the forward propagation device 100. Here, the number of types of detection targets is specifically, for example, the number of types of human joint points whose coordinates are detected by the neural network when human pose estimation is performed by the forward propagation device 100. Also, when object detection is performed by the forward propagation device 100, the number of types of detection targets is the number of classes (the number of types of objects) used in object detection by the neural network. Furthermore, when semantic region segmentation is performed by the forward propagation device 100, the number of types of detection targets is the number of classes (the number of types of meanings) used in semantic region segmentation by a neural network. In this way, the convolutional layer execution unit 111 performs a convolutional layer operation on the input feature map to generate a mask heat map whose number is the number of types of detection targets of the neural network.

[0041] When the forward propagation device 100 performs inference, the weight values ​​of the convolutional layer for the mask unit 110 to generate J heat maps are values ​​machine-learned using a heat map generated based on a correct label, as described below. That is, in the inference phase, the weight values ​​of the convolutional layer executed by the convolutional layer execution unit 111 are values ​​machine-learned using a heat map generated based on a correct label.

[0042] The summing unit 112 sums up the values ​​of the J feature maps (mask heat maps) generated by the convolutional layer executing unit 111 for each coordinate. In this way, the summing unit 112 generates a feature map (sometimes referred to as a composite heat map in this disclosure) in which the J feature maps (mask heat maps) are combined into one feature map (mask heat map). In the summing process of the summing unit 112, for example, the sum of the J feature maps is calculated as shown in the following formula (4).

[0043] <Formula (4)> TIFF0007670142000010.tif1956

[0044] However, M hLet Z be the input feature map (mask heat map) input to the summation processing unit 112, and Z' be the output feature map output from the summation processing unit 112. The summation processing may not be a simple summation as shown in formula (4). For example, the summation processing unit 112 may calculate a weighted sum by weighting J feature maps. For example, a small weight is set for a relatively unimportant class (a class that appears less frequently in the data set), and a large weight is set for a relatively important class (a class that appears less frequently in the data set). A relatively unimportant class is, for example, a class that appears less frequently than other classes in a data set used by the learning device 200 described later, and can be specified in advance. Similarly, a relatively important class is a class that appears more frequently than other classes in a data set, and can be specified in advance. For a class that is known in advance to be unimportant, a weight value of 0 may be set to ignore this class. By calculating the weighted sum in this way, it may be possible to omit the calculation of the image area of ​​the unimportant class, thereby enabling further speedup.

[0045] The resolution adjustment unit 113 changes the resolution of the feature map (synthesized heat map) output by the summation processing unit 112 according to the resolution of the sparse convolutional layer. That is, after the summation processing, the resolution of the synthetic heat map is changed to the required resolution (the resolution of the output feature map of each sparse convolutional layer 120 to which the mask unit 110 supplies a binary mask) by the resolution adjustment processing by the resolution adjustment unit 113. For example, when generating a mask larger than the height H and width W of the input feature map (output feature map of the summation processing unit 112) of the resolution adjustment unit 113, the resolution adjustment unit 113 performs upsampling (upconversion) using nearest neighbor interpolation or the like. In addition, when generating a mask smaller than the height H and width W of the input feature map, the resolution adjustment unit 113 performs downsampling (downconversion) using average value pooling, maximum value pooling or the like. The required resolution can be specified from the parameters of the sparse convolutional layer 120. In addition, the resolution adjustment unit 113 can omit processing when resolution adjustment is not required.

[0046] In this way, since feature maps with various resolutions can be generated by the resolution adjustment unit 113, various resolutions can be supported by one mask unit 110. As a result, one mask unit 110 can supply binary masks to more sparse convolution layers (for example, about ten layers).

[0047] The binarization unit 114 generates a binary mask by binarizing the coordinate values ​​of the resolution-adjusted mask (synthesized heat map) using a predetermined threshold value γ. in It should be noted that, when resolution adjustment is not required, the binarization unit 114 converts the mask output by the summation processing unit 112 into M in Let us assume that.

[0048] <Formula (5)> TIFF0007670142000011.tif1367

[0049] The extended mask generating unit 115 performs the above-mentioned extension process on the binary mask generated by the binarization unit 114, thereby generating a binary mask by extending a non-zero image region in the binary mask generated by the binarization unit 114. In this manner, the extended mask generating unit 115 does not generate a binary mask by performing again a series of processes from the convolution layer executing unit 111 to the binarization unit 114, but generates a new binary mask using the binary mask output by the binarization unit 114. As described above, the extended mask generating unit 115 may further generate a new binary mask by performing an extension process on the binary mask obtained by the extension process.

[0050] As described above, the binary mask output by the binarization unit 114 is generated for the operation of the final layer of the subsequent sparse convolutional layer 120, as in the second layer in FIG. 3. The extended mask generation unit 115 generates a binary mask for the operation of the sparse convolutional layer preceding the final layer, from the binary mask generated by the binarization unit 114. Furthermore, as necessary, the extended mask generation unit 115 generates a binary mask for the operation of the sparse convolutional layer preceding the final layer. With this configuration, it is possible to reduce the time required for the generation process of the binary mask for each sparse convolutional layer.

[0051] The binarization unit 114 and the extended mask generation unit 115 supply the generated binary mask to the sparse convolutional layer 120. Note that the sparse convolutional layer 120 is not limited to being connected in series as in Fig. 4, but may be connected in parallel as in Fig. 5, for example, and the mask unit 110 can generate and supply a binary mask for any CNN structure.

[0052] Next, a description will be given of a learning device 200 for the forward propagation device 100. The learning device 200 learns parameters of a convolution layer executed by the convolution layer execution unit 111 of the mask unit 110 of the forward propagation device 100 and parameters of a neural network executed by the layer execution unit 102.

[0053] 7 is a block diagram showing an example of the configuration of a learning device 200. The learning device 200 uses a sparse CNN model structure 93 and a data set 94 as input, and outputs learned weights 95. The sparse CNN model structure 93 includes data representing the structure of the mask unit 110 of the binary mask generation unit 101 and data representing the CNN structure executed by the layer execution unit 102. The data set 94 is image data to which a correct answer label has been assigned. The learned weights 95 include learned values ​​of weights used in the convolution layer execution unit 111 of the mask unit 110 and learned values ​​of weights used in the CNN executed by the layer execution unit 102.

[0054] 7, the learning device 200 has an initialization unit 201, a preprocessing unit 202, a forward propagation unit 203, a loss calculation unit 204, a heat map generation unit 205, a mask loss calculation unit 206, a parameter update unit 207, and a learning end determination unit 208. Learning by the learning device 200 is particularly different from general learning procedures in that it includes processing by the heat map generation unit 205 and processing by the mask loss calculation unit 206.

[0055] The initialization unit 201 initializes parameters such as the weights of the sparse CNN. The initialization by the initialization unit 201 is not only the weights of the sparse convolutional layer 120 executed by the layer execution unit 102, but also the initialization of all learning parameters of the sparse CNN, such as the weights in the convolutional layer execution unit 111 of the mask unit 110.

[0056] The preprocessing unit 202 performs predetermined preprocessing (eg, image rotation processing, distortion correction processing, etc.) on an input image (an image included in the data set 94) input to the learning device 200 for learning.

[0057] The forward propagation unit 203 performs forward propagation processing in the forward propagation device 100 by using the input image preprocessed by the preprocessing unit 202 and the sparse CNN model structure 93 (i.e., the forward propagation device 100). This allows forward propagation processing such as calculations for each sparse convolution layer 120 and each mask unit 110 to be performed.

[0058] Next, the loss calculation unit 204, the heat map generation unit 205, and the mask loss calculation unit 206 will be described with reference to Fig. 8. Fig. 8 is a schematic diagram for explaining the calculation of the loss.

[0059] The loss calculation unit 204 calculates a first loss function Loss main The first loss function Loss main is the same as the loss function generally calculated in CNN. The first loss function Loss main is a loss function that calculates a loss using the correct label included in the dataset 94 and the output feature map output by the layer execution unit 102 (i.e., the output feature map output by the forward propagation device 100).

[0060] The heat map generating unit 205 calculates the second loss function Loss mask The ground truth heat map required for the calculation of Generate TIFF0007670142000012.tif45. Second loss function Loss mask is a loss function calculated to learn the weights of the mask unit 110, and is calculated by the mask loss calculation unit 206 described later. The heat map generation unit 205 generates a correct answer heat map by arranging a two-dimensional normal distribution so as to correspond to the coordinate position specified by the correct answer label. More specifically, the heat map generation unit 205 generates a heat map (two-dimensional image) in which a two-dimensional normal distribution is arranged, and in which the maximum value of the normal distribution exists at the position of the detection target.

[0061] 9 is a schematic diagram showing an example of generating a correct heat map by the heat map generating unit 205. For example, when human posture estimation is performed by the forward propagation device 100, a correct heat map is generated by arranging a normal distribution so that the maximum value of the two-dimensional normal distribution overlaps with the joint point coordinates indicated by the correct label. TIFF0007670142000013.tif45 can be generated. That is, in this case, the heat map generating unit 205 generates a heat map in which the maximum value of the two-dimensional normal distribution exists at the coordinates of the human joint points indicated in the correct label for human pose estimation as the correct heat map. Note that in the case of human pose estimation, the number of feature maps J of the correct heat map is the same as the number of joint points to be detected. That is, a feature map corresponding to each joint point is generated.

[0062] In addition, for example, when performing object detection using the forward propagation device 100, the two-dimensional normal distribution is transformed to fit the rectangle (bounding box surrounding the object) indicated by the correct label, and the normal distribution is arranged so that the maximum value of the normal distribution overlaps with the position (center of gravity) within the bounding box, thereby generating a correct heat map. TIFF0007670142000014.tif45 can be generated. In the deformation of the two-dimensional normal distribution, the variance is adjusted. For example, as shown in FIG. 9, since the rectangle corresponding to a person is vertically long, in the generation of the correct answer heat map for the detection of a person, the two-dimensional normal distribution is deformed to increase the variance in the y direction. In this way, when performing object detection using the forward propagation device 100, the heat map generating unit 205 generates a heat map in which the maximum value of the two-dimensional normal distribution exists as the correct answer heat map within the rectangular area indicated by the correct answer label for object detection. In the case of object detection, the number of feature maps J of the correct answer heat map is the same as the number of classes (the number of types of objects) used in the object detection. In other words, a feature map corresponding to each class is generated.

[0063] Furthermore, when performing semantic region segmentation using the forward propagation device 100, the heat map generation unit 205 first calculates a rectangle surrounding the object region of each object indicated by the correct label. The heat map generation unit 205 then transforms the two-dimensional normal distribution to fit this rectangle, arranges the normal distribution so that the maximum value of the normal distribution coincides with a position (center of gravity) within the rectangle, and further performs trimming according to the correct label to generate a correct heat map. TIFF0007670142000015.tif45 is generated. That is, the heat map generating unit 205 generates a heat map in which the value of the coordinate corresponding to the outside of the object area indicated by the correct answer label is zero. Note that, in the case where semantic area division is performed by the forward propagation device 100, the transformation process of the two-dimensional normal distribution by the heat map generating unit 205 is the same as that in the case where object detection is performed by the forward propagation device 100. In this way, in the case where semantic area division is performed by the forward propagation device 100, the heat map generating unit 205 performs the following process. The heat map generating unit 205 generates, as the correct answer heat map, a heat map in which the maximum value of the two-dimensional normal distribution exists within a rectangular area surrounding the area indicated by the correct answer label for semantic area division and the value outside the area indicated by the correct answer label is set to zero. Note that, in the case of semantic area division, the number of feature maps J of the correct answer heat map is the same as the number of classes (the number of types of meaning) used in the semantic area division. That is, a feature map corresponding to each class is generated.

[0064] The mask loss calculation unit 206 calculates the second loss function Loss mask As shown in Figure 8, Loss mask is a mask heat map generated by m mask units 110 included in the sparse CNN (forward propagation device 100). TIFF0007670142000016.tif532 and the correct heat map generated by the heat map generating unit 205 The mask loss calculation unit 206 calculates the loss using the second loss function Loss mask To calculate the loss, the mask loss calculation unit 206 calculates the difference between the mask heat map generated by the mask unit 110 and the correct heat map generated by the heat map generation unit 205. The mask loss calculation unit 206 calculates the loss expressed by the following formula (6) using, for example, a mean squared error (MSE). mask Calculate.

[0065] <Formula (6)> TIFF0007670142000018.tif21102

[0066] Where: TIFF0007670142000019.tif57 is a mask heat map of the j-th mask unit 110 among the m mask units 110; The file is TIFF0007670142000020.tif547.

[0067] Overall loss function for sparse CNN all As shown in the following equation (7), Loss main and Loss mask It is defined as a weighted sum of

[0068] <Formula (7)> TIFF0007670142000021.tif687

[0069] Here, α(0<α) is the loss mask and Loss main This is a parameter for adjusting the balance between the loss calculated using the correct label as in this embodiment. mask Using this method, it is possible to accurately learn a two-dimensional binary mask that represents the pixels on which the convolution is performed.

[0070] The parameter update unit 207 performs error backpropagation using the loss function shown in equation (7) to update parameters such as the weights of the sparse CNN. Furthermore, the learning end determination unit 208 determines whether or not a predetermined learning end condition is satisfied, and when the learning end condition is satisfied, performs control so as to output the learned parameter (weight) value.

[0071] 10 is a flowchart showing an example of the operation of the learning device 200. Hereinafter, the flow of processing by the components of the learning device 200 described above will be described with reference to FIG.

[0072] In step S1, prior to carrying out learning, the learning device 200 reads out the sparse CNN model structure 93 from the storage device. Next, in step S2, the initialization unit 201 initializes parameters (weights). Next, in step S3, the preprocessing unit 202 performs a predetermined preprocessing on the input image. Next, in step S4, the forward propagation unit 203 performs a forward propagation process using the preprocessed input image and the sparse CNN model structure 93. In step S5, the loss calculation unit 204 calculates a first loss function Loss main Then, in step S6, the heat map generation unit 205 generates a correct heat map based on the correct labels included in the dataset 94. In step S7, the mask loss calculation unit 206 calculates the second loss function Loss mask Then, in step S8, the parameter update unit 207 updates the parameters by performing error backpropagation. Next, in step S9, the learning end determination unit 208 determines whether a predetermined learning end condition is satisfied. For example, the learning end determination unit 208 determines whether a series of processes has been repeated a preset number of times. If the predetermined learning end condition is not satisfied, the process returns to step S3. As a result, a series of processes is repeated for various images in the data set 94. If the predetermined learning end condition is satisfied, the process proceeds to step S10. In step S10, the learning device 200 outputs parameters such as the weights of the sparse CNN (the weights of the convolution layer of the mask unit 110 and the weights of the sparse convolution layer of the layer execution unit 102) as learned weights 95. Therefore, when the forward propagation device 100 performs inference, the weights learned by the learning device 200 are used.

[0073] Next, a description will be given of an example of the hardware configuration of the forward propagation device 100 and the learning device 200. Fig. 11 is a block diagram showing an example of the hardware configuration of the forward propagation device 100 and the learning device 200. As shown in Fig. 11, the forward propagation device 100 and the learning device 200 include an input / output interface 301, a memory 302, and a processor 303.

[0074] The input / output interface 301 is an interface for connecting to other devices so as to be able to communicate with them as necessary.

[0075] The memory 302 is configured, for example, by a combination of a volatile memory and a non-volatile memory. The memory 302 is used to store software (computer programs) including one or more instructions executed by the processor 303, data used for various processes, and the like.

[0076] The processor 303 performs processing of each of the above-mentioned components by reading and executing software (computer programs) from the memory 302. The processor 303 may be, for example, a microprocessor, a microprocessor unit (MPU), or a central processing unit (CPU). The processor 303 may include multiple processors. In this way, the forward propagation device 100 and the learning device 200 have the functionality of a computer.

[0077] The program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0078] The embodiment has been described above. According to the present embodiment, the binary mask generated by the binary mask generating unit 101 makes it possible to realize high-speed processing by omitting part of the product-sum calculation in the sparse convolution layer 120. More specifically, it is possible to perform the product-sum calculation for the region where the detection target exists (i.e., the foreground region) and omit the product-sum calculation for the other regions (background region). In particular, the parameters for generating the binary mask are learned using the correct answer label. More specifically, the neural network that generates the binary mask corresponding to the shape of the detection target is learned by comparing the heat map generated based on the shape of the object shown in the correct answer data with the heat map generated by the neural network. Therefore, it is possible to specify the foreground region with high accuracy. Also, as described above, the mask unit 110 includes the resolution adjusting unit 113 and the extended mask generating unit 115, and can generate various binary masks while suppressing the processing load. Therefore, it is possible to suppress the time required for generating the binary mask. Furthermore, in this embodiment, the binary mask is generated by the mask unit 110, which has a simpler structure than the CNN model, and therefore the processing speed can be improved compared to the case where a binary mask is generated by the CNN model.

[0079] <Embodiment 2> Next, a second embodiment will be described. The second embodiment is an embodiment configured from characteristic elements of the first embodiment. FIG. 12 is a block diagram showing an example of a configuration of an information processing system 1 according to the second embodiment. The information processing system 1 includes a forward propagation device 2 of a neural network and a learning device 3 for the forward propagation device 2.

[0080] The forward propagation device 2 includes a mask generation unit 4 that generates a binary mask, and a layer execution unit 5 that performs a sparse convolutional layer operation according to the value of each coordinate of the binary mask. In particular, the mask generation unit 4 generates a binary mask as follows. The mask generation unit 4 generates a predetermined number of first heat maps by performing a convolutional layer operation on the input feature map. Here, the predetermined number is the number of types of detection targets of the neural network. Next, the mask generation unit 4 generates a composite heat map in which the predetermined number of first heat maps are combined into one heat map by summing the values ​​of the predetermined number of first heat maps for each coordinate. Then, the mask generation unit 4 generates a binary mask by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold value.

[0081] The learning device 3 includes a first heat map acquisition unit 6, a second heat map acquisition unit 7, a loss calculation unit 8, and an update unit 9. The first heat map acquisition unit 6 acquires the predetermined number of first heat maps generated by the forward propagation device 2. The second heat map acquisition unit 7 acquires the predetermined number of second heat maps generated based on the correct label of the detection target. The loss calculation unit 8 calculates the difference between the first heat map acquired by the first heat map acquisition unit 6 and the second heat map acquired by the second heat map acquisition unit 7. Then, the update unit 9 updates the weight values ​​of the convolution layer for generating the first heat map in the forward propagation device 2 based on the difference calculated by the loss calculation unit 8.

[0082] According to this embodiment, the learning device 3 performs learning to generate a binary mask, and the forward propagation device 2 uses this binary mask to realize high-speed processing that omits some of the product-sum operations.

[0083] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention.

[0084] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) A forward propagation device for a neural network, comprising: a mask generating means for generating a binary mask; A layer execution means for performing an operation of a sparse convolution layer according to the value of each coordinate of the binary mask; having The mask generating means A convolutional layer is operated on the input feature map to generate a heat map of the number of types of detection targets of the neural network; The heat map values ​​are summed for each coordinate to generate a composite heat map, which is a composite of the heat maps for the number of detection target types. The binary mask is generated by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold value. Forward propagation device. (Appendix 2) The weight values ​​of the convolutional layer for generating a heat map of the number of types of detection targets are values ​​machine-learned using a heat map generated based on the correct answer label. 2. A forward propagation apparatus as described in appended claim 1. (Appendix 3) The mask generating means further varies a resolution of the composite heat map in response to a resolution of the sparse convolutional layer. 3. A forward propagation apparatus as claimed in claim 1 or 2. (Appendix 4) The mask generation means generates the binary mask for the operation of a final layer of the successive sparse convolutional layers, and further generates, from the generated binary mask, a binary mask for the operation of the sparse convolutional layer preceding the final layer. 4. A forward propagation device according to claim 1 . (Appendix 5) The number of types of detection objects is the number of types of human joint points whose coordinates are detected by the neural network. 5. A forward propagation device according to any one of claims 1 to 4. (Appendix 6) The number of types of detection targets is the number of classes used in object detection by the neural network. 5. A forward propagation device according to any one of claims 1 to 4. (Appendix 7) The number of types of detection targets is the number of classes used in semantic region segmentation by the neural network. 5. A forward propagation device according to any one of claims 1 to 4. (Appendix 8) A learning device for a forward propagation device of a neural network, comprising: a first heat map acquisition means for acquiring a first heat map of the number of types of detection targets of the neural network, the first heat map being generated by the forward propagation device; A second heat map acquisition means for acquiring a second heat map of the number of types of the detection objects generated based on the correct labels of the detection objects; a loss calculation means for calculating a difference between the first heat map and the second heat map; an update means for updating weight values ​​of a convolutional layer for generating the first heat map in the forward propagation device based on the calculated difference; having The forward propagation device is a device that generates a binary mask and performs an operation of a sparse convolution layer according to the values ​​of each coordinate of the generated binary mask; The forward propagation device, in the process of generating the binary mask, A convolutional layer is operated on the input feature map to generate the first heat map of the number of types of detection targets of the neural network; generating a composite heat map by summing the values ​​of the first heat maps for the number of types of detection targets for each coordinate, and combining the first heat maps for the number of types of detection targets into one heat map; The binary mask is generated by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold value. Learning device. (Appendix 9) The method further includes a correct answer heat map generating means for generating a heat map as the second heat map by arranging a two-dimensional normal distribution so as to correspond to the coordinate position specified by the correct answer label. 9. The learning device according to claim 8. (Appendix 10) The correct heat map generating means generates, as the second heat map, a heat map in which a maximum value of the two-dimensional normal distribution exists at coordinates of human joint points indicated in the correct label for human pose estimation using the neural network. 10. The learning device according to claim 9. (Appendix 11) The correct answer heat map generating means generates, as the second heat map, a heat map in which a maximum value of the two-dimensional normal distribution exists within a rectangular area indicated by a correct answer label for object detection using the neural network. 10. The learning device according to claim 9. (Appendix 12) The correct answer heat map generating means generates, as the second heat map, a heat map in which the maximum value of the two-dimensional normal distribution exists within a rectangular region surrounding a region indicated by a correct answer label for semantic region segmentation using the neural network, and values ​​outside the region indicated by the correct answer label are set to zero. 10. The learning device according to claim 9. (Appendix 13) A neural network forward propagation device; a learning device for the forward propagation device; Equipped with The forward propagation device comprises: a mask generating means for generating a binary mask; A layer execution means for performing an operation of a sparse convolution layer according to the value of each coordinate of the binary mask; having The mask generating means A convolutional layer is operated on the input feature map to generate a first heat map of the number of types of detection targets of the neural network; generating a composite heat map by summing the values ​​of the first heat maps for the number of types of detection targets for each coordinate, and synthesizing the first heat maps for the number of types of detection targets into a single heat map; generating the binary mask by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold; The learning device includes: a first heat map acquisition means for acquiring the first heat map of the number of types of detection targets of the neural network, the first heat map being generated by the forward propagation device; A second heat map acquisition means for acquiring a second heat map of the number of types of the detection objects generated based on the correct labels of the detection objects; a loss calculation means for calculating a difference between the first heat map and the second heat map; an update means for updating weight values ​​of the convolutional layer for generating the first heat map in the forward propagation device based on the calculated difference; have Information processing system. (Appendix 14) The learning device further includes a correct answer heat map generating means for generating a heat map as the second heat map by arranging a two-dimensional normal distribution so as to correspond to a coordinate position specified by the correct answer label. 14. The information processing system according to claim 13. (Appendix 15) A processing method for a neural network forward propagation device, comprising: Generate a binary mask, Performing calculations on a sparse convolutional layer according to the values ​​of each coordinate of the binary mask; In the process of generating the binary mask, A convolutional layer is operated on the input feature map to generate a heat map of the number of types of detection targets of the neural network; The heat map values ​​are summed for each coordinate to generate a composite heat map, which is a composite of the heat maps for the number of detection target types. The binary mask is generated by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold value. Processing methods. (Appendix 16) The neural network forward propagation device computer a mask generation step for generating a binary mask; A layer execution step of performing an operation of a sparse convolution layer according to the value of each coordinate of the binary mask; Run the command, In the mask generating step, A convolutional layer is operated on the input feature map to generate a heat map of the number of types of detection targets of the neural network; The heat map values ​​are summed for each coordinate to generate a composite heat map, which is a composite of the heat maps for the number of detection target types. The binary mask is generated by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold value. A non-transitory computer-readable medium on which a program is stored. [Explanation of symbols]

[0085] 1. Information Processing Systems 2. Forward propagation device 3 Learning device 4 Mask generation section 5th Layer Executive 6 First heat map acquisition section 7 Second heat map acquisition section 8 Loss calculation section 9 Update section 91 Input images 92 Output results 93 Model Structure 94 Datasets 100 Forward propagation device 101 Binary Mask Generator 102 Layer Executive 110 Mask Unit 111 Convolutional layer execution unit 112 Total processing unit 113 Resolution adjustment section 114 Binarization section 115 Extended Mask Generator 120 Sparse Convolutional Layers 200 Learning Device 201 Initialization section 202 Pretreatment section 203 Forward propagation section 204 Loss calculation section 205 Heat map generator 206 Mask Loss Calculation Unit 207 Parameter Update Section 208 Learning completion determination unit 301 Input / Output Interface 302 Memory 303 Processor

Claims

1. A forward propagation device for a neural network, comprising: a mask generating means for generating a binary mask; A layer execution means for performing an operation of a sparse convolution layer according to the value of each coordinate of the binary mask; having The mask generating means A convolutional layer is operated on the input feature map to generate a plurality of heat maps corresponding to the number of types of detection targets of the neural network; generating a composite heat map by combining the heat maps into one heat map by summing the heat map values ​​for each coordinate; The binary mask is generated by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold value. Forward propagation device.

2. The weight values ​​of the convolutional layer for generating the plurality of heat maps are values ​​machine-learned using a plurality of heat maps corresponding to the number of types of the detection targets generated based on ground truth labels.

2. The forward propagation apparatus according to claim 1.

3. The mask generating means further varies a resolution of the composite heat map in response to a resolution of the sparse convolutional layer.

3. A forward propagation device according to claim 1 or 2.

4. The mask generation means generates the binary mask for the operation of a final layer of the successive sparse convolutional layers, and further generates, from the generated binary mask, a binary mask for the operation of the sparse convolutional layer preceding the final layer. A forward propagation device according to any one of claims 1 to 3.

5. The number of types of detection objects is the number of types of human joint points whose coordinates are detected by the neural network. A forward propagation device according to any one of claims 1 to 4.

6. The number of types of detection targets is the number of classes used in object detection by the neural network. A forward propagation device according to any one of claims 1 to 4.

7. The number of types of detection targets is the number of classes used in semantic region segmentation by the neural network. A forward propagation device according to any one of claims 1 to 4.

8. A learning device for a forward propagation device of a neural network, comprising: a first heat map acquisition means for acquiring a plurality of first heat maps generated by the forward propagation device, the first heat maps corresponding to the number of types of detection targets of the neural network; a second heat map acquisition means for acquiring a plurality of second heat maps corresponding to the number of types of the detection objects generated based on the correct labels of the detection objects; a loss calculation means for calculating a difference between the first heat map and the second heat map; an update means for updating weight values ​​of a convolutional layer for generating the first heat map in the forward propagation device based on the calculated difference; having The forward propagation device is a device that generates a binary mask and performs an operation of a sparse convolution layer according to the values ​​of each coordinate of the generated binary mask; The forward propagation device, in the process of generating the binary mask, A convolutional layer is operated on the input feature map to generate a plurality of the first heat maps corresponding to the number of types of detection targets of the neural network; generating a composite heat map by combining the first heat maps into one heat map by summing the values ​​of the first heat maps for each coordinate; The binary mask is generated by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold value. Learning device.

9. A processing method for a neural network forward propagation device, comprising: Generate a binary mask, Performing calculations on a sparse convolutional layer according to the values ​​of each coordinate of the binary mask; In the process of generating the binary mask, A convolutional layer is operated on the input feature map to generate a plurality of heat maps corresponding to the number of types of detection targets of the neural network; generating a composite heat map by combining the heat maps into one heat map by summing the heat map values ​​for each coordinate; The binary mask is generated by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold value. Processing methods.

10. The neural network forward propagation device computer a mask generation step for generating a binary mask; A layer execution step of performing an operation of a sparse convolution layer according to the value of each coordinate of the binary mask; Run the command, In the mask generating step, A convolutional layer is operated on the input feature map to generate a plurality of heat maps corresponding to the number of types of detection targets of the neural network; generating a composite heat map by combining the heat maps into one heat map by summing the heat map values ​​for each coordinate; The binary mask is generated by binarizing the values ​​of each coordinate of the composite heat map using a predetermined threshold value. program.

Images