Information processing apparatus, processing method, and program

Abstract

To solve a problem in which power required for accessing a memory is increased, and the cost for the memory is increased.SOLUTION: An information processing apparatus performs processing on feature data of a plurality of feature images according to a network structure having a plurality of layers. The information processing apparatus has data holding means that holds the feature data, first reading means that reads the feature data on the basis of a read-out pattern, enlargement means that enlarges the read-out feature data, and convolution processing means that performs convolution processing on the feature data enlarged by the enlargement means. The enlargement means and the convolution processing means perform the enlargement and convolution processing on the feature data of one feature image, and subsequently perform the enlargement and convolution processing on the feature data of the subsequent feature image.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing device, a processing method, and a program. [Background technology]

[0002] Convolutional Neural Networks (CNN) are known as a method used in deep learning. CNN has multiple layers connected hierarchically. CNN calculates image-formatted feature data for an input image in each layer and generates a feature image containing the feature data. CNN processes multiple feature images (channels) in each layer. For example, there is a CNN with four layers, each of which has one or multiple feature images. This CNN performs convolution processing using weight coefficients, which are learned filter coefficients, and feature data assigned to the pixels of the feature images. Convolution processing is a product-sum operation that includes multiple multiplications and cumulative additions.

[0003] When the positional accuracy of object detection results based on the output of CNN is important, when detecting the exact area of ​​an object in an image, or when applying CNN to image processing, in order to improve accuracy, it is necessary to increase the resolution of feature images by processing such as deconvolution. Deconvolution processing can be decomposed into enlargement processing, which increases the resolution, and convolution processing. In the enlargement processing of deconvolution processing, the resolution is increased by outputting zero values. Patent Document 1 discloses a device for enlarging an image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4069300 Summary of the Invention [Problem to be solved by the invention]

[0005] The technique described in Patent Document 1 can realize image enlargement processing, but it requires a line buffer to temporarily store the image, and there is a problem that the power required to access the memory increases in proportion to the size of the image, resulting in high memory costs. [Means for solving the problem]

[0006] In order to solve this problem, for example, an information processing device of the present invention has the following configuration. An information processing device that processes feature data of a plurality of feature images according to a network structure having a plurality of hierarchical layers, A data storage means for storing feature data; a first read means for reading the feature data based on a read pattern; an enlarging means for enlarging the read feature data; a convolution processing means for performing a convolution process on the feature data enlarged by the enlargement means, The enlargement means and the convolution processing means perform the enlargement and convolution processing on the feature data of one feature image, and then perform the enlargement and convolution processing on the feature data of the next feature image. It is characterized by: Effect of the Invention

[0007] According to the present invention, it is possible to reduce the power required for accessing the memory and reduce the cost of the memory. [Brief description of the drawings]

[0008] [Figure 1] 5 is a flowchart of a convolutional neural network process in the first embodiment. [Diagram 2] 1 shows an example of the structure of a convolutional network in the first embodiment. [Diagram 3] FIG. 2 is a block diagram showing an example of the configuration of a convolution processing device according to the first embodiment. [Figure 4]FIG. 2 is a block diagram of a CNN processing unit according to the first embodiment. [Diagram 5] FIG. 11 is a block diagram of a CNN processing unit according to the second embodiment. [Figure 6] 4 is an example showing the relationship between a network and convolution processing in the first embodiment. [Figure 7] 13 shows an example of enlargement processing and convolution processing in the first embodiment (filter size 3×3, enlargement ratio R=2). [Figure 8] 13 shows an example of enlargement processing and convolution processing in the first embodiment (filter size 3×3, enlargement magnification R=1). [Figure 9] FIG. 4 is a diagram showing the enlargement process of the deconvolution process in the first embodiment. [Figure 10] FIG. 13 is a diagram showing a nearest neighbor interpolation method according to another embodiment. [Figure 11] FIG. 13 is a diagram showing an interpolation method according to another embodiment. [Figure 12] 13 is a diagram showing Pixel Shuffle in another embodiment. [Figure 13] 13 shows an example of enlargement processing and convolution processing in the second embodiment (filter size 1×1, enlargement ratio R=2). [Figure 14] 13 is an example of enlargement processing and convolution processing in another embodiment (filter size 3×3, enlargement ratio R=2). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] <First embodiment> Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] <Example of configuration of data parallel processing device> 3 is a block diagram showing an example of the configuration of a convolution processing device according to the present invention. The convolution processing device is an example of an information processing device, such as a computer. The convolution processing device includes an input unit 301, a data storage unit 302, a communication unit 303, a display unit 304, a CNN processing unit 305, a CPU (Central Processing Unit) 306, a ROM (Read Only Memory) 307, a RAM (Random Access Memory) 308, and an image processing unit 309.

[0012] The input unit 301 is a device for inputting instructions and data from a user, and includes a keyboard, a pointing device, buttons, and the like.

[0013] The data storage unit 302 is a unit for storing image data, and is usually configured with a hard disk, a solid state drive (SSD), a flexible disk, a CD-ROM, a CD-R or DVD, a memory card, a CF card, a smart media, an SD card, a memory stick, an xD picture card, a USB memory, or the like. In addition to image data, the data storage unit 302 can also store programs and other data. Alternatively, a part of the RAM 308 described later may be used as the data storage unit 302. Alternatively, the data storage unit 302 may be configured virtually such that a storage device of a device connected via a communication unit 303 described later is used via the communication unit 303.

[0014] The display unit 304 is a device that displays images before and after image processing, or displays images such as a GUI, and generally uses a CRT or liquid crystal display. Alternatively, it may be an external display device connected by a cable or the like. Alternatively, the display unit 304 and the input unit 301 may be the same device, as in a known touch screen device. In that case, input via the touch screen is treated as input to the input unit 301.

[0015] The CPU 306 is a calculation processing device that executes a program and controls the operation of the entire device. The ROM 307 and the RAM 308 provide the CPU 306 with programs, data, a working area, and the like required for the processing. When a program required for the processing described below is stored in the data storage unit 302 or the ROM 307, the program is once loaded into the RAM 308 and then executed. When the convolution processing device receives a program via the communication unit 303, the program is once recorded in the data storage unit 302 and then loaded into the RAM 308, or is directly loaded from the communication unit 303 to the RAM 308 and then executed. Although FIG. 3 shows a configuration with one CPU 306, multiple CPUs 306 may be provided. In addition to the CPU 306, the convolution processing device may have a calculation processing device such as a GPU (Graphics Processing Unit), an MPU (Micro Processing Unit), an NPU (Neural Processing Unit), or a QPU (Quantum Processing Unit).

[0016] The image processing unit 309 receives a command from the CPU 306, reads out the image data written in the data storage unit 302, adjusts the range of pixel values, and writes the adjusted image data back to the RAM 308. The image processing unit 309 may be realized by, for example, another CPU or a calculation processing device such as a GPU.

[0017] The CNN processing unit 305 performs convolutional neural network processing (steps S101 to S114) including product-sum operations using the image processing results stored in the RAM 308 according to the flowchart in Fig. 1 described later, and outputs the processed results to the data storage unit 302 (or the RAM 308). The CNN processing unit 305 may be realized by another arithmetic processing device such as a CPU or a GPU.

[0018] The CPU 306 performs image processing or image recognition on the video image (in the case of multiple frames) based on the result of the convolutional neural network processing. The CPU 306 stores the result of the processed image processing or image recognition in the RAM 308.

[0019] The communication unit 303 is an I / F for communication between devices. Although the input unit 301, data storage unit 302, and display unit 304 are all included in one device in Fig. 3, some of these components may be connected via a communication path using a known communication method, and the whole may be configured in this way.

[0020] The system configuration of the device includes various other components in addition to those described above, but as they are not the focus of the present invention, a description of them will be omitted.

[0021] <Network to be processed> FIG. 2 shows an example of the structure of a convolutional neural network executed by the CNN processor. The CNN processor 305 executes the convolutional neural network to output a feature image 204 as a confidence map that indicates the likelihood of the position and area of ​​a detected object in an input image. The network structure includes information on each layer (connection relationships between layers, filter structure, filter size, bit width of weighting coefficients, size, bit width, number of feature images, etc.). This network has four layers (input layer and layers 1-3), and each layer has multiple feature images. One feature image has multiple feature data. The multiple input images and feature images correspond to multiple channels. There are three input images, each consisting of three channels: R (red), G (green), and B (blue). The filters and multiple feature data are hierarchically structured.

[0022] The feature image in the current layer is calculated using the feature image in the previous layer and the weighting coefficient corresponding to the previous layer. To calculate one feature image in the current layer, information on multiple feature images in the previous layer is required. The calculation formula for the convolution process to generate a feature image is Equation 1.

[0023]

number

[0024] The calculations in each layer will be described. In the input layer, the CNN processing unit 305 calculates the results of the convolution process, activation process, and pooling process using three input images 200 and weighting coefficients based on Equation 1, and generates multiple feature images 201 in layer 1. In layer 1, the CNN processing unit 305 calculates the results of the convolution process and activation process using multiple feature images 201 and weighting coefficients based on Equation 1, and generates multiple feature images 202 in layer 2. In the first half of layer 2, the CNN processing unit 305 enlarges the multiple feature images 202 in the spatial direction, and generates multiple feature images 203 in the second half of layer 2. Example I' of feature images after enlargement i,j (m) can be expressed by the following formula.

[0025]

number

[0026]

number

[0027] In the latter half of layer 2, the CNN processor 305 calculates the results of the convolution process and activation process using multiple feature images 203 and weighting coefficients based on Equation 1, and generates one feature image 204 in layer 3. The feature image 204 is a confidence map output by the network. The processes in the first half and second half of layer 2 are merged, and the enlargement process in the first half and the convolution process in the second half are performed in the same layer. When enlarging the feature image 202 based on Equation 2, the processes in layer 2 are deconvolution process and activation process.

[0028] An example of a network and convolution processing is shown in Figure 6. The CNN processing unit 305 extracts feature data from the same position of four feature images 601 in layer 1, performs convolution processing, and calculates the result of activation processing. The result is that the features at the same position in layer 1 and layer 2 become the feature data of image 602.

[0029] <Flowchart of Convolutional Neural Network Processing in the Present Embodiment> 4 shows the configuration of the CNN processing unit 305. The CNN processing unit 305 has a data storage unit 408, a control unit 401, a feature data storage unit 402, a weighting coefficient storage unit 404, a feature data reading unit 403, a feature data enlargement unit 405, a convolution processing unit 406, an activation pooling processing unit 407, and a generation unit 409.

[0030] The data storage unit 408 temporarily stores feature data of a plurality of input images and feature images, weighting coefficients, and network structure information.x,y The feature data storage unit 402 stores the feature image I(m). The feature data reading unit 403 reads out the feature data from the feature data storage unit 402.

[0031] The generating unit 409 generates a read pattern for reading feature data from the data holding unit 408 based on the network structure. Specifically, the generating unit 409 acquires a magnification ratio, a filter size, a processing line, and the like related to the network structure from the data holding unit 408, and generates a read pattern. For example, the generating unit 409 may generate a read pattern based on the filter size of the convolution processing unit 406 of each layer. In this way, the generating unit 409 can generate a read pattern according to the number of feature data input to the convolution processing unit 406.

[0032] The feature data enlargement unit 405 enlarges the feature data read from the data storage unit 408 based on the read pattern, and outputs the enlarged feature data to the convolution processing unit 406. Here, the feature data enlargement unit 405 does not enlarge all the feature data of a feature image of one channel all at once, but enlarges a portion of the feature data necessary for the convolution processing in order, and outputs the enlarged feature data to the convolution processing unit 406.

[0033] The convolution processing unit 406 calculates the convolution processing result from the weighting coefficient and the feature data based on the formula 1. Here, the convolution processing unit 406 does not obtain the feature data after all the feature data of the feature image of one channel is enlarged, but obtains part of the feature data of the enlarged feature image from the feature data enlargement unit 405 and sequentially performs the convolution processing. In this way, the convolution processing unit 406 can perform the convolution processing on part of the feature data of the feature image. As a result, unlike the conventional technology, the feature data enlargement unit 405 and the convolution processing unit 406 do not need to write the enlarged feature data back to the memory and then read it again to perform the convolution processing. Since the enlargement processing and the convolution processing are fused by the layer fusion method, the feature data enlargement unit 405 and the convolution processing unit 406 perform the enlargement processing and the convolution processing continuously on the same layer. In other words, the feature data enlargement unit 405 and the convolution processing unit 406 perform enlargement and convolution processing on the feature data of one feature image, and then perform enlargement and convolution processing on the feature data of the next feature image. In this way, the enlarged feature data is sequentially convoluted, so that the memory capacity required to hold the enlarged feature data can be reduced, and the need to store the enlarged feature data in memory can be reduced. As a result, the convolution processing unit 406 can reduce the number of times that the enlarged feature data is read from the memory, thereby reducing the power required to access the memory and reducing the memory cost.

[0034] The activation pooling unit 407 calculates the activation pooling processing result based on the convolution processing result.

[0035] The control unit 401 has a CPU, a GPU, a sequencer, or the like, and executes each process of steps S101 to S114. The control unit 401 will explain each step of the flowchart shown in Fig. 1 based on the configuration of the CNN processing unit 305 shown in Fig. 4. The control unit 401 may implement each function of the CNN processing unit 305 shown in Fig. 4, for example, by reading a program.

[0036] In step S 101 , the control unit 401 reads out feature data of a plurality of input feature images, weighting coefficients, and network structure information from the RAM 308 , and stores them in the data storage unit 408 .

[0037] In step S102, the control unit 401 starts a loop for processing layers, and processes the first layer, layer 1. Therefore, at the start of the loop, layer 1 is the layer to be processed.

[0038] In step S103, the control unit 401 sets the enlargement magnification R of the feature image in accordance with the network structure information stored in the data storage unit 408. The enlargement magnification R of the feature image is a positive integer equal to or greater than 1. If the feature image is not to be enlarged, then 1 is set.

[0039] In step S104, the control unit 401 starts a loop for processing the output feature image, and calculates the output feature data in sequence.

[0040] In step S105, the control unit 401 performs initialization by setting the convolution processing result held in the convolution processing unit 406 to a zero value.

[0041] In step S106, the control unit 401 sets a filter size for the generation unit 409. The generation unit 409 generates a read pattern based on the enlargement magnification, filter size, processing line, etc. The read pattern indicates the address and the number of pieces when reading feature data from the feature data storage unit 402. The read pattern differs depending on the enlargement magnification, filter size, processing line, etc. The generation unit 409 has a means for storing the read pattern inside, and the generation unit 409 may refer to the read pattern stored inside in advance depending on the enlargement magnification, filter size, processing line, etc.

[0042] In step S107, the control unit 401 starts a loop for processing the input feature images, and processes the input feature data in order. When there are multiple feature images, that is, feature images for multiple channels, the control unit 401 processes them in the order of the channels.

[0043] In step S108, the control unit 401 reads out some of the input feature images from the data storage unit 408, and transfers them to the feature data storage unit 402. Then, the control unit 401 reads out some of the weighting coefficients from the data storage unit 408, and transfers them to the weighting coefficient storage unit 404.

[0044] In step S109, the feature data enlargement unit 405 receives a control signal from the control unit 401 and performs enlargement processing based on Equation 2 in accordance with the enlargement magnification R of the feature image. If the enlargement magnification R of the feature image is greater than 1, some of the feature data will become zero values. The feature data readout unit 403 refers to the readout pattern generated by the generation unit 409 and reads out feature data from the feature data storage unit 402. The feature data enlargement unit 405 enlarges the read out feature data. The convolution processing unit 406 receives a control signal from the control unit 401 and calculates the convolution processing result using Equation 1 based on the input feature data enlarged in accordance with the filter size and the weighting coefficient.

[0045] An example of the enlargement and convolution processes is shown in Fig. 7. The enlargement magnification R of the feature image is 2, and the filter size is 3 × 3. The enlargement process is divided into multiple steps.

[0046] 7(a), in the first processing of the first line of the output feature image, the feature data enlargement unit 405 reads out four pieces of feature data 702 from the pre-enlargement feature image 701 based on the read pattern, and inserts five pieces of feature data with zero values ​​to create 3×3 pieces of feature data 704 of the enlarged feature image 703 corresponding to the feature data 702. The convolution processing unit 406 partially calculates the convolution processing result 706 of the feature image 705 after the convolution processing, using the 3×3 pieces of feature data 704 and weighting coefficients.

[0047] As shown in Fig. 7(b), in the second processing of the first line of the output feature image, the feature data enlargement unit 405 reads two pieces of feature data 708 from the pre-enlargement feature image 707 based on the read pattern, and inserts seven pieces of feature data with zero values ​​to create 3 x 3 pieces of feature data 710 of the enlarged feature image 709 corresponding to the feature data 708. The number of feature data read out is different from that of the first processing. The convolution processing unit 406 partially calculates the convolution processing result 712 of the feature image 711 after the convolution processing, using the 3 x 3 pieces of feature data and weighting coefficients.

[0048] In the third and subsequent processes of the first line of the output feature image, the first or second readout pattern is used, so a detailed description thereof will be omitted.

[0049] 7(c), in the first processing of the second line of the output feature image, the feature data enlargement unit 405 reads two pieces of feature data 714 from the pre-enlargement feature image 713 based on the read pattern and inserts seven pieces of feature data with zero values ​​to create 3×3 pieces of feature data 716 of the enlarged feature image 715. The convolution processing unit 406 partially calculates the convolution processing result 718 of the post-convolution feature image 717 using the 3×3 pieces of feature data 716 and the weighting coefficients.

[0050] As shown in Fig. 7(d), in the second processing of the second line of the output feature image, the feature data enlargement unit 405 reads one piece of feature data 720 from the pre-enlargement feature image 719 based on the read pattern, and inserts eight pieces of feature data with zero values ​​to create 3 x 3 pieces of feature data 722 of the enlarged feature image 721. The number of feature data read out is different from that in the first processing. The convolution processing unit 406 partially calculates the convolution processing result 724 of the post-convolution feature image 723 using the 3 x 3 pieces of feature data 722 and the weighting coefficient.

[0051] In the third and subsequent processes of the second line of the output feature image, the first or second readout pattern is used, so a detailed description thereof will be omitted.

[0052] There are four types of total readout patterns, each including four pieces of feature data 702, two pieces of feature data 708, two pieces of feature data 714, and one piece of feature data 720. The number of feature data read out from the pre-enlargement feature image differs depending on the readout pattern.

[0053] In this embodiment, since the enlargement process and the convolution process of the feature image are performed on the same layer, the feature data enlargement unit 405 can directly input the enlarged feature data to the convolution processing unit 406. Specifically, the feature data enlargement unit 405 performs the enlargement process on part of the feature data of the feature image and outputs it to the convolution processing unit 406. The convolution processing unit 406 performs the convolution process on the feature data sequentially acquired from the feature data enlargement unit 405. The feature data enlargement unit 405 and the convolution processing unit 406 repeatedly perform the enlargement and convolution process on the feature data of one feature image, and after completing the enlargement and convolution process on the feature data of the one feature image, perform the enlargement and convolution process on the feature data of the next feature image. As a result, the feature data enlargement unit 405 does not need to temporarily store the feature data 704, 710, 716, 722 of the enlarged feature images 703, 709, 715, 721 in a memory such as the data storage unit 408, thereby reducing the power consumed by the convolution processing unit 406 to access the memory, thereby reducing memory costs. Also, when the enlargement factor R of the feature image is 1, the feature data read from the data storage unit 408 can be processed by the convolution processing unit 406 without being enlarged.

[0054] The feature data enlargement unit 405 outputs the feature data of the processed feature image to the convolution processing unit 406. The convolution processing unit 406 performs convolution processing regardless of the enlargement magnification R of the feature image and whether the feature image is enlarged or not. This eliminates the need to provide multiple convolution processing units 406 according to the enlargement magnification R, thereby reducing the circuit size.

[0055] FIG. 8 shows examples of enlargement and convolution processing. The enlargement magnification R of the feature image is 1, which is equivalent to processing without enlargement. The filter size is 3×3. The enlargement processing is divided into multiple rounds, but the readout pattern for each enlargement processing is the same. In the first processing of the first line of the output feature image, nine feature data 802 are read from the feature image 801 before enlargement. The feature data after enlargement becomes the feature data before enlargement. More feature data are read from the feature image before enlargement than in the example where the enlargement magnification R of the feature image is 2. The convolution processing unit 406 uses the 3×3 feature data and weighting coefficients to calculate the convolution processing result 804 of the feature image 803 after convolution processing for a part of the feature image 801.

[0056] In step S110, the control unit 401 determines whether the input feature image processing loop has ended. If all input feature images have been processed, the process proceeds to step S111. If the processes in steps S108 and S109 have not been completed for all input feature images, the process returns to step S107 and starts processing the next input feature image.

[0057] In step S111, the activation / pooling unit 407 receives a control signal from the control unit 401 and performs activation processing based on the convolution processing result stored in the convolution processing unit 406. The activation processing result is calculated by the following formula.

[0058]

number

[0059] In addition, the activation / pooling unit 407 performs pooling processing based on the result of activation processing according to the layer information, and adjusts the size of the output feature image as necessary.

[0060] In step S112, the control unit 401 stores the activation pooling process result in the feature data storage unit 402, and handles it as a feature image of the next layer.

[0061] In step S113, the control unit 401 determines whether the loop of output feature image processing has ended. If all output feature images have been processed, the process proceeds to step S114. If not, the process returns to step S105 to start processing the next output feature image.

[0062] In step S114, the control unit 401 determines whether the layer processing loop has ended. If all layers have been processed, the process of the convolutional neural network ends. If not, the process returns to step S103, the processing target layer is changed, and the process of the next layer is started.

[0063] In this embodiment, the enlargement and convolution processes of a feature image of one channel are performed in a loop (S107 to S110) for processing the input feature image before the feature data of the next input feature image is read and enlarged, etc., so there is no need to perform the enlargement process of the feature data every time the feature data of an input feature image is read out and to store the enlarged feature image in a memory such as a line buffer. This makes it possible in this embodiment to reduce the power required to access memory and reduce memory costs.

[0064] In this embodiment, the feature data enlargement unit 405 performs enlargement processing on part of the feature data of the feature image, and outputs the feature data after the enlargement processing to the convolution processing unit 406. As a result, in this embodiment, the convolution processing unit 406 can perform convolution processing on part of the feature data of the feature image in order.

[0065] In this embodiment, the feature data enlargement unit 405 performs enlargement processing on the feature data of the feature image in accordance with the enlargement magnification R of the feature image, and outputs the feature data to the convolution processing unit 406. This makes it possible to use a common convolution processing unit 406 even if the enlargement magnification R is different. In other words, there is no need to provide different convolution processing units 406 for processing a layer where the enlargement magnification R of the feature image is 1 and a layer where the enlargement magnification R of the feature image is not 1, and the circuit scale can be reduced.

[0066] <Second embodiment> <There are multiple feature data readout units> In the first embodiment, an example has been described in which the feature data enlargement unit 405 enlarges the feature data in accordance with the enlargement magnification R of the feature image, but when the enlargement magnification R of the feature image is 1, it is not necessary to enlarge the feature data by the feature data enlargement unit 405. In the second embodiment, configurations and steps different from those of the first embodiment will be described.

[0067] 5 shows the configuration of the CNN processing unit 305. The CNN processing unit 305 is composed of a data storage unit 408, a control unit 401, a feature data storage unit 402, a weighting coefficient storage unit 404, a first feature data reading unit 501, a second feature data reading unit 502, a feature data enlargement unit 405, a convolution processing unit 406, an activation and pooling processing unit 407, and a generation unit 409. The data storage unit 408 temporarily stores feature data of a plurality of input feature images, weighting coefficients, and network structure information. The weighting coefficient storage unit 404 stores the weighting coefficients C x,y The feature data storage unit 402 stores the feature image I(m). The convolution processing unit 406 calculates the convolution processing result from the weighting coefficients and the feature data based on Equation 1. The activation pooling processing unit 407 calculates the activation pooling processing result based on the convolution processing result.

[0068] In step S109 of the second embodiment, the feature data enlargement unit 405 receives a control signal from the control unit 401 and performs enlargement processing according to the enlargement magnification R of the feature image. If the enlargement magnification R of the feature image is not 1, the first feature data reading unit 501 reads out feature data from the feature data storage unit 402. The feature data enlargement unit 405 enlarges the feature data read out by the first feature data reading unit 501 and outputs it to the feature data selection unit 503. If the enlargement magnification R of the feature image is 1, the second feature data reading unit 502 reads out feature data from the feature data storage unit 402 and outputs the feature data to the feature data selection unit 503 without enlarging it. Here, when the filter size is the same, the number of feature data read out by the first feature data reading unit 501 is smaller than the number of feature data read out by the second feature data reading unit 502. An example of a filter size of 3×3 will be described. When the enlargement ratio R is 2, the number of feature data read by the first feature data reading unit 501 is 1, 2, or 4, which is the same as the processing of the first embodiment shown in Fig. 7. When the enlargement ratio R is 1, the number of feature data read by the second feature data reading unit 502 is 9, which is the same as the processing of the first embodiment shown in Fig. 8. The feature data selection unit 503 selects the feature data obtained from either the feature data enlargement unit 405 or the second feature data reading unit 502, and outputs the selected feature data to the convolution processing unit 406.

[0069] <Different filter sizes> In the first embodiment, an example has been described in which the enlargement magnification R of the feature image is 2 and the filter size is 3×3, but the filter size is not limited to 3×3 and may be any size.

[0070] An example of the enlargement process and the convolution process is shown in Fig. 13. The enlargement magnification R of the feature image is 2, and the filter size is 1 × 1. The enlargement process is divided into multiple steps.

[0071] 13(a), in the first processing of the first line of the output feature image, the feature data enlargement unit 405 reads one piece of feature data 1302 from the pre-enlargement feature image 1301 based on the read pattern, and creates 1×1 pieces of feature data 1304 of the enlarged feature image 1303. The convolution processing unit 406 partially calculates the convolution processing result 1306 of the post-convolution feature image 1305 using the 1×1 piece of feature data of the enlarged feature image 1303 and the weighting coefficient.

[0072] 13(b), in the second processing of the first line of the output feature image, the feature data enlargement unit 405 does not need to read out feature data based on the read pattern, and instead creates 1×1 feature data 1310 of an enlarged feature image 1309 by inputting one piece of feature data with a zero value. The number of feature data read out is different from that in the first processing. The convolution processing unit 406 partially calculates a convolution processing result 1312 of a post-convolution feature image 1311 using the 1×1 feature data of the enlarged feature image 1309 and a weighting coefficient.

[0073] In the third and subsequent processes of the first line of the output feature image, the first or second readout pattern is used, so a detailed description thereof will be omitted.

[0074] 13(c), in the first processing of the second line of the output feature image, the feature data enlargement unit 405 creates 1×1 feature data 1316 of an enlarged feature image 1315 by inserting one piece of feature data with a value of zero. The number of feature data read out differs from the first processing of the first line. The convolution processing unit 406 partially calculates a convolution processing result 1318 of a post-convolution feature image 1317 using the 1×1 feature data of the enlarged feature image 1315 and a weighting coefficient.

[0075] 13(d), in the second processing of the second line of the output feature image, the feature data enlargement unit 405 creates 1×1 feature data 1322 of an enlarged feature image 1321 by inserting one piece of feature data with a zero value. The number of feature data read out is different from the first processing of the first line. The convolution processing unit 406 partially calculates a convolution processing result 1324 of a post-convolution feature image 1323 using the 1×1 feature data of the enlarged feature image 1321 and a weighting coefficient.

[0076] In the third and subsequent processes of the second line of the output feature image, the first or second readout pattern is used, so a detailed description thereof will be omitted.

[0077] All read patterns are of one type, and one piece of feature data 1302 is read out.

[0078] In this embodiment, a readout pattern is generated and enlarged in accordance with a filter size of 1×1, so the amount of readout feature data is smaller than in the first embodiment, and processing time and power consumption can be reduced.

[0079] <Other embodiments> <Magnification of characteristic image> In the first and second embodiments, an example has been described in which the enlargement magnification R of a feature image is 1 or 2, but the enlargement magnification R is not limited to 1 or 2 and may be any positive integer. Also, in the second embodiment, an example has been described in which there are two feature data reading units, but the number of feature data reading units is not limited to two and may be the same as the settable enlargement magnification R of a feature image, or may be any number.

[0080] <Enlargement process: Nearest neighbor interpolation method> In the first and second embodiments, an example has been described in which the feature data enlargement unit 405 outputs zero values ​​as shown in FIG. 9 to enlarge a feature image. When a feature image is enlarged by outputting zero values, the zero values ​​may become discontinuous with the values ​​of adjacent pixels, which may reduce the accuracy of the recognition process. In order to improve the recognition accuracy of the CNN, the feature image may be enlarged by copying the feature data to adjacent pixels as shown in FIG. 10. The feature data after enlargement is the same as the output of the nearest neighbor interpolation method that is often used for upsampling. The numbers written in the feature data 901 before enlargement and the feature data 1001 after enlargement are indexes of the feature data. The feature data of the same index in the feature data after enlargement have the same value. When the enlargement magnification of the feature image is R, the feature image I' after enlargement and the feature image I before enlargement can be expressed by the following formula.

[0081]

number

[0082] In the configurations of the CNN processing unit 305 in the first and second embodiments, a common convolution processing unit 406 is used even if the enlargement magnification R is different. Therefore, by changing the processing of the generation unit 409 and the feature data reading unit 403, multiple types of enlargement processing can be realized.

[0083] The processing different from that in the first embodiment will be described. An example of the enlargement processing and the convolution processing is shown in Fig. 14. The enlargement magnification R of the characteristic image is 2, and the filter size is 3 x 3. The enlargement processing is divided into multiple steps.

[0084] 14(a), in the first processing of the first line of the output feature image, the feature data reading unit 403 reads out four pieces of feature data 1402 from the pre-enlargement feature image 1401 based on the reading pattern, and the feature data enlargement unit 405 copies some of the feature data to create enlarged 3×3 pieces of feature data 1404. The convolution processing unit 406 partially calculates the convolution processing result 1406 using the 3×3 pieces of feature data and weighting coefficients.

[0085] As shown in Fig. 14(b), in the second processing of the first line of the output feature image, the feature data reading unit 403 reads out four pieces of feature data 1408 from the pre-enlargement feature image 1401 based on the read pattern, and the feature data enlargement unit 405 copies some of the feature data to create 3 x 3 pieces of enlarged feature data 1410. The number of feature data read out is the same as in the first processing. The convolution processing unit 406 partially calculates the convolution processing result 1412 using the 3 x 3 pieces of feature data and weighting coefficients.

[0086] In the third and subsequent processes of the first line of the output feature image, the first or second readout pattern is used, so a detailed description thereof will be omitted.

[0087] 14(c), in the first processing of the second line of the output feature image, the feature data reading unit 403 reads out four pieces of feature data 1416 from the pre-enlargement feature image 1413 based on the reading pattern, and the feature data enlargement unit 405 copies some of the feature data to create the enlarged 3×3 pieces of feature data 1416. The convolution processing unit 406 partially calculates the convolution processing result 1418 using the 3×3 pieces of feature data and weighting coefficients.

[0088] As shown in Fig. 14(d), in the second processing of the second line of the output feature image, the feature data reading unit 403 reads out four pieces of feature data 1420 from the pre-enlargement feature image 1419 based on the read pattern, and the feature data enlargement unit 405 copies some of the feature data to create enlarged 3 x 3 pieces of feature data 1422. The number of feature data read out is the same as in the first processing. The convolution processing unit 406 partially calculates the convolution processing result 1424 using the 3 x 3 pieces of feature data and weighting coefficients.

[0089] In the third and subsequent processes of the second line of the output feature image, the first or second readout pattern is used, so a detailed description thereof will be omitted.

[0090] There is one type of all read patterns, and four pieces of feature data 1402, 1408, 1414, and 1420 are read out.

[0091] In this embodiment, the feature image is enlarged by copying the feature data, so all read patterns are one type. Unlike the four types of read patterns in the first embodiment, it is necessary to read four pieces of feature data each time to improve recognition accuracy.

[0092] <Enlargement process: other interpolation methods> An example of copying feature data as shown in FIG. 10 has been described, but the value copied from different feature data may be discontinuous with the value of adjacent pixels, which may adversely affect the results of the recognition process. In order to improve the recognition accuracy of CNN, the feature image may be enlarged by performing an interpolation process other than the nearest neighbor interpolation method based on the feature data before enlargement as shown in FIG. 11. The numbers written in the feature data before enlargement 901 and the feature data after enlargement 1101 are indexes of the feature data. The diagonal line pattern represents the interpolated feature data. When the enlargement magnification R of the feature image is 2 and the bilinear interpolation method is used, the enlarged feature image I' and the feature image before enlargement I can be expressed by Equation 5.

[0093]

number

[0094] Furthermore, when bilinear interpolation is used, the enlarged feature image I' and the unenlarged feature image I can also be expressed by Equation 6. Unlike FIG. 11, all feature data after enlargement may be calculated.

[0095]

number

[0096] Since it is necessary to calculate the interpolated result using a plurality of feature data, it is necessary to read out more feature data than in the nearest neighbor interpolation method, and the readout pattern differs from that in the nearest neighbor interpolation method.

[0097] The interpolation algorithm is not limited to bilinear interpolation, and any interpolation algorithm may be used, including bicubic interpolation, which is often used for upsampling.

[0098] <Enlargement process: Pixel Shuffle> In the first and second embodiments, an example has been described in which the feature data enlargement unit 405 outputs zero values ​​as shown in Fig. 9 to enlarge a feature image, but in order to improve the recognition accuracy of CNN, the feature images may be enlarged by rearranging the feature data of four feature images using the Pixel Shuffle method as shown in Fig. 12. The numbers written in the feature data 901 before enlargement and the feature data 1201 after enlargement are indexes of the feature data.

[0099] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0100] The disclosure of this specification includes the following information processing device, processing method, and program. (Item 1) An information processing device that processes feature data of a plurality of feature images according to a network structure having a plurality of hierarchical layers, A data storage means for storing feature data; a first read means for reading the feature data based on a read pattern; an enlarging means for enlarging the read characteristic data; a convolution processing means for performing a convolution process on the feature data enlarged by the enlargement means, The enlargement means and the convolution processing means perform the enlargement and convolution processing on the feature data of one feature image, and then perform the enlargement and convolution processing on the feature data of the next feature image. 23. An information processing apparatus comprising: (Item 2) the enlarging means sequentially enlarges the feature data of a part of the feature image; The convolution processing means executes the convolution processing on a portion of the expanded feature data. 2. The information processing device according to item 1, (Item 3) The data storage means stores a magnification ratio, The enlargement means enlarges the feature data in accordance with the enlargement ratio read from the data storage means and outputs the enlarged feature data to the convolution processing means. 3. The information processing device according to item 1 or 2. (Item 4) a second readout means for reading out the feature data based on a readout pattern different from that of the first readout means; a selection means for selecting one of the feature data read by the enlargement means and the second read means, The convolution means executes the convolution process on the selected feature data. 4. The information processing device according to any one of items 1 to 3, (Item 5) The read pattern may be generated based on information about the network structure. 5. The information processing device according to any one of items 1 to 4, (Item 6) The convolution processing means further includes a generating means for generating the readout pattern in accordance with the number of pieces of the feature data input to the convolution processing means. 6. The information processing device according to item 5, (Item 7) The number of the feature data input to the convolution processing means and the number of the read patterns generated by the read generating means differ depending on the filter size of the convolution processing of each of the plurality of layers. 7. The information processing device according to item 6, (Item 8) The number of the feature data read by the first read means varies depending on the read pattern. 8. The information processing device according to any one of items 1 to 7, (Item 9) The number of the feature data read by the first read means varies depending on the enlargement magnification of each of the plurality of layers. 4. The information processing device according to item 3. (Item 10) The number of readout patterns generated by the generating means varies depending on the magnification factor of the feature data of each of the plurality of layers. 6. The information processing device according to item 5, (Item 11) The expanding means expands the feature data with zero values. 11. The information processing device according to any one of items 1 to 10, (Item 12) The enlarging means enlarges the feature data by copying the feature data to the adjacent feature data. 12. The information processing device according to any one of items 1 to 11, (Item 13) The enlarging means enlarges the feature data by an interpolation process using the feature data. 13. The information processing device according to any one of items 1 to 12, (Item 14) When the filter size is the same, the number of the feature data read by the first read means is smaller than the number of the feature data read by the second read means. 5. The information processing device according to item 4, (Item 15) A processing method of an information processing device that processes feature data of a plurality of feature images according to a network structure having a plurality of hierarchical layers, comprising: a data storage step of storing feature data; a first read step of reading out the feature data based on a read pattern; an enlarging step of enlarging the read feature data; a convolution processing step of performing a convolution process on the feature data expanded by the expansion step, The enlargement step and the convolution step include performing the enlargement and convolution processing on the feature data of one feature image, and then performing the enlargement and convolution processing on the feature data of the next feature image. A processing method comprising: (Item 16) A program for causing a computer to function as each of the means of the information processing device according to any one of items 1 to 14.

[0101] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0102] 403: feature data reading unit; 405: feature data enlargement unit; 406: convolution processing unit; 409: generation unit; 501: feature data reading unit; 502: feature data reading unit; 503: feature data selection unit.

Claims

1. An information processing device that processes feature data of multiple feature images according to a network structure having multiple layers, A data storage means for storing feature data, A first reading means that reads the feature data based on a reading pattern, An enlargement means for enlarging the read-out feature data, The system includes a convolution processing means that acquires the feature data augmented by the augmentation means without storing it as augmented feature data in the data holding means, and performs convolution processing on it. The scaling means and the convolution processing means perform scaling and convolution processing on the feature data of one feature image, and then perform scaling and convolution processing on the feature data of the next feature image. An information processing device characterized by the following:

2. The enlargement means sequentially enlarges the feature data of a portion of the feature image, The convolution processing means performs the convolution process on a portion of the expanded feature data. The information processing apparatus according to feature 1.

3. The data holding means holds the magnification ratio, The scaling means scales the feature data according to the scaling factor read from the data holding means and outputs it to the convolution processing means. The information processing apparatus according to feature 1.

4. A second reading means reads the feature data based on a reading pattern different from that of the first reading means, The system further comprises a selection means for selecting either the feature data amplified by the augmentation means or the feature data read out by the second reading means, The convolution processing means performs the convolution process on the selected feature data. The information processing apparatus according to feature 1.

5. The system further comprises a generation means for generating the readout pattern based on the information of the network structure. The information processing apparatus according to feature 1.

6. The generation means generates the readout pattern according to the number of feature data input to the convolution processing means. The information processing apparatus according to feature 5.

7. The number of feature data input to the convolution processing means and the number of readout patterns generated by the generation means differ depending on the filter size of each of the multiple layers of the convolution processing. The information processing apparatus according to feature 6.

8. The number of feature data items read by the first reading means varies depending on the reading pattern. The information processing apparatus according to feature 1.

9. The number of feature data items read by the first reading means varies depending on the magnification ratio of each of the multiple layers. The information processing apparatus according to claim 3.

10. The number of readout patterns generated by the generation means varies depending on the magnification factor of the feature data in each of the multiple layers. The information processing apparatus according to feature 5.

11. The augmentation means augments the feature data using zero values. The information processing apparatus according to feature 1.

12. The enlargement means enlarges the feature data by copying the feature data to adjacent feature data. The information processing apparatus according to feature 1.

13. The augmentation means augments the feature data by interpolation processing using the feature data. The information processing apparatus according to feature 1.

14. If the filter size is the same, the number of feature data read by the first reading means is smaller than the number of feature data read by the second reading means. The information processing apparatus according to feature 4.

15. A processing method for an information processing device that processes feature data of multiple feature images according to a network structure having multiple layers, A data retention process for retaining feature data, A first reading step of reading the feature data based on the reading pattern, An enlargement step is performed to enlarge the read-out feature data, The process includes a convolution process which acquires the feature data augmented by the augmentation process without retaining it as augmented feature data in the data retention process, and performs a convolution process on it. The aforementioned scaling and convolution processes are performed on the feature data of one feature image, and then on the feature data of the next feature image. A processing method characterized by the following.

16. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 14.

Images