Information processing apparatus, information processing method, and program

Abstract

To provide an information processing apparatus and the like that achieve a skip connection at low cost.SOLUTION: An information processing apparatus includes: a convolutional hierarchy which has a plurality of convolution layers and in which each of the plurality of convolution layers propagates output data based on a feature quantity vector extracted from input data from a preceding stage to a subsequent stage; coupling means for coupling a forward propagation path that sequentially propagates the output data through each convolution layer between some convolution layers to the other convolution layers with a bypass path that bypasses the forward propagation path and propagates the output data from the some convolution layers to the other convolution layers; and processing means that performs processing of extracting the feature quantity vector from the input data at each of the plurality of convolution layers. The processing means performs re-extraction processing of re-extracting the feature quantity vector included in convolution layers up to a convolution layer where bypassing through the bypass path starts among the plurality of convolution layers. In a case where the processing means performs the re-extraction processing, the coupling means couples an output result from the forward propagation path and a result of the re-extraction processing performed by the processing means.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Skip connections have traditionally been used in neural network training. Skip connections are a configuration in deep neural networks that enables forward or backward propagation between distant layers by bypassing multiple intermediate layers to connect to the next layer. While skip connections can alleviate the gradient vanishing problem, they also reduce the generalization performance of neural networks. Therefore, Patent Document 1 discloses a technology that selects skip connections to be disabled and blocks error propagation only to the selected skip connections. The technology disclosed in Patent Document 1 selects skip connections to be disabled each time a neural network is trained. This enables repeated training using neural networks with different connection methods for each layer. Therefore, the technology disclosed in Patent Document 1 enables ensemble learning, thereby improving the generalization performance of the neural network as a whole. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 167665 Summary of the Invention [Problem to be solved by the invention]

[0004] The skip connection also requires the continuous retention of the processing results up to that point. Generally, the longer the processing results are retained, the larger the circuit area used as a storage area becomes. Therefore, the technology disclosed in Patent Document 1 as a whole improves the generalization performance of a neural network, but also increases costs due to the increased circuit area used as a storage area. For example, cache memories used to retain processing results are often composed of SRAMs, but SRAMs (Static Random Access Memory) are generally expensive memories. Therefore, it is desirable not to increase the circuit area used as the SRAM storage area. However, if the circuit area used as the SRAM storage area is not increased, there is a possibility that the memory storage area for retaining the processing results will be insufficient, making it impossible to implement the skip connection.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and aims to make it possible to realize skip joining at low cost. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present disclosure includes: a convolutional layer having a plurality of convolutional layers, each of which propagates output data based on a feature vector extracted from input data input from a previous stage to a subsequent stage; a forward propagation path that propagates the output data in sequence from a portion of the plurality of convolutional layers to each of the convolutional layers located between a portion of the plurality of convolutional layers and another portion of the plurality of convolutional layers; and a detour path that detours the forward propagation path and propagates the output data from the portion of the convolutional layers to the other portion of the plurality of convolutional layers; and processing means that performs processing to extract the feature vector from the input data in each of the plurality of convolutional layers, wherein the processing means performs a re-extraction processing to re-extract the feature vector included in the plurality of convolutional layers up to a convolutional layer where detour by the detour path starts, and the combining means, when the re-extraction processing is performed by the processing means, combines an output result from the forward propagation path and a result of the re-extraction processing performed by the processing means. [Effects of the Invention]

[0007] According to the present disclosure, skip joining is possible at low cost. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a block diagram showing the configuration of an inference execution device. [Figure 2] FIG. 2 is a conceptual diagram illustrating an example of the configuration of an inference unit. [Figure 3] FIG. 1 is a schematic diagram showing the configuration of a skip coupling. [Figure 4] FIG. 2 is a conceptual circuit diagram of a filter that constitutes an inference unit. [Figure 5] FIG. 1 is a schematic diagram showing the vicinity of an input section of a CNN. [Figure 6] 1 is a flowchart outlining the processing performed in a convolutional layer. [Figure 7] FIG. 1 is a schematic diagram showing the details of neurons that make up a CNN. [Figure 8]10 is a flowchart illustrating a convolution process. [Figure 9] FIG. 1 is a schematic diagram showing the vicinity of an output section of a CNN. [Figure 10] FIG. 2 is a conceptual circuit diagram of a filter that constitutes an inference unit. [Figure 11] FIG. 1 is a conceptual diagram showing images used for learning. [Figure 12] FIG. 2 is a conceptual circuit diagram of a filter that constitutes an inference unit. [Figure 13] FIG. 10 is a flow diagram illustrating learning. [Figure 14] 10 is a flowchart showing a re-extraction operation. [Figure 15] FIG. 10 is a diagram illustrating an example in which feature vectors in the fifth layer are skip-combined. [Figure 16] FIG. 2 is a conceptual circuit diagram of a filter that constitutes an inference unit. [Figure 17] 1 is a schematic diagram of CNN processing. [Figure 18] FIG. 2 is a conceptual circuit diagram of a filter that constitutes an inference unit. [Figure 19] FIG. 10 is a schematic diagram illustrating skip joins using dimensionally compressed feature vectors. [Figure 20] FIG. 1 is a schematic diagram of a model that performs skip connections. [Figure 21] FIG. 10 is a schematic diagram illustrating details of a skip connection. [Figure 22] FIG. 10 is a diagram showing a setting screen when training a CNN model. [Figure 23] FIG. 10 is a diagram illustrating an example of a detailed setting screen when training a CNN model. [Figure 24] FIG. 10 is a diagram showing another example of a detailed setting screen when training a CNN model. [Figure 25] 1 is a flowchart illustrating automatic CNN model design. DETAILED DESCRIPTION OF THE INVENTION

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the present disclosure, and the combinations of features described in the following embodiments are not necessarily essential to the solutions of the present disclosure. Note that the same reference numerals are used to designate the same components.

[0010] <<First Embodiment>> [overview] As neural networks undergo repeated training, the gradient obtained in backpropagation becomes smaller and eventually disappears. This is known as the gradient vanishing problem. To address this issue, skip connections are used. Skip connections are a configuration in deep neural networks that enables forward propagation or backpropagation between distant layers by creating a detour path that skips multiple intermediate layers and connects to a previous layer. Skip connections create a detour path that skips some of the multiple layers constituting a neural network, allowing the detour path and the forward propagation path to be established in parallel. This path configuration allows features to be propagated to distant layers via a separate path, skipping some of the multiple layers. This makes it possible to propagate features that disappear due to convolutional processing, etc., performed in earlier layers to later layers. However, implementing skip connections requires the continuous retention of feature vectors extracted in each layer. Therefore, compared to a case where skip connections are not used, skip connections require more memory storage. Furthermore, when skip connections are used to retain feature vectors for each layer and use them for processing as needed, they are stored in cache memory rather than main memory to ensure processing efficiency. This requires a larger cache memory. SRAM is typically used for cache memory, but SRAM is expensive. Therefore, in this embodiment, instead of continuing to retain feature vectors extracted in each layer, the following operation is performed to achieve skip connections at low cost. That is, input data is re-acquired from main memory to cache memory, and feature vectors for each layer are re-extracted from the beginning of the layer to the corresponding layer as a re-extraction process. This operation makes it possible to achieve skip connections without increasing the circuit area used as a storage area for expensive SRAM. The model configuration of the neural network is not particularly limited. For example, it may be a convolutional neural network that constitutes an encoder / decoder model. Alternatively, it may be an inverted residual model, which exists in models typified by ResNet.

[0011] The main terms used in this specification are defined as follows. <Term definition> Artificial neurons: An artificial neuron is a processing unit consisting of a filter and an activation function unit. The convolution coefficients of the filter are also called "weights." The convolution coefficients of the filter are also referred to as "weights of the artificial neuron" as appropriate. The artificial neuron receives input data for the filter. For example, if the artificial neuron is a 3x3 filter, it receives 5x5 input data, transfers the convolved value to the activation function unit, and outputs the feature value calculated by the activation function unit. Activation function part: This is a function with nonlinear response characteristics. A sigmoid function is used, but a ReLU (Rectified Linear Unit) function may also be used. If a function with nonlinear response characteristics is used, the relationship between input and output will have nonlinear response characteristics, but this is not limited to this. For example, the activation function unit may be a function with linear response characteristics. Alternatively, the activation function unit may be an identity function. For example, when sending a feature vector to a distant layer using a skip connection, the activation function unit may be implemented as an identity function. layer: A processing unit consisting of multiple artificial neurons. In principle, the same data is input to each artificial neuron. However, the convolution coefficients (weights) of each artificial neuron can be set differently depending on the features you want to obtain. The reason it is composed of multiple artificial neurons is to analyze the input data from multiple angles. Features: The output from one artificial neuron is called a feature. Different artificial neurons output different features. Note that features may be output from artificial neurons as a constant index, such as intensity. Feature vector: A vector consisting of multiple feature quantities output from one layer. The order of this vector is hereinafter referred to as a "channel."

[0012] [Prerequisite technology] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the embodiment described here, it is assumed that the necessary learning results for performing inference on an EdgeAI terminal have been learned externally in advance. Here, an EdgeAI terminal is a product that can enjoy the results of artificial intelligence as a standalone product. An EdgeAI terminal does not need to be equipped with both the "learning" and "inference" functions required for a CNN (Convolutional Neural Network). A product can achieve "inference" by retaining parameters that are the results of learning prepared in advance. CNN is a type of pattern recognition that uses machine learning. CNN is also one of the processing methods that manufacturers use to achieve high functionality in order to differentiate their products. An overview of the operation of CNN to achieve pattern recognition will be described.

[0013] [Overview of CNN operation] First, features of input data are extracted according to a pre-defined feature extraction method. The feature extraction method is described below. Feature extraction can be achieved by massive convolution processing using a multi-stage filter. This multi-stage filter is composed of multiple filters and multiple activation function units. Multiple activation function units are placed downstream of each of the multiple filters. A pair of a filter and an activation function unit corresponds to the "artificial neuron" defined above. The activation function unit is, for example, a function whose response to an input is nonlinear. Each filter has convolution coefficients. A method for determining the convolution coefficients is described below. The convolution coefficients can be determined by using massive amounts of data in advance to determine the pattern type. Specifically, the convolution coefficients can be determined by preparing massive amounts of correct data and optimizing the convolution coefficients until the probability of unknown data being correct increases. Hereinafter, this determination method is referred to as "learning." Features of the input data are extracted by performing massive convolution processing using the convolution coefficients obtained as a result of learning. The feature values ​​of the input data obtained in this way are not limited to one type, as they are obtained by each artificial neuron. At least some of the multiple feature values ​​of the input data correspond to the "feature vector" defined above. In this way, CNN extracts feature vectors from the input data.

[0014] Next, the CNN identifies which of the predetermined patterns the feature vector matches from the output of the final layer of the CNN. In this way, the input data is classified into known patterns. This achieves pattern recognition. This type of pattern recognition corresponds to the "inference" mentioned above.

[0015] A CNN may be implemented using an encoder-decoder model consisting of an encoding layer and a decoding layer. The encoder-decoder model may determine pixel attributes on a pixel-by-pixel basis. The encoder-decoder model may determine attributes for all pixels in an image. In other words, the encoder-decoder model can determine attributes on a pixel-by-pixel basis. Hereinafter, this process will be referred to as "region segmentation." Alternatively, this process may be referred to as "segmentation." Note that "segmentation" here corresponds to so-called semantic segmentation. By accumulating pixel attribute determination results for each pixel, it is possible to identify whether consecutive pixels are the same object. Specifically, the encoding layer extracts wide-area features by downsampling the input data. Meanwhile, the decoding layer derives a final determination result by upsampling the extracted features to the same resolution as the input data. A CNN configured as this encoder-decoder model has, for example, the following characteristics. First, the input data passes through numerous layers before reaching a final determination result. As a result, resolution changes in intermediate processing layers.

[0016] It also has the following characteristics: The artificial neurons used in CNNs are equipped with the above-mentioned filters. The filters perform convolution processing on input data. As explained above, these filters have convolution coefficients. The convolution coefficients are so-called "weights." In CNNs, the features obtained through the model are compared with true values. Specifically, in CNNs, the difference between the calculated features and the true values ​​is calculated. This difference is called the "error." The method of finding "weights" that minimize this error is called backpropagation. Furthermore, optimizing the convolution coefficients by repeatedly using backpropagation is a specific example of the above-mentioned "learning." In this way, determining the convolution coefficients through learning is also a characteristic of CNNs.

[0017] Skip join These characteristics can lead to the following problems. For example, backpropagation may not proceed correctly during training. This is because the results of backpropagation become smaller as the layers become deeper, preventing learning from progressing. Hereinafter, this phenomenon is referred to as "gradient vanishing." Alternatively, the resolution may change with each layer, causing information indicating local features of the input data retained during encoding to be lost. These problems can result in a degradation of accuracy during training. To address this degradation, "skip connections" have traditionally been used. In an encoding / decoding model, skip connections can be achieved by reusing data from the encoding layer in the convolutional processing of the decoding layer. This operation improves the quality of the information during decoding by using information lost during encoding. At the same time, this operation enables optimal backpropagation during training, including feedback components generated by skip connections. This enables training that restores local edges lost during encoding. It also enables accurate determination of image region boundaries. However, when skip connections are implemented, for example, the processing results of the encoding layer must be passed to the decoding layer. Therefore, the more layers are used during encoding, the more the processing results of each artificial neuron are stored in SRAM. This is because in order to use the encoding results during decoding, they must all be stored in SRAM.

[0018] CNNs can also be used for image recognition. To use CNNs for image recognition, all that is required is a convolution process performed on the entire image. Here, we explain a more specific example of the filters used in convolution. For example, assume that a single 3x3 filter is applied to an image. Convolution is a process in which the value of the central pixel is determined by the sum of the products of the convolution coefficients and the pixels in the image. Therefore, when applying a 3x3 filter to a 3x3 image, only the value of the central pixel is determined. If a 3x3 filter were to be applied to the neighboring pixels around the 3x3 image, a 5x5 image would be required. The surrounding pixels required for convolution processing according to the required image area are hereinafter referred to as "margin." As the size of each filter increases and the number of two-dimensional filter layers increases across the CNN as a whole, even more margin is required. Therefore, the amount of margin required increases three-dimensionally. When the margin increases in this way, the amount of memory storage space used must also increase accordingly. For example, in convolution processing, data retrieved from main memory is loaded into cache memory. Cache memory is generally made up of SRAM. Therefore, when the margin increases, the amount of SRAM storage space used increases. In particular, when multiple large-scale filters are stacked, the amount of SRAM storage space used increases three-dimensionally compared to one or two filters.

[0019] For the above reasons, performing convolution processing using filters across multiple layers requires a huge amount of SRAM storage space. Furthermore, performing skip connections also requires a huge amount of SRAM storage space. For example, in an encoder-decoder model, performing skip connections can improve the reliability of data during decoding, but the required SRAM storage space increases exponentially. Because SRAM is expensive, significantly increasing the SRAM storage space results in high costs. On the other hand, if the SRAM storage space is not increased, the cache memory required for skip connections will be insufficient. Note that although an example of skip connections in an encoder-decoder model has been described, even in other models, skip connections typically require a large amount of cache memory, making it difficult to perform skip connections at low cost. Therefore, in this embodiment, the configuration and operation that enable skip connections to be performed at low cost will be described in order below.

[0020] [The entire inference execution device] FIG. 1 is a block diagram showing the configuration of an inference execution device. The inference execution device 100 is an information processing device implemented in a product main body. In this embodiment, the product main body is assumed to be a printer. However, the product main body in which the inference execution device is implemented is not limited to a printer. The configuration of this embodiment can also be applied to personal computers, smartphones, and other products that incorporate a CPU or similar processing circuits such as ASICs and FPGAs. The inference execution device 100 includes a data transfer I / F 101, a data bus 102, and a dynamic random access memory (DRAM) 103. The inference execution device 100 also includes a central processing unit (CPU) 104, an inference unit 105, and a read-only memory (ROM) 106. The data transfer I / F 101 is an interface for inputting and outputting data from devices external to the product (not shown). Examples of external devices include devices that can generate or store input data and transfer the input data to the product main body, such as personal computers or mobile phones. The data bus 102 is a data bus for transferring various data received from the data transfer I / F 101 to the functional blocks described below. The DRAM 103 is an area for temporarily storing various data received from the data transfer I / F 101. The CPU 104 receives and transmits input data stored in the DRAM 103 via the data bus 102 and performs necessary processing. The inference unit 105 is a functional block that receives data divided into image blocks and performs inference internally. The inference unit 105 includes an SRAM. The ROM 106 is an area for storing various data to be provided to the inference unit 105. For example, the ROM 106 can store convolution coefficients determined based on the results of prior learning. The size of the image blocks to be passed from the DRAM 103 to the inference unit 105, as described below, is also stored here. These configurations are merely examples, and any storage medium may be used instead of the ROM 106. The storage medium may be, for example, a hard disk drive (HDD) or an external memory via a USB interface. In this embodiment, inference is performed by the inference unit 105.However, firmware for realizing an equivalent mechanism may be stored in a storage medium and processed by the CPU 104. Furthermore, as part of functional expansion, the size of an image block to be passed from the DRAM 103 to the inference unit 105 via the data transfer I / F 101 may be exchanged as a parameter.

[0021] [Inference part] FIG. 2 is a conceptual diagram illustrating an example configuration of the inference unit 105. The inference unit 105 in FIG. 2 is assumed to operate according to an encoder-decoder model. Examples of encoder-decoder models include SegNet and U-Net. The inference unit 105 realizes each functional configuration as an inference execution unit 200 by the CPU 104 executing various programs. The inference execution unit 200 includes an encoding layer 201 and a decoding layer 202. The encoding layer 201 includes an input layer 203 and a processing layer 204. The encoding layer 201 encodes the features of input data. The decoding layer 202 decodes the processing results obtained in the encoding layer 201 and extracts a feature vector. Input data is input to the input layer 203. Here, a layer refers to a single operating entity that achieves some processing using multiple filters in a CNN model. Multiple filters are not necessarily required as a physical configuration. Furthermore, if the convolution coefficients are updated incrementally and the processing results are fed to the next filter, two consecutive filter processes are performed. Here, the input layer 203 is shown as an example of such a layer. The processing layer 204 receives input data from the input layer 203 and performs subsequent processing. Through this processing, encoding is achieved in the first half. These subsequent layers, like the input layer, are also configured using multiple filters. The decoding side, like the encoder side, also has processing layers using multiple filters. In the example shown in Figure 2, each layer is represented as a cube with rectangular faces, and its size indicates the resolution. That is, as the encoding side layer processing progresses, the resolution decreases, while as the decoding side layer processing progresses, the resolution increases. The following explanation will discuss the use of multiple filters in succession. Furthermore, the final output from the decoding side is uniquely determined by the processing of the activation function unit in the final layer. The probability of pixel attributes is determined by the processing result of the activation function unit. In the example shown in Figure 2, an encoder-decoder model is assumed, so a description of the CNN's decoding layer is omitted. In the example shown in Figure 2, the CNN combines multiple two-dimensional filters to form several layers. Encoding and decoding are performed by combining the formed layers.A feature vector is obtained through these processes. Note that the inference unit 105 in FIG. 2 is assumed to be an encoder-decoder model, but the model is not particularly limited to this. For example, a ResNet model may be assumed. In the ResNet model, multiple convolutional layers and pooling layers are provided after the input layer, followed by a fully connected layer and an output layer.

[0022] [Skip Join Overview] Next, skip connections will be described. FIG. 3 is a schematic diagram showing the configuration of skip connections. In this embodiment, the encoding layer 201 is represented by seven rectangles in the diagram. Each of the seven rectangles represents a layer. Each layer has multiple artificial neurons. The length of each rectangle indicates the resolution of the input data. That is, as the length of the rectangle decreases, the resolution of the input data decreases. Conversely, as the length of the rectangle increases, the resolution of the input data increases. Therefore, the encoding layer 201 in FIG. 3 is shown as having seven layers. Note that the layer configuration is not limited to this. Each layer may be configured by combining artificial neurons so as to extract desired features. The convolution layer performs convolution processing using a product-sum operation, and the pooling layer aggregates representative values ​​from the convolution processing results. As a result, the input data is thinned while the features of the input data are extracted, resulting in compression processing of the input data (hereinafter also referred to as downsampling). In other words, downsampling is pooling performed by aggregating multiple values ​​obtained by convolution processing into a representative value using a specific algorithm. A specific algorithm for performing this pooling is, for example, a process of calculating the average value of multiple values ​​obtained by the convolution process. This results in the multiple values ​​obtained by the convolution process being aggregated into a single representative value. Alternatively, this algorithm is a process of calculating the maximum value among the multiple values ​​obtained by the convolution process. This results in the multiple values ​​obtained by the convolution process being aggregated into a single representative value. In this way, pooling can suppress performance degradation when the coordinate position within the image changes. Note that, if a pooling layer is not used, downsampling may be performed by increasing the stride of the filter scanned during convolution, thereby obtaining the features of the image after scaling. Regardless of the method used, a feature vector that serves as the output value can be obtained from any layer during encoding. The same applies to the processing layer on the decoding side. However, in the decoding layer, an upsampling layer is used to expand the resolution of the features. In normal processing, data is input to the input layer 203, and processing proceeds from the input layer to the stage after the input layer. This processing direction is the forward propagation direction.The output layer 301 is a layer that outputs the feature vector at this point. The dimension addition layer 302 is a layer that adds dimensions using the feature vectors output from the output layer 301. Here, we will discuss dimension addition. Generally, the dimension of the sum obtained by adding an nth-order vector and an nth-order vector is nth. Mathematical addition is not defined for an nth-order vector and an mth-order vector. We will assume that dimension addition does not mean vector addition, but simply arranging vectors of different dimensions to generate an (n+m)th-order vector. Hereinafter, this type of processing will be referred to as "dimensional merging." A processing method that aligns the output of any layer on the encoding side so as to add dimensions when inputting any layer on the decoding side is called skip merging. In other words, skip merging is an operation that increases the components of a vector. Note that in the upsampling layer, data may be expanded by interpolation. Alternatively, data may be expanded by transpose convolution or ascending convolution.

[0023] For the above reasons, the information processing device in this embodiment has the following configuration, regardless of the model. That is, the information processing device includes a convolutional layer, a combining means, and a processing means. The convolutional layer has multiple convolutional layers. In the convolutional layer, each of the multiple convolutional layers propagates output data based on a feature vector extracted from input data input from the previous stage to the subsequent stage. Here, the previous stage means the stage immediately preceding each convolutional layer. Also, the subsequent stage means the stage immediately following each convolutional layer. The combining means is realized by the CPU 104 in FIG. 1. The combining means combines a forward propagation path and a detour path. The forward propagation path propagates output data in sequence from some convolutional layers to other convolutional layers among the multiple convolutional layers. The detour path propagates output data from some convolutional layers to other convolutional layers, bypassing the forward propagation path. The processing means is realized by the CPU 104 of FIG. 1. The CPU 104 of FIG. 1 extracts feature vectors from input data in each of the multiple convolution layers. The CPU 104 of FIG. 1 performs a re-extraction process to re-extract feature vectors included in the multiple convolution layers up to the convolution layer where the detour using the detour path begins. When the re-extraction process is performed by the processing means, the combining means combines the output result from the forward propagation path with the result of the re-extraction process performed by the processing means. With this configuration, skip combining is possible because feature vectors from each layer are re-extracted without storing feature vectors from each layer in a cache memory in the forward propagation path. This enables skip combining at low cost. Note that the input data is composed of multiple elements. The multiple elements are, for example, multiple pixels. Therefore, the input data is composed of, for example, multiple pixels. Each of the multiple convolution layers has a filter for which multiple convolution coefficients are specified. This filter will be described later with reference to FIGS. 4, 10, 12, 16, and 18. 1 extracts a feature vector by performing convolution processing based on a plurality of pixels and a plurality of convolution coefficients in each of a plurality of convolution layers. This operation makes it possible to extract a feature vector using a filter.Specifically, the CPU 104 in Fig. 1 performs a product-sum operation on input data while shifting a filter at a constant stride, thereby obtaining features representing local characteristics of the input data for each filter shift, and extracting a set of the obtained features as a feature vector. This operation makes it possible to extract a feature vector from input data using a filter. Note that shifting the filter here means shifting, at a constant stride, the area of ​​the pixels of the input data expanded in the storage area that is processed using the convolution coefficients of the filter. Therefore, it does not mean physically moving the filter.

[0024] [Filter circuit configuration] FIG. 4 is a conceptual circuit diagram of a filter 400 constituting the inference unit 105. The filter 400 includes an SRAM 401 and a register 402. In the example shown in FIG. 4, data 403 and a data set 404 of convolution coefficients are loaded in a storage area of ​​the SRAM 401. The data 403 is acquired from the DRAM 103, which functions as the main memory, and loaded in a predetermined storage area of ​​the SRAM 401. The data 403 is composed of pixels d1 to d9. The data set 404 of convolution coefficients is composed of pixels c1 to c9 and is arranged in a 3×3 matrix. In addition, a data set 405 of r1 to r9 is arranged in a 3×3 matrix in the register 402, in the same arrangement as the data set 404 of convolution coefficients. The data set 405 of r1 to r9 is used to maintain the positional relationship (coordinates) of the 3×3 matrix during convolution processing.

[0025] [Method of generating convolution coefficients] Next, a method for generating convolution coefficients will be described. [Generation environment] FIG. 5 is a schematic diagram showing the vicinity of an input unit of a CNN. In this embodiment, the convolution coefficients may be generated using a personal computer (not shown) as the learning device. The learning device is not limited to a personal computer, but may also be a product such as a printer or smartphone that has a CPU or similar processing circuit such as an ASIC or FPGA built into it. Alternatively, the inference execution device 100 may generate the convolution coefficients through learning.

[0026] Data 501 is input data. For example, if the input data is image data, data 501 is prepared as a 3x3 pixel array with three channels (R, G, and B) for each coordinate, as shown in the figure. Artificial neurons 502 through 507 are elements that process data 501. In this example, artificial neurons 502 through 507 store convolution coefficients for convolving data 501. The stored convolution coefficients are for three channels (R, G, and B). As will be described later, these values ​​are currently variables to be generated. For example, artificial neuron 502 stores 3x3 convolution coefficients for three channels (R, G, and B) for convolving data 501. Artificial neurons 502 through 507 can store convolution coefficients with different characteristics. This is because one convolution process can extract one feature. To extract multiple different features, multiple convolution processes can be performed. In this embodiment, an example will be described in which a first processing layer having six artificial neurons 502 to 507 and a second processing layer having four artificial neurons 510 to 513 are provided as convolutional layers. The first processing layer having the artificial neurons 502 to 507 can extract six feature quantities for the subsequent processing layer after completing convolution processing for each of the artificial neurons 502 to 507. The second processing layer having the artificial neurons 510 to 513 can extract four feature quantities for the subsequent processing layer after completing convolution processing for each of the artificial neurons 510 to 513. In other words, the artificial neurons 510 to 513 receive the feature quantities extracted by each of the artificial neurons 502 to 507 from the previous processing layer and similarly perform convolution processing to extract the four feature quantities for the subsequent processing layer.

[0027] [Study Summary] FIG. 6 is a flowchart outlining the processing performed in the convolutional layer. The processing shown in FIG. 6 may be implemented by the CPU 104. The following description will explain an example in which the CPU 104 executes the processing. Note that some or all of the functions of the steps in FIG. 6 may be implemented by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process indicates a step in the flowchart.

[0028] The process shown in FIG. 6 starts with the execution of learning processing in the convolution layer. In S601, the CPU 104 reads input data from the DRAM 103. In S602, the CPU 104 loads the read input data in a storage area of ​​the SRAM 401. The CPU 104 reads convolution coefficients based on a program prepared in advance in the ROM 106 and loads them in a storage area of ​​the SRAM 401. It is preferable that the input data and the convolution coefficients are loaded in different storage areas of the SRAM 401. In S603, the CPU 104 sets the convolution coefficients loaded in the storage area of ​​the SRAM 401 in a storage area on the register 402. In S604, the CPU 104 performs convolution processing based on multiple pixels included in the input data and the convolution coefficients. The process of S604 will be described in detail later. In S605, the CPU 104 records the results of the convolution processing in a storage area of ​​the SRAM 401. In S606, the CPU 104 determines whether input data to be processed in the convolutional layer remains based on whether all pixels have been processed. If not all pixels have been processed, the CPU 104 returns the process of S606 to S604. If all pixels have been processed, the CPU 104 advances the process of S606 to S607. In S607, the CPU 104 determines whether a next filter is required as the next process. If a next filter is required, the CPU 104 returns the process of S607 to S603 and sets convolution coefficients for the filter of the second convolutional layer in the register 402. Thereafter, in S606, the CPU 104 performs convolution processing on the results of the first convolutional layer using the convolution coefficients of the filter of the second convolutional layer. In this way, when processing of all filters is completed, the CPU 104 ends the process of S607, thereby ending the processes of S601 to S607.

[0029] [Overview of Artificial Neurons] FIG. 7 is a schematic diagram showing details of an artificial neuron 700 that constitutes a CNN. The artificial neuron 700 includes a convolution unit 701 and an activation function unit 702. The artificial neuron 700 is included in a convolution layer. The artificial neuron 700 is a processing mechanism that receives input from the previous stage of the convolution layer and outputs it to the next stage of the convolution layer. The convolution unit 701 performs convolution processing using convolution coefficients. The activation function unit 702 has a function with nonlinear characteristics. Specifically, the activation function unit 702 includes a softmax function or a ReLU function. The activation function unit 702 outputs the result of function processing using the result of the convolution unit 701 as input. Depending on the result of the convolution unit 701, the output from the activation function unit 702 may be weak. In other words, whether or not information is transmitted from the activation function unit 702 to the next layer depends on the convolution coefficients used in the convolution unit 701. This process is repeated for each subsequent stage, up to the final stage (not shown) of the model, to extract features. In other words, the activation function unit 702 calculates features, which are components of a feature vector, based on the results of the convolution process output from the convolution unit 701. As described above, a layer having multiple convolution layers is referred to as a convolution layer. A convolution layer may also have multiple pooling layers. Each of the multiple pooling layers may be arranged after each of the multiple convolution layers, and may aggregate the feature vector into a representative value as output data. Aggregation is an operation of extracting one feature from multiple features falling within a specific range. For example, the maximum value of multiple feature values ​​falling within a specific range may be extracted. Alternatively, the average value of multiple feature values ​​falling within a specific range may be extracted. An upsampling layer may also be arranged after the convolution layer. The CPU 104 may expand the output data in the upsampling layer to increase the size of the representative value to that of the input data, and output the resulting data as subsequent data. For example, an upsampling layer stretches the output data in the X and Y directions to increase the size of the representative values ​​to the input data.

[0030] [Convolution processing] FIG. 8 is a flowchart illustrating the convolution process. The process shown in FIG. 8 may be implemented by CPU 104. The following description will explain an example in which CPU 104 executes the process. Note that some or all of the functions of the steps in FIG. 8 may be implemented by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process indicates a step in the flowchart.

[0031] The process shown in FIG. 8 begins when the convolution process is called. In S801, the CPU 104 sets a convolution coefficient in the register 402. In S802, the CPU 104 multiplies one pixel out of a plurality of pixels expanded in a storage area on the SRAM 401 by the convolution coefficient set in the register 402. The CPU 104 collects and adds the multiplication results for the number of elements included in one filter 400. Here, the elements included in the filter 400 refer to the convolution coefficients. The number of elements refers to the number of convolution coefficients. A more specific description of the convolution process will be made with reference to FIG. 10.

[0032] [Get Error] Figure 9 is a schematic diagram showing the vicinity of the output section of a CNN. In this example, an activation layer 901 is shown. The activation layer 901 includes an activation function unit 702. When the layer containing the artificial neuron 700 in Figure 7 reaches the final stage, it is output through the activation function unit 702. This operation obtains the features of the image input as input data. Therefore, a CNN model obtains features from input data using a massive number of filter calculations and activation functions. The overall configuration of the model, consisting of processing units including filters and activation functions, depends on the basic design of the model used. When a publicly known model is used, it depends on the configuration of that model. Furthermore, when building a model structure itself, it is determined by how many artificial neurons 700 with filter sizes are used and how many layers each of these neurons will have. True features that indicate the features of the subject captured in the image data input can be obtained using other methods. For example, a value can be determined by human visual judgment. Hereinafter, this value will be referred to as the "correct answer." The error can be obtained by taking the difference between the value obtained from the CNN model and the correct answer. An upsampling layer may be arranged before the activation layer 901. That is, the activation layer 901 may be arranged after the upsampling layer. The activation layer 901 may reconstruct subsequent image data onto which data acquired from the previous stage is mapped. When an upsampling layer is arranged before the activation layer 901, the activation layer 901 may acquire subsequent data in which the size of the representative value is increased to match the input data. When no upsampling layer is arranged before the activation layer 901 and a convolution layer is arranged, the activation layer 901 may acquire a representative value that aggregates feature vectors. The CPU 104 may classify objects appearing in image data composed of multiple pixels based on the subsequent image data reconstructed by the activation layer 901. The CPU 104 may calculate convolution coefficients based on the subsequent image data reconstructed by the activation layer 901 and the input data.

[0033] [Detailed example of convolution processing] FIG. 10 is a conceptual circuit diagram of a filter constituting the inference unit 105. As shown in FIG. 10, a margin data set 1001 is secured around pixels d1, d2, d3, d4, and d7. The margin data set 1001 includes o1 to o7. The margin data set 1001 is expanded in a storage area of ​​the SRAM 401 to determine r1 in the register 402. r1 is an index of the coordinates of the corresponding convolution process. r2 and subsequent values ​​are also indexes of the coordinates of the corresponding convolution processes. After the convolution process is performed using the margin data set 1001 to determine r1, the values ​​of o1, o5, and o6 are discarded, and convolution process is performed using o4, d3, and d6 to determine r2. Thereafter, convolution processes are performed in the same manner, and the results of the convolution processes are transferred to the register 402. During this transfer, a process called padding is performed to fill in the values ​​of areas where there are no pixels with "0." This margin allows some of the position information that would otherwise be lost during the convolution process to be secured, making it possible to improve the accuracy of the feature vector.

[0034] <Modified example of overlap 1> FIG. 12 is a conceptual circuit diagram of a filter constituting the inference unit 105. In the example shown in FIG. 12, the amount of margin used is reduced compared to the example shown in FIG. 10. In the example shown in FIG. 12, the margin data set 1201 is arranged to the left of d1, d4, and d7. According to the example shown in FIG. 12, the margin data set 120 provides spatial locality in the data arrangement in the left-right direction of the storage area of ​​the SRAM 401, which is suitable for cases where data progresses in the left-right direction. Furthermore, even with such margin, some of the position information that would have been lost due to the convolution process can be secured, thereby improving the accuracy of the feature vector.

[0035] <Modified example of overlap 2> FIG. 16 is a conceptual circuit diagram of a filter constituting the inference unit 105. In the example shown in FIG. 16, the amount of margin used is reduced compared to the example shown in FIG. 10. In the example of FIG. 16, the margin data set 1601 is arranged immediately above d1, d2, and d3. According to the example of FIG. 16, the margin data set 1601 provides spatial locality in the data arrangement in the vertical direction of the recording area of ​​the SRAM 401, which is suitable for cases where data progresses vertically. Furthermore, even with such margin, some of the position information that would have been lost in the convolution process can be secured, thereby improving the accuracy of the feature vector.

[0036] <Modified example of overlap margin 3> FIG. 18 is a conceptual circuit diagram of a filter constituting the inference unit 105. In the example shown in FIG. 18, the amount of overlap used is reduced compared to the example shown in FIG. 10. In the example shown in FIG. 18, o1, o2, o3, and o4 in the overlap data set 1801 are spaced apart by one pixel. Also, in the example shown in FIG. 18, o1, o5, o6, and o7 in the overlap data set 1801 are spaced apart by one pixel. According to the example shown in FIG. 18, the overlap data set 1801 provides spatial locality in the data arrangement at equal intervals in the recording area on the SRAM 401, which is suitable for cases where data progresses at a constant pace. Furthermore, even with such an overlap, some of the position information that would have been lost in the convolution process can be secured, thereby improving the accuracy of the feature vector.

[0037] [Learning and Processing Units] FIG. 11 is a conceptual diagram showing images used for learning. Image division and augmentation will be described. An original image 1101 is any image that will be used as the basis for images used for learning. Here, the original image 1101 is divided into several regions. Segmented images 1102 are images obtained by dividing the original image 1101. A set of augmented images 1103 is a group of multiple images generated by processing the segmented images 1102. For example, they are generated through processes such as mirror inversion or partial overwriting of pixels of any image element such as a photograph, text, or graphics. Details of the augmentation method will be omitted.

[0038] [study] FIG. 13 is a flowchart illustrating learning. The processing shown in FIG. 13 may be implemented by CPU 104. The following description will explain an example in which CPU 104 executes the processing. Note that some or all of the functions of the steps in FIG. 13 may be implemented by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process indicates a step in the flowchart.

[0039] The processing shown in FIG. 13 is initiated by user input. A specific embodiment of user input will be described in the third embodiment. In this embodiment, it is assumed that learning is performed based on user input. However, if the learning execution device and the inference execution device are configured as the same information processing device, processing may be initiated based on feedback from the inference execution device.

[0040] In S1301, the CPU 104 divides any one image used for learning into any number of images, thereby obtaining divided images 1102 in FIG. 11. In S1302, the CPU 104 augments the divided images, thereby obtaining a group of augmented images 1103 in FIG. 11. In S1303, the CPU 104 processes any one image obtained from the group of augmented images 1103 using a CNN model. Through the processing of S1303, features are extracted from the group of augmented images 1103. Details of the processing of S1303 will be described using FIG. 17. FIG. 17 is a schematic diagram of CNN processing. FIG. 17 shows an example in which a filter 1702 is applied to an augmented enlarged display image 1701, and calculation results for each pixel are obtained within a bold-framed area 1703.

[0041] In S1304, the CPU 104 stores the extracted feature amounts. For example, the extracted feature amounts are stored in the SRAM 401. In S1305, the CPU 104 determines whether processing has been completed for all padded images. If processing has been completed for all padded images, the CPU 104 advances processing of S1305 to processing of S1306. If processing has not been completed for all padded images, the CPU 104 returns processing of S1305 to processing of S1303. In S1306, the CPU 104 adds up all of the stored information amounts. Specifically, all of the feature amounts stored in the processing of S1304 are added up. The feature amount obtained by adding up all of the feature amounts in this way is hereinafter referred to as the "total feature amount." In S1307, the CPU 104 calculates an error by calculating the difference between the correct feature amount added the same number of times as the number of padding processes and the total feature amount. In S1308, the CPU 104 uses the backpropagation algorithm to propagate errors in the direction opposite to the forward propagation direction, thereby updating the convolution coefficients identified by the filters in each convolution layer. Since the backpropagation algorithm is a well-known technique, a detailed description thereof will be omitted. In S1309, the CPU 104 determines whether error propagation has been completed for all segmented images 1102. If not, the CPU 104 returns the process of S1310 to S1302 and starts padding for the next segmented image. The next process uses convolution coefficients and transposed convolution coefficients that reflect the results of the backpropagation algorithm executed in the previous process. By repeating this backpropagation process, the convolution coefficients and transposed convolution coefficients are sequentially optimized. If the error propagation has been completed, the CPU 104 advances the process of S1309 to S1310. In S1310, the CPU 104 determines whether processing has been completed for all images. If not, CPU 104 returns the process of S1310 to the process of S1301 and performs another division of the original image. If completed, CPU 104 ends the processes of S1310 and S1310. The process of finding the convolution coefficients and transposed convolution coefficients used in the model by propagating the error between the known correct answer and the feature vector obtained from the model in the forward and reverse directions is called learning.The convolution coefficients and transposed convolution coefficients obtained in this manner are stored in advance as parameters in the ROM 106 of the product main body, making it possible to perform inference in the product main body. Note that in this embodiment, image augmentation is performed after image division, but this is not particularly limited. In other words, the original image may be augmented first, and then the image may be divided.

[0042] [Conducting inference] The parameters thus obtained are output as the probability of the recognition result of the type of image the input data is. In this way, a pattern can be identified by evaluating the degree of match with a typical pattern as a probability. Note that in this embodiment, convolution using two-dimensional image data and a two-dimensional filter has been described as an example. However, the application is not limited thereto. That is, a similar configuration may be used when a one-dimensional filter is used for pattern recognition from one-dimensional time-series data such as speech. Alternatively, a similar configuration may be used when a three-dimensional filter is used for pattern recognition from three-dimensional data using voxels. In addition, the effects of the present application can be similarly obtained by constructing a suitable configuration according to the dimensions of the feature.

[0043] Skip Join Details Next, details of skip connections will be described using Figures 14, 15, 19, and 20. First, the configuration of skip connections will be described using Figures 19, 20, and 15, and an example of the operation of skip connections will be described using Figure 14. Figure 19 is a schematic diagram illustrating skip connections using dimensionally reduced feature vectors. Figure 19 shows an example in which the encoding layer 201 includes an output layer 2002 and a next layer 2003. Figure 19 also shows an example in which the decoding layer 202 includes an intermediate layer 2004, a post-upsampling layer 2005, and a next layer 2006. The post-upsampling layer 2005 has the same function as the upsampling layer described above. The intermediate layer 2004 is the final layer to be skipped in the skip connections and includes a convolutional layer. That is, a detour path is formed that starts from an input layer (not shown) and detours between the output layer 2002 and the intermediate layer 2004, and a forward propagation path is formed from the input layer (not shown) to the intermediate layer 2004. Here, the convolutional layers included in the detour path are the convolutional layers included up to the output layer 2002. Although not shown, a convolutional layer is arranged before the output layer 2002. For example, if the model is SegNet or U-Net, multiple convolutional layers and pooling layers are arranged before the output layer 2002. Note that, for example, if the model is U-Net, the information passed during skip coupling is the entire feature, whereas if the model is SegNet, the information passed during skip coupling is the pooling coordinate index. The pooling coordinate index is information indicating the position at which pooling was performed. While FIG. 19 shows an example in which multiple artificial neurons 2001 are included in the output layer 2002, similar artificial neurons are also included in the next layer 2003, intermediate layer 2004, post-upsampling layer 2005, and next layer 2006. Focusing on one artificial neuron 2001, the artificial neuron 2001 receives a feature vector from a previous layer (not shown) and calculates a feature. This feature is assumed to be one channel. For example, the output layer 2002 outputs eight-channel feature vectors. Input data analysis is processed in the forward propagation direction. Therefore, these eight-channel feature vectors are input to the next layer 2003.On the other hand, on the decoding side, feature vectors are input from the hidden layer 2004 to the post-upsampling layer 2005. At this time, the feature vectors from the output layer 2002 and the hidden layer 2004 are dimensionally combined. Here, the feature vectors from the output layer 2002 used for skip combining will be described. In this embodiment, the number of channels in this feature vector is thinned out. For example, of the three RGB channels, only the R channel is discarded, and only the GB2 channel is dimensionally combined. In other words, the dimensions of the feature vectors used for skip combining are limited to between one and seven channels. The greater the number of channels, the more easily the skip combining effect can be achieved. On the other hand, the fewer channels, the more effectively the SRAM storage area used can be reduced. This is because performing skip combining involves a process of successively rewriting the convolution coefficients stored in the SRAM storage area of ​​one filter while also retaining the processing results. Furthermore, thinning out channels and performing skip combining has the effect of reducing the number of stored processing results. The feature vectors thus dimensionally combined are input to the post-upsampling layer 2005. The processing result is input to the next layer 2006 in the decoding layer 202. In this way, it is possible to reduce the amount of SRAM storage space used by skip joining. When thinning out the number of channels, the channels to be thinned out can be selected. For example, it is possible to select a method of collectively thinning out consecutive channels or a method of discretely thinning out channels. Here, eight channels are used as an example, but the number of channels is arbitrary. Furthermore, the layers to be connected can be selected arbitrarily. In this embodiment, an example is shown in which the number of channels of the feature vector is thinned out to reduce the amount of SRAM storage space used by skip joining. However, if the amount of SRAM storage space used by skip joining can be reduced, the target for thinning does not depend only on the number of channels. For example, the data length of the feature vector can be thinned out. For example, only four bits of the eight RGB bits are selected, and the remaining four bits are discarded. In this way, limiting the data length of the feature vector to less than the data length of the original feature vector has the effect of reducing the amount of SRAM storage space used by skip joining.Also, a method of thinning out the number of feature quantities in the calculation results 903 for pixels within the thick-framed area 1703 in FIG. 17 has the effect of reducing the amount of storage area used in the SRAM associated with skip joining.

[0044] Error propagation is performed from the post-upsampling layer 2005 to the hidden layer 2004 in the reverse direction of forward propagation. The convolution coefficients are updated to minimize the gradient loss and minimize the error. This point will be explained further. The output layer 2002 originally outputs eight-channel feature vectors. However, when building a CNN model using machine learning, it is impossible to determine which channels are optimal for data analysis. Therefore, a learning process is performed to assign higher weights to the optimal channels. Therefore, optimizing the weights of the remaining channels results in skip connections using only the significant channels. Alternatively, Fourier series expansion of one-dimensional data can determine the intensity (amplitude) of each frequency. While a low-pass filter would simply cut out the high-frequency components, this is not the case with machine learning. Learning is performed to increase the weights (coefficients) of significant frequency bands depending on the input. As a result, dimensionality reduction is possible by limiting the feature vectors in skip connections to the significant channels. This reduces SRAM storage space usage while mitigating performance degradation. When optimizing the convolution coefficients using such an error backpropagation method, the initial convolution coefficients may start from any value.

[0045] To implement skip connections, it is necessary to temporarily store the output of neurons in each layer in the SRAM storage area. To reiterate, if skip connections are not used, this temporary storage area in the SRAM storage area is unnecessary. Therefore, when processing reaches a layer requiring skip connections, it is possible to re-extract (also referred to as regeneration) the necessary output of the encoding layer. That is, when processing reaches the hidden layer 2004, the CPU 104 stores only the results in the SRAM storage area. The CPU 104 also retrieves input data from the DRAM 103 again and proceeds with processing in the forward propagation direction from the input layer 203. When the CPU 104 reaches the output layer 2002, it performs a dimensional join with the results stored in the hidden layer 2004 and inputs them to the post-upsampling layer 2005. By performing these operations during inference, the CPU 104 can also reduce the amount of SRAM storage space used by the CNN. In addition, in order to reduce the amount of storage space used in the SRAM, a method has been described in which processing proceeds in the forward propagation direction from the input layer 203 in order to re-extract the output of the encode layer 201 required for dimensional combination. However, when re-extracting, processing does not necessarily have to proceed in the forward propagation direction from the input layer 203. For example, by storing the feature vector output from the output layer 2002 in the encode layer 201 in the storage space of the SRAM, processing may start from that feature vector. An example of this will be described using FIG. 15.

[0046] [Variations of skip joins] FIG. 15 is a diagram showing an example in which feature vectors in the fifth layer are skip-connected. FIG. 15 shows an example in which multiple layers are arranged in the encoding layer. Processing in each layer progresses in the forward propagation direction. As processing progresses in the forward propagation direction, the number of dimensions (number of channels) increases. Here, it is also possible to retain features in a layer with 24 dimensions. This operation allows calculations to be restarted from the layer with 24 dimensions, thereby improving calculation efficiency. Note that while the example in FIG. 15 shows a model including an encoding layer, the stage subsequent to the encoding layer is not particularly limited. For example, a model in which a decoding layer is arranged subsequent to the encoding layer may also be used. Alternatively, a model consisting of only an encoding layer or only a decoding layer may also be used.

[0047] In this example, we have described a method for reducing the amount of SRAM storage space required for skip joins by thinning out features to reduce the number of features stored in the SRAM. We have also described a method for not storing features required for skip joins in the SRAM, but re-extracting features when the required layer is reached. Which method to use to reduce SRAM storage space usage can be selected for each layer. Re-extracting features when the required layer is reached is more effective in reducing the amount of SRAM storage space required for skip joins. This is because skip joins can be performed without continuously storing features for skip joins in the SRAM. Therefore, re-extracting features is preferable in terms of reducing SRAM storage space usage. However, this method increases the amount of processing because re-extracting features requires re-executing processes that have already been performed. The choice of which method to use for each layer is a trade-off between SRAM storage space usage and processing speed. This is because, as the amount of processing increases, processes that incorporate parallelism can be executed simultaneously, resulting in an overall increase in processing speed.

[0048] FIG. 20 is a schematic diagram of a model that performs skip connections. When re-extracting features for dimensional connection, the amount of processing required for re-extraction increases with each layer in the encoding layer moving forward in the forward propagation direction. In FIG. 20, output layers 2101, 2103, and 2105 are shown, but it is assumed that at least one convolutional layer is placed before each of them. Furthermore, output layer 2101 is connected to dimensional connection layer 2102 via skip connections. Output layer 2103 is connected to dimensional connection layer 2104 via skip connections. Output layer 2105 is connected to dimensional connection layer 2106 via skip connections. The re-extraction of the outputs of output layer 2101, output layer 2103, and output layer 2105 is described below. The path using skip connections in output layer 2101 is the path that requires the least amount of processing for calculation. Furthermore, the path using skip connections in output layer 2105 is the path that requires the most processing. In this embodiment, the dimensional combination layer 2102 performs dimensional combination using a method of re-extracting feature vectors output from the output layer 2101. The reason for this is that the output layer 2101 has fewer convolutional layers arranged in the preceding stages compared to the output layers 2103 and 2105, thereby significantly reducing the amount of SRAM storage space used for skip combinations while minimizing the amount of processing required for re-extraction. Furthermore, the dimensional combination layer 2104 and the dimensional combination layer 2106 perform dimensional combination using a method of thinning out the feature vectors output from the output layers 2103 and 2105 and storing them in SRAM. The reason for this will be explained using the example of the dimensional combination layer 2106. The dimensional combination layer 2106 needs to re-extract the output of the output layer 2105. However, re-extracting the output layer 2105 requires a large amount of processing. The reason is that the amount of processing required for re-extraction increases with each layer in the encoding layer moving forward in the forward propagation direction. For this reason, a method of storing thinned features in SRAM is selected for the output layer 2103 and the output layer 2105. In this embodiment, a method of re-extracting features is selected for the output layer 2101 closest to the input, and a method of storing thinned features in SRAM is selected for the other output layers 2103 and 2105. However, the selection method is not limited to this. It is possible to select which method is better for each layer, taking into account the amount of reduction in SRAM storage area and processing efficiency.In any case, by re-extracting the feature vector when performing skip joins, it is not necessary to store the feature vectors, which reduces the amount of storage space used in SRAM. Furthermore, when training, feature vectors equivalent to those used in normal skip joins are trained, and the convolution coefficients are optimized, allowing inference with the same accuracy, thereby achieving the effects of the present invention.

[0049] Skip join behavior Fig. 14 is a flowchart showing the re-extraction operation. The process shown in Fig. 14 may be implemented by CPU 104. The following description will explain an example in which CPU 104 executes the process. Note that some or all of the functions of the steps in Fig. 14 may be implemented by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process indicates a step in the flowchart.

[0050] The process shown in FIG. 14 starts when the CPU 104 starts managing skip joins. The CPU 104 determines whether the process has reached the intermediate layer 2004. If the process has reached the intermediate layer 2004, the CPU 104 advances the process of S1401 to S1402. If the process has not reached the intermediate layer 2004, the CPU 104 continues the process of S1401. In S1402, the CPU 104 stores in the SRAM 401 only the results obtained when the process has reached the intermediate layer 2004. In other words, the feature vectors extracted in the convolutional layer preceding the intermediate layer 2004 are not stored. In S1403, the CPU 104 acquires data again from the DRAM 103. Here, the data acquired again by the CPU 104 from the DRAM 103 is input data. In S1404, the CPU 104 proceeds with the process sequentially from the input layer. In other words, the CPU 104 re-obtains feature vectors extracted by each convolutional layer from the input layer to the output layer 2002. In S1405, the CPU 104 determines whether the output layer 2002 has been reached. If the output layer 2002 has been reached, the CPU 104 advances the process of S1405 to S1406. If the output layer 2002 has not been reached, the CPU 104 returns the process of S1405 to S1404 and continues the process of extracting feature vectors for each layer until the output layer 2002 is reached. In S1406, the CPU 104 performs dimensional combination of the result when the intermediate layer 2004 has been reached and the result when the output layer 2002 has been reached. In other words, the CPU 104 performs dimensional combination of the feature vectors re-extracted before the output layer 2002 has been reached and the feature vectors when the intermediate layer 2004 has been reached. During the re-extraction, the connecting elements arranged between the hidden layer 2004 and the post-upsampling layer 2005 are made to wait for the input of the re-extraction result. For example, a buffer (delay element) for the amount required for the re-extraction may be arranged between the hidden layer 2004 and the connecting elements. In S1407, the CPU 104 inputs the result of the dimensional combination to the post-upsampling layer 2005, and ends the processing of S1401 to S1407. Note that in S1401 to S1407, if a pooling layer is arranged after the convolution layer, the feature vectors may be aggregated into a representative value as output data.

[0051] <<Second Embodiment>> In the first embodiment described above, the method for reducing the amount of SRAM storage space used for skip joins can be selected from two options: thinning out features or re-extracting features without storing them in SRAM. Either method reduces the amount of SRAM storage space used. The feature vectors passed to the next layer in the encoding layer and the thinned feature vectors are the same as the feature vectors used in dimensional joins. This poses a problem of reduced accuracy. This embodiment describes a method for improving accuracy while reducing the amount of SRAM storage space used. Figure 21 is a schematic diagram illustrating the details of skip joins. In this embodiment, an encoding layer 2202 is newly added. The encoding layer 2202 has an optional output layer 2203. The output layer 2203 includes multiple artificial neurons 2201. The learning of the CNN model in this embodiment will be described. When an image is input to the CNN model, processing is performed in the forward propagation direction in the encoding layer 201 and the encoding layer 2202. For example, the output layer 2002 outputs eight channels of feature vectors. The feature vectors output from the output layer 2002 are input to the next layer 2003, which is located after the output layer 2002. Meanwhile, in the decode layer 202, the feature vectors are input from the hidden layer 2004 to the post-upsampling layer 2005. At this time, the feature vectors output from the output layer 2203 in the encode layer 2202 and the feature vectors output from the hidden layer 2004 are dimensionally combined. In the first embodiment, the amount of SRAM storage space used was reduced by thinning the features output from the output layer 2002 or by re-extracting the feature vectors from the output layer 2002 during dimensional combination. In this embodiment, the feature vectors output from the output layer 2203 used for dimensional combination can be designed to have the number of channels, data length, and pixel count equal to or smaller than the size of the feature vectors thinned or re-extracted in the first embodiment. This is because the model structure of the encode layer 2202 can be freely changed.For example, a case will be described in which the feature vector output from the output layer 2203 is set to the same size as the feature vector that was thinned or re-extracted for dimension combination in the first embodiment. As a specific example, this will be compared with a case in which the feature vector output from the output layer 2002 is thinned from 8 channels to 4 channels in order to perform dimension combination with the feature vector output from the intermediate layer 2004 in the first embodiment. In the first embodiment, 4 of the 8 channels of the feature vector output from the output layer 2002 are used both as input to the next layer 2003 and as a feature vector for dimension combination. Therefore, in model training, it is not possible to optimize each use, and filter coefficients must be determined so that the feature vector is effective for both uses. On the other hand, in this embodiment, the feature vector output from the output layer 2002 is used only as input to the next layer 2003. Furthermore, when performing dimension combination with the feature vector output from the intermediate layer 2004, the feature vector output from the output layer 2203 is used. These two feature vectors can be optimized during training, thereby improving accuracy while minimizing the amount of storage space used in the SRAM 401. In this embodiment, the feature vector output from the output layer 2203 and the feature vector thinned or re-extracted in the first embodiment are described as having the same size. However, these do not necessarily need to match. In any case, by re-extracting the feature vector when performing skip joining, it is not necessary to store the feature vector, thereby reducing the amount of storage space used in the SRAM. Furthermore, during training, the encoding layer for feature extraction and the encoding layer for skip joining are trained separately, and the filter coefficients are optimized to contribute to accuracy, enabling inference with equal or higher accuracy and achieving the effects of the present invention.

[0052] <<Third Embodiment>> In the first embodiment described above, two methods for reducing SRAM storage usage associated with skip joins are presented: a method for thinning features for each layer, and a method for re-extracting features without storing them in SRAM. When the feature thinning method is selected, the user must select which elements to thin and to what extent based on the number of channels, data length, and number of pixels of the feature vector. In the second embodiment, in addition to the above selection, the user must also select the structure of the CNN model that outputs the feature vector used in dimension joins. Both embodiments achieve the effect of reducing SRAM storage usage. However, the user must select the elements to thin, the degree of thinning, and whether to perform re-extraction. This embodiment describes a method for automatically selecting the SRAM storage usage reduction while reducing SRAM usage. FIG. 22 illustrates an example of a skip join setting screen 2301 used when training a CNN model. The setting screen 2301 is shown in FIG. 22. The setting screen 2301 functions as a user interface that accepts user operations. Therefore, the user can make selections related to reducing SRAM storage usage via the setting screen 1301. An automatic / manual model design switch button 2302 and a learning start button 2307 are located in the upper right area of ​​the setting screen 2301. A selection button 2306, a thinning selection button 2304, and a regeneration selection button 2305 are located in the lower right area of ​​the setting screen 2301. A CNN model structure display screen 2303 is located in the left area of ​​the setting screen 2301. The automatic / manual model design switch button 2302 accepts a selection of whether to automatically or manually perform a selection associated with reducing the amount of SRAM storage space used. In this embodiment, two types of selections, automatic and manual, are accepted, but this is not limiting. For example, more detailed selections may be accepted, such as manually setting some of the model design items and automatically setting the other items. Next, a case will be described in which the automatic / manual model design switch button 2302 is pressed and a selection associated with reducing the amount of SRAM storage space used by the CNN model is automatically performed.Pressing the automatic / manual model design switch button 2302 transitions to a detailed settings screen, which allows the user to set the basis for automatic selection of the reduction in SRAM storage usage. FIG. 23 illustrates an example of the detailed settings screen used when training a CNN model. The detailed settings screen 2401 allows the user to set the extent to which SRAM storage usage associated with skip joins will be reduced. The detailed settings screen 2401 includes a setting bar 2402. The user can change the position of the setting bar 2402 between 0% and 100%. By operating the setting bar 2402, the user accepts the reduction rate of SRAM storage usage associated with skip joins, and the CPU 104 sets the design details of the CNN model based on the reduction rate. Although the present embodiment accepts the reduction rate of SRAM storage usage, this is not limiting. For example, the CPU 104 may accept the number of product-sum operations, processing speed, and SRAM storage usage for the entire CNN model, and then select the reduction rate of SRAM storage usage for the CNN model based on the acceptance rate.

[0053] Next, a description will be given of a case where the automatic / manual model design switch button 1302 is pressed and a manual selection is made to reduce the SRAM storage space usage of the CNN model. A schematic diagram of the CNN model is displayed on the CNN model structure display screen 1303. When manually selecting a method for reducing the SRAM storage space usage of the CNN model, a method for reducing the SRAM storage space usage based on skip connection can be set for each layer that performs skip connection. The thinning selection button 2304 and the regeneration selection button 2305 indicate options for reducing the SRAM storage space usage described in the first embodiment. That is, the CPU 104 accepts a user's selection of either a method for thinning feature vectors or a method for regenerating feature vectors. In this embodiment, the CPU 104 accepts a selection of two types of methods for reducing the SRAM storage space usage described in the first embodiment, but this is not limiting. For example, as described in the second embodiment, a selection of a method for reducing the SRAM storage space usage in which different coefficient filters are used during encoding and dimension connection may be accepted. For each layer where skip coupling is performed, the method for reducing the amount of SRAM storage space used in each layer can be set by selecting either the thinning selection button 2304 or the regeneration selection button 2305 and pressing the selection button 2306. For each layer where skip coupling is performed, when the thinning selection button 2304 is selected and the selection button 2306 is pressed, the screen transitions to a detailed setting screen where the user can set which elements to thin and to what extent. FIG. 24 is a diagram showing another example of a detailed setting screen 2501 for training a CNN model. The detailed setting screen 2501 is a screen transitioned to when the thinning selection button 2304 is selected and the selection button 2306 is pressed. A decision button 2506 is located in the upper right area of ​​the detailed setting screen 2501. A channel selection button 2502, a data length selection button 2503, and a pixel selection button 2504 are located in the upper left area of ​​the detailed setting screen 2501. In the lower left area of ​​the detailed setting screen 2501, a channel selection button 2502, a data length selection button 2503, and a pixel selection button 2504 each accept a selection of which elements to thin out to reduce the amount of storage area used in the SRAM.The setting bar 2505 operates in response to the buttons arranged in the lower left region of the detailed setting screen 2501. That is, when the channel selection button 2502 is selected, the thinning rate for the channel element is accepted. When the data length selection button 2503 is selected, the thinning rate for the data length element is accepted. When the pixel selection button 2504 is selected, the thinning rate for the pixel element is accepted. After selecting these, operating the setting bar 2505, and then pressing the confirm button 2506, the reduction setting for the SRAM memory space usage selected for the layer that performs skip coupling can be set. Return to FIG. 22. When training a CNN model that reduces the SRAM memory space usage, input the settings and press the start learning button 2307 to start learning the CNN model. In this embodiment, an example of the setting screen 2301 for training a CNN model that reduces the SRAM memory space usage associated with skip coupling is shown. However, the configurable items are not limited to this. For example, the system may have a means for inputting learning conditions when learning is performed, a means for interrupting learning midway, a means for executing inference, etc. Furthermore, the form of the screen is not limited to this form, and the layout and input means may differ from those in this embodiment.

[0054] Next, an example of a method for automatically selecting a CNN model to reduce the amount of SRAM storage space used will be described. A method called Neural Architecture Search (NAS) is generally used to search for a highly accurate CNN model. This method can search for a highly accurate model structure and a selection that reduces the amount of SRAM storage space used. The reason for this is that the quality of the model structure or design setting values ​​can be reflected in the error during learning. FIG. 25 is a flowchart illustrating automatic CNN model design. The process shown in FIG. 25 may be implemented by the CPU 104. The following description will discuss an example in which the CPU 104 executes the process. Note that some or all of the functions of the steps in FIG. 25 may be implemented by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process indicates the step in the flowchart.

[0055] The process shown in FIG. 25 is initiated based on a user input from the learning start button 1307. In the process shown in FIG. 25, steps identical to those shown in FIG. 13 are treated as identical steps, and their description will be omitted. In S2501, the CPU 104 sets model design specifications based on a probability distribution. Model design specifications include, for example, model structure and selections related to reducing SRAM storage usage. The model structure refers to the model architecture, such as a structure specified by SegNet, U-Net, or ResNet. This probability distribution may be random at the beginning of learning. As learning progresses, this probability distribution changes so that a setting value that reduces the error from the correct answer calculated in S1307 is more likely to be selected. In S2502, the CPU 104 receives the user setting values ​​shown in FIG. 25 set by the user via the setting screen 1301. The CPU 104 determines whether the set CNN model satisfies the user setting values ​​shown in FIG. 25. Here, the term "penalty" is defined. The penalty is an evaluation index that reflects whether the CNN model set in S2501 satisfies the user setting value in FIG. 25. In this embodiment, if the CNN model satisfies the user setting value in FIG. 25, the penalty value is set to 0. On the other hand, if the CNN model does not satisfy the user setting value in FIG. 25, the penalty value increases according to the degree of deviation from the user setting value in FIG. 25. For example, if a user sets a 50% reduction rate in SRAM storage space usage due to skip joins, that setting value is input as the user setting value in S2502 in FIG. 25. In S2501, the CPU 104 sets a method for reducing SRAM storage space usage for each layer that performs skip joins. Therefore, it is possible to calculate the extent to which SRAM storage space usage can be reduced compared to a case where no reduction is made. The CPU 104 compares the value with the user setting value in FIG. 25. If the user setting value in FIG. 25 is satisfied, the output of S1307 is used as is in S1308. On the other hand, if the user setting values ​​in FIG. 25 are not satisfied, the CPU 104 reflects the penalty value in the output of S1307 and uses it in S1308.This is because it is necessary to make the combination of setting values ​​selected in S2501 less likely to be selected. For example, if the reduction rate of SRAM storage space usage is 50% or more, no penalty is applied. If the reduction rate is less than 50%, a penalty corresponding to the insufficient reduction rate is applied. In this embodiment, a penalty is applied to the output in S1307. However, the method of applying the penalty is not limited to this. For example, the output in S1307 may be multiplied according to the insufficient reduction rate. This operation allows the value used in S1308 to reflect not only the determination performance of the CNN model but also the merits or demerits of the selection associated with the reduction in SRAM storage space usage set in S2501. The backpropagation performed in S1308 can optimize the convolution coefficients of the CNN model and the selection associated with the reduction in SRAM storage space usage set in S2501. In this embodiment, the process of S2502 is executed between S1307 and S1308, but this is not particularly limited. The evaluation in S2502 may be performed at any time between S1301 and S1308. Alternatively, S2502 does not need to be performed for each repetition of the processing between S1301 and S1310. In any case, by re-extracting feature vectors when performing skip joins, it is not necessary to store the feature vectors, and the amount of SRAM used can be reduced. Furthermore, when performing learning, selections that reduce the amount of SRAM storage used are automatically learned, and the convolution coefficients, model structure, and setting values ​​are optimized to contribute to accuracy, thereby suppressing a decrease in accuracy and achieving the desired effect.

[0056] While various examples and embodiments of the present disclosure have been shown and described above, the spirit and scope of the present disclosure are not limited to the specific descriptions within this specification.

[0057] For example, in the above-described embodiment, an example of a CNN model including an encoding layer and a decoding layer was described. Furthermore, as a method for reducing the amount of SRAM storage space used due to dimensional merging, an example of dimensional merging between the output of an output layer in the encoding layer and the output of an intermediate layer in the decoding layer was given. However, the CNN model does not necessarily have to include an encoding layer and a decoding layer. For example, dimensional merging can also be performed in a CNN model that includes an encoding layer but does not include a decoding layer. Alternatively, outputs from two or more output layers in the encoding layer may be dimensionally merged and used as input to the next layer. Even in such a CNN model, the effect of reducing the amount of SRAM storage space used due to dimensional merging can be achieved by using the method used in the above-described embodiment.

[0058] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0059] The disclosure of the present embodiment includes configurations typified by the following information processing device, information processing method, and program.

[0060] <Configuration 1> a convolutional hierarchy including a plurality of convolutional layers, each of which propagates output data based on a feature vector extracted from input data input from a previous stage to a subsequent stage; a combining means for combining a forward propagation path that sequentially propagates the output data to each of the convolutional layers located between a part of the convolutional layers and another part of the convolutional layers among the plurality of convolutional layers, and a detour path that detours around the forward propagation path and propagates the output data from the part of the convolutional layers to the other part of the convolutional layers; a processing means for performing a process of extracting the feature vector from the input data in each of the plurality of convolution layers; Equipped with the processing means performs a re-extraction process of re-extracting the feature vectors included in the plurality of convolutional layers up to a convolutional layer where the detour by the detour path starts, The information processing device is characterized in that, when the re-extraction process is performed by the processing means, the combining means combines the output result from the forward propagation path with the result of the re-extraction process performed by the processing means.

[0061] <Configuration 2> the input data is composed of a plurality of pixels; Each of the plurality of convolution layers has a filter for which a plurality of convolution coefficients are specified; The information processing device according to configuration 1, wherein the processing means extracts the feature vector by performing convolution processing based on the plurality of pixels and the plurality of convolution coefficients in each of the plurality of convolution layers.

[0062] <Configuration 3> The convolutional layer includes a plurality of pooling layers; 3. The information processing device according to configuration 2, wherein each of the plurality of pooling layers is arranged after each of the plurality of convolution layers, and aggregates the feature vectors into a representative value as the output data.

[0063] <Configuration 4> An upsampling layer is further provided at a subsequent stage of the convolution layer to expand the output data, The information processing device according to configuration 3, wherein the processing means expands the output data in the upsampling layer to increase the size of the representative value to the input data, and outputs the result as subsequent data.

[0064] <Configuration 5> an activation layer arranged downstream of the upsampling layer and configured to reconstruct downstream image data onto which the downstream data is mapped; 5. The information processing device according to configuration 4, wherein the processing means classifies the subject appearing in the image data composed of the plurality of pixels based on the subsequent image data reconstructed by the activation layer.

[0065] <Configuration 6> An activation layer is further provided at a rear stage of the convolution layer, and reconstructs the rear stage image data to which the representative values ​​are mapped, 4. The information processing device according to configuration 3, wherein the processing means classifies the subject appearing in the image data composed of the plurality of pixels based on the subsequent image data reconstructed by the activation layer.

[0066] <Configuration 7> each of the plurality of convolutional layers includes a plurality of artificial neurons; Each of the plurality of artificial neurons a convolution unit that performs the convolution process using the convolution coefficients; an activation function unit that calculates feature quantities that are components of the feature quantity vector based on the result of the convolution processing output from the convolution unit; and 7. The information processing device according to configuration 5 or 6, wherein the processing means calculates the convolution coefficients based on the subsequent-stage image data reconstructed by the activation layer and the input data.

[0067] <Configuration 8> The information processing device according to configuration 2, wherein the processing means performs product-sum on the input data while shifting the filter by a constant stride, thereby obtaining a feature representing a local feature of the input data for each shift of the filter, and extracts a set of the obtained feature values ​​as the feature vector.

[0068] <Configuration 9> a first storage device that functions as a main memory; a second storage device that functions as a cache memory; Furthermore, the first storage device stores the input data; 9. The information processing device according to any one of configurations 1 to 8, wherein the second storage device stores the feature vectors extracted in each of the plurality of convolutional layers.

[0069] <Configuration 10> 10. The information processing device according to configuration 9, wherein the processing means acquires the input data from the first storage device when performing the re-extraction process.

[0070] <Configuration 11> the first storage device is composed of a DRAM, 11. The information processing device according to configuration 9 or 10, wherein the second storage device is configured by an SRAM.

[0071] <Configuration 12> 3. The information processing device according to configuration 2, wherein the convolutional layer is configured such that divided data obtained by dividing image data made up of the input data into certain spatial regions is input to the convolutional layer.

[0072] <Configuration 13> An information processing method for an information processing device having a plurality of convolution layers, a step of propagating output data based on feature vectors extracted from input data input from a previous stage side, from each of the plurality of convolution layers to a subsequent stage side; a step of combining a forward propagation path that propagates the output data in order to each convolutional layer between some convolutional layers and other some convolutional layers among the plurality of convolutional layers, and a detour path that detours around the forward propagation path and propagates the output data from some convolutional layers to some other convolutional layers; performing a process of extracting the feature vector from the input data in each of the plurality of convolutional layers; Including, the step of performing the processing performs a re-extraction process of re-extracting the feature vectors included in the plurality of convolutional layers up to the convolutional layer where the detour by the detour path starts, The information processing method is characterized in that, when the re-extraction process is performed in the processing step, the combining step combines the output result from the forward propagation path with the result of the re-extraction process performed in the processing step.

[0073] <Configuration 14> 14. A program for causing a computer to execute each step of the information processing method according to claim 13. [Explanation of symbols]

[0074] 100 Inference execution device 101 Data Transfer Interface 102 Data Bus 103 DRAM 104 CPU 105 Reasoning part 106 ROM

Claims

1. a convolutional hierarchy including a plurality of convolutional layers, each of which propagates output data based on a feature vector extracted from input data input from a previous stage to a subsequent stage; a combining means for combining a forward propagation path that sequentially propagates the output data to each of the convolutional layers located between a part of the convolutional layers and another part of the convolutional layers among the plurality of convolutional layers, and a detour path that detours around the forward propagation path and propagates the output data from the part of the convolutional layers to the other part of the convolutional layers; a processing means for performing a process of extracting the feature vector from the input data in each of the plurality of convolution layers; Equipped with the processing means performs a re-extraction process of re-extracting the feature vectors included in the plurality of convolutional layers up to a convolutional layer where the detour by the detour path starts, The information processing device is characterized in that, when the re-extraction process is performed by the processing means, the combining means combines the output result from the forward propagation path with the result of the re-extraction process performed by the processing means.

2. the input data is composed of a plurality of pixels; Each of the plurality of convolution layers has a filter for which a plurality of convolution coefficients are specified; 2. The information processing apparatus according to claim 1, wherein the processing means extracts the feature vector by performing convolution processing based on the plurality of pixels and the plurality of convolution coefficients in each of the plurality of convolution layers.

3. The convolutional layer includes a plurality of pooling layers; 3. The information processing device according to claim 2, wherein each of the plurality of pooling layers is arranged after each of the plurality of convolution layers, and aggregates the feature vectors into a representative value as the output data.

4. An upsampling layer is further provided at a subsequent stage of the convolution layer to expand the output data, 4. The information processing apparatus according to claim 3, wherein the processing means expands the output data in the upsampling layer to increase the size of the representative value to the input data, and outputs the result as subsequent data.

5. an activation layer arranged downstream of the upsampling layer and configured to reconstruct downstream image data onto which the downstream data is mapped; 5. The information processing apparatus according to claim 4, wherein said processing means classifies the objects appearing in the image data composed of the plurality of pixels based on the subsequent image data reconstructed by the activation layer.

6. An activation layer is further provided at a rear stage of the convolution layer, and reconstructs the rear stage image data to which the representative values ​​are mapped, 4. The information processing apparatus according to claim 3, wherein said processing means classifies the objects appearing in the image data composed of the plurality of pixels based on the subsequent image data reconstructed by the activation layer.

7. each of the plurality of convolutional layers includes a plurality of artificial neurons; Each of the plurality of artificial neurons a convolution unit that performs the convolution process using the convolution coefficients; an activation function unit that calculates feature quantities that are components of the feature quantity vector based on the result of the convolution processing output from the convolution unit; and 6. The information processing apparatus according to claim 5, wherein said processing means calculates said convolution coefficients based on said subsequent-stage image data reconstructed by said activation layer and said input data.

8. 3. The information processing device according to claim 2, wherein the processing means performs product-sum on the input data while shifting the filter at a constant stride, thereby obtaining a feature representing a local feature of the input data for each shift of the filter, and extracting a set of the obtained feature values ​​as the feature vector.

9. a first storage device that functions as a main memory; a second storage device that functions as a cache memory; Furthermore, the first storage device stores the input data; The information processing device according to claim 1 , wherein the second storage device stores the feature vectors extracted in each of the plurality of convolution layers.

10. 10. The information processing apparatus according to claim 9, wherein said processing means, when performing said re-extraction processing, acquires said input data from said first storage device.

11. the first storage device is composed of a DRAM, 10. The information processing apparatus according to claim 9, wherein the second storage device is an SRAM.

12. 3. The information processing apparatus according to claim 2, wherein the image data formed from the input data is divided into predetermined spatial regions, and divided data is input to the convolution layer.

13. An information processing method for an information processing device having a plurality of convolution layers, a step of propagating output data based on feature vectors extracted from input data input from a previous stage side, from each of the plurality of convolution layers to a subsequent stage side; a step of combining a forward propagation path that propagates the output data in order to each convolutional layer between some convolutional layers and other some convolutional layers among the plurality of convolutional layers, and a detour path that detours around the forward propagation path and propagates the output data from some convolutional layers to some other convolutional layers; performing a process of extracting the feature vector from the input data in each of the plurality of convolutional layers; Including, the step of performing the processing performs a re-extraction process of re-extracting the feature vectors included in the plurality of convolutional layers up to the convolutional layer where the detour by the detour path starts, The information processing method is characterized in that, when the re-extraction process is performed in the processing step, the combining step combines the output result from the forward propagation path with the result of the re-extraction process performed in the processing step.

14. A program for causing a computer to execute each step of the information processing method according to claim 13.

Images