Information processing apparatus, embedded apparatus, inference method, and program

Abstract

To reduce a storage capacity required for convolution computation of a convolutional neural network while suppressing a decline in reliability of a computation result.SOLUTION: An information processing apparatus executes inference using a convolutional neural network, and includes: acquisition means that acquires target data from data for inference input in the information processing apparatus; and computation means that executes convolution computation using computation data including the target data acquired by the acquisition means and margin data different from the target data that is required to obtain computation result data in a predetermined size, and outputs the computation result data. The acquisition means acquires first data, which is part of the margin data, from a data group existing around the target data separately from the target data in the data for inference, and does not acquire second data, which is the margin data except the first data, from the data group.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] This disclosure relates to inference using convolutional neural networks. [Background technology]

[0002] Convolutional neural networks (CNNs) are one of the preferred methods for pattern recognition. CNN-based inference involves repeated execution of massive convolution operations using multi-stage filters to extract specific features. This requires hardware resources commensurate with the amount of computation required. Meanwhile, due to demands for compactness and cost reduction in products, it is not always possible to equip these with sufficient hardware resources. In particular, SRAMs used as CNN filters tend to be costly because they increase the area of ​​integrated circuits, making capacity reduction a challenge.

[0003] To address this issue, Patent Document 1 discloses a technique for decomposing the weight matrix (filter) of a machine learning model into multiple matrices with a predetermined width, thereby changing the model to a machine learning model of any size while maintaining as much inference accuracy as possible. Zero padding is also known as a technique for reducing SRAM capacity. In image convolution, a square filter is scanned over the input image, performing local product-sum operations and concentrating values ​​on the pixel at the center of the filter. However, if the filter is located near the edge of the input image, part of the filter extends outside the image area. Zero padding is a process in which the protruding peripheral pixels are filled with "0" instead of being acquired from the input image. In other words, zero padding eliminates the need to store data from the peripheral area of ​​the input image in SRAM, thereby reducing SRAM capacity. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-012553 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when zero padding is performed to reduce SRAM capacity, the data used in the convolution calculation contains data that is not the true value, which reduces the reliability of the calculation results.In other words, when implementing CNN in a product, trying to increase the reliability of the calculation results becomes difficult due to hardware resource constraints, and choosing a feasible method results in a problem of reduced reliability of the calculation results.

[0006] An object of the present disclosure is to reduce the storage capacity required for CNN convolution operations while suppressing a decrease in the reliability of the operation results. [Means for solving the problem]

[0007] The present disclosure provides an information processing device that performs inference using a convolutional neural network, comprising: an acquisition means that acquires target data from inference data input to the information processing device; and a calculation means that performs a convolution operation using calculation data including the target data acquired by the acquisition means and margin data, which is data separate from the target data and is necessary to obtain calculation result data of a predetermined size, and outputs the calculation result data, wherein the acquisition means acquires first data, which is a portion of the margin data, from a data group that exists around the target data in the inference data separately from the target data, and does not acquire second data of the margin data excluding the first data from the data group. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to reduce the storage capacity required for CNN convolution operations while suppressing a decrease in the reliability of the operation results. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of a printer as an example of an information processing apparatus. [Figure 2] FIG. 1 is a conceptual diagram showing an example of a model structure of a CNN that constitutes an inference unit. [Figure 3] FIG. 2 is a conceptual diagram illustrating the internal configuration of a filter. [Figure 4] FIG. 2 is a diagram illustrating the functional configuration of an inference unit and the data processing process. [Figure 5] FIG. 10 is a diagram illustrating the division of image data for inference and margin data of divided image blocks. [Figure 6] 10 is a flowchart illustrating the flow of inference. [Figure 7] FIG. 10 is a diagram illustrating a data acquisition method as a comparative example. [Figure 8] 8 is a flowchart illustrating the flow of the convolution operation process in the case of FIG. 7. [Figure 9] FIG. 8 is a diagram illustrating the calculation procedure of the convolution operation in the case of FIG. 7. [Figure 10] FIG. 2 is a diagram illustrating a data acquisition method according to the first embodiment of the present disclosure. [Figure 11] 11 is a flowchart illustrating the flow of the convolution operation process in the case of FIG. 10. [Figure 12] 11 is a diagram for explaining the calculation procedure of the convolution operation in the case of FIG. 10. FIG. [Figure 13] FIG. 1 is a diagram illustrating a method of accessing a DRAM. [Figure 14] FIG. 10 is a diagram illustrating a data acquisition method according to a first modified example of the first embodiment. [Figure 15] FIG. 15 is a diagram for explaining the calculation procedure of the convolution operation in the case of FIG. 14. [Figure 16] FIG. 10 is a diagram showing a data acquisition method according to Modification 2. [Figure 17] FIG. 10 is a diagram showing an example of margin data spanning multiple lines. [Figure 18] FIG. 10 is a diagram showing a data acquisition method according to Modification 3. [Figure 19]FIG. 2 is a diagram illustrating the relationship between a printer as an inference device and a learning device. [Figure 20] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning device. [Figure 21] FIG. 2 is a diagram illustrating the functional configuration of a learning device and a data processing process. [Figure 22] FIG. 1 is a schematic diagram showing the vicinity of an input section of a typical CNN model. [Figure 23] FIG. 2 is a schematic diagram showing an overview of a processing section in a processing layer. [Figure 24] FIG. 1 is a schematic diagram showing the vicinity of an output section of a CNN. [Figure 25] 10 is a flowchart illustrating the overall flow of a learning process executed by the learning device. [Figure 26] FIG. 10 is a diagram illustrating an example of a UI screen for setting a model structure and model conditions. [Figure 27] 10 is a flowchart illustrating the flow of a learning process. [Figure 28] FIG. 10 is a diagram illustrating division and padding of training data. [Figure 29] 10 is a flowchart illustrating the overall flow of a learning process in the third embodiment. [Figure 30] FIG. 10 is a diagram illustrating a system configuration and a functional configuration of an information processing system according to a third embodiment. [Figure 31] FIG. 10 is a diagram showing a UI screen for print settings as an example of use case settings. [Figure 32] FIG. 11 is a diagram illustrating a system configuration and a functional configuration according to a modified example of the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present disclosure is not limited to the illustrated configurations. First, the terms used in this specification will be explained.

[0011] <Term definition> Neurons: A neuron is a processing unit consisting of a filter and an activation function. The coefficients of the filter are called "filter coefficients," "weights," "neuron weights," etc. A neuron performs a convolution operation using the data that is the subject of the operation performed by the filter (hereinafter referred to as "operation data"), and obtains "operation result data." The "calculation data" includes "target data" and "margin data." In this specification, "target data" refers to image block data obtained by dividing externally input image data for inference or learning data (e.g., image data in units of pages) into predetermined sizes. The image data in units of pages from which target data is obtained is also referred to as the "original image." The unit of the original image is not limited to pages and can be any unit. "Margin data" is data that exists on the periphery of the target data and extends beyond the filter during convolution operations on the target data. The width of the margin data is determined by the model structure, including the filter size and layer structure. For example, when a 3x3 filter is used, the calculation data includes the target data and one line of margin data (row and column) above, below, left, and right of the target data. Also, when a 5x5 filter is used, the calculation data includes the target data and two lines of margin data (row and column) above, below, left, and right of the target data. In either case, the size of the calculation result data is the same as the target data. "Reference data" (first data) is data among the margin data that the inference unit acquires from the original image. Reference data is actual data acquired from a data group existing around the target data (image block) in the original image. Note that the filter size and data size are examples for explanation purposes, and are not limited to these and may be any values. Note that, as will be described in detail later, in the present disclosure, data among the margin data excluding the reference data (second data) is not acquired from the above data group. "Input data" is a data group including "target data" and "reference data", which is obtained from the original image and stored in the SRAM of the inference unit.

[0012] Activation function: An activation function is a function with nonlinear response characteristics. Sigmoid functions and ReLU functions are often used because the relationship between input and output is expected to have nonlinear response characteristics.

[0013] layer: A layer is a processing unit consisting of multiple neurons. In principle, common calculation data is input to each neuron. However, different weights may be set for the filter coefficients (weights) of each neuron depending on the features to be obtained. The reason why a layer is composed of multiple neurons is to analyze the calculation data from multiple angles.

[0014] Features: The output from one neuron is called a feature, and different neurons output different feature intensities.

[0015] Feature vector: A vector consisting of features output from one layer. The dimension of a vector is also referred to as a "channel" below. The terms "dimension" and "channel" are used differently depending on the context, as conventions change which term is more appropriate for the context.

[0016] The following embodiments will be described using a printer as an example of an information processing device and an embedded device according to the present disclosure. Note that the present disclosure is not limited to printers, but can also be applied to image forming devices such as MFPs (Multifunction Peripherals), various image processing devices such as imaging devices and video devices, and general information processing devices such as PCs and smartphones. Furthermore, while the following description deals with image data as an example, the data to be processed by the present disclosure is not limited to two-dimensional data such as image data, but can also be applied to one-dimensional time-series data such as audio data. In this case, the present disclosure can also be applied to embedded devices such as audio equipment and lighting equipment.

[0017] First Embodiment In the first embodiment, a printer will be described as an example of an information processing device that executes inference using a CNN (convolutional neural network).

[0018] (Hardware configuration) Fig. 1 is a block diagram showing the hardware configuration of a printer, which is an example of an information processing apparatus according to this embodiment. As shown in Fig. 1, the printer 100 includes a CPU 101, a ROM 102, a RAM 103, an inference unit 104, a data transfer I / F 105, an operation panel 106, and a printing unit 107. The ROM 102, the RAM 103, the inference unit 104, the data transfer I / F 105, the operation panel 106, and the printing unit 107 are connected to the CPU 101 via a data bus 108.

[0019] The data transfer I / F 105 is an interface for inputting and outputting data to and from an external device (not shown). The connection method for the data transfer I / F 105 is not particularly limited, and may be, for example, USB, IEEE1394, or the like. The connection may be wired or wireless. The external device may be, for example, a personal computer, a personal digital assistant, a smartphone, or the like, and is capable of generating and storing image data to be used for inference and transferring the data to the printer 100. The data transfer I / F 105 transfers the image data for inference input from the external device to the CPU 101 via the data bus 108.

[0020] The data bus 108 is a data transmission path through which the image data for inference received from the data transfer I / F 105 is input to the CPU 101 and the data output from the CPU 101 is transferred to each unit within the printer 100 . The RAM 103 is a storage area that temporarily stores data received from the data transfer I / F 105, and is configured with a volatile memory such as a DRAM (Dynamic Random Access Memory). The RAM 103 is also used as a work memory for processing executed by the CPU 101.

[0021] The CPU 101 calls a program stored in the ROM 102, loads it into the RAM 103, and executes processing in accordance with the program while using the RAM 103 as a work memory. For example, the CPU 101 transfers the image data for inference stored in the RAM 103 to the inference unit 104 via the data bus 108. The CPU 101 also sets information necessary for inference, such as division conditions for the image data for inference, reference data acquisition conditions, padding conditions, and filter coefficients, in the inference unit 104. This information may be stored in advance in the ROM 102 as parameters, or may be set by the program.

[0022] The inference unit 104 sets the filter coefficients input from the CPU 101 in the filter. The inference unit 104 also acquires calculation data from the image data for inference of a predetermined unit size stored in the DRAM 103 based on the division conditions, the reference data acquisition conditions, and the padding conditions. The inference unit 104 executes a convolution operation using the filter coefficients and the acquired calculation data to obtain calculation result data. By repeatedly executing such multiple filter processes, the inference unit 104 outputs a feature vector as the inference result and transfers it to the CPU 101. The configuration and processing of the inference unit 104 will be described in detail later.

[0023] ROM 102 is a non-volatile memory that stores the programs according to this embodiment, an OS (Operating System), and data necessary for processing. The programs include a processing program that the CPU 101 uses to cause the inference unit 104 to execute inference using CNN. ROM 102 also stores information necessary for inference, such as division conditions for image data for inference, reference data acquisition conditions, padding conditions, and filter coefficients. Inference and this information will be described later. In this embodiment, it is assumed that the filter coefficients are values ​​that have been learned in advance by a learning device separate from the printer 100 and are stored in ROM 102.

[0024] Printing unit 107 performs printing operations based on instructions from CPU 101. The printing method of printing unit 107 is not particularly limited, and may be, for example, an inkjet method or an electrophotographic method.

[0025] Note that the configuration of the printer 100 described above is an example, and the present disclosure is not limited thereto. For example, any storage medium may be used instead of the ROM 102. The storage medium may be, for example, an HDD or an external memory via a USB interface. In this embodiment, inference is performed by the inference unit 104. However, firmware for realizing processing equivalent to that performed by the inference unit 104 may be stored in the storage medium, and the CPU 101 may execute the firmware to perform the processing of the inference unit 104. Furthermore, as part of functional expansion, the block size (division size) of the target data acquired by the inference unit 104 may be a parameter that can be set by the user.

[0026] The operation panel 106 includes an input unit through which the user inputs operations for the printer 100, and a display unit that displays various information such as the status of the printer 100 and setting information for the printer 100. The input unit is configured, for example, with a touch panel, hard keys, etc., and inputs the input information to the CPU 101. The display unit has a display such as an LCD and a display control circuit, and displays the information input from the CPU 101 on the display.

[0027] (Inference part) FIG. 2 is a conceptual diagram showing an example of a model structure of a CNN (convolutional neural network) constituting the inference unit 104. The inference unit 104 has an encoding unit 201 and a decoding unit 202 to perform inference using the CNN. The encoding unit 201 is a collection of several processing layers, which will be described later. Throughout the encoding unit 201, the features of the data to be processed are encoded. The decoding unit 202 decodes the processing results obtained by the encoding unit 201 and extracts a feature vector.

[0028] The input layer 203 of the encoding unit 201 is the first processing layer that processes the data to be processed. While the processing layer is composed of multiple filters, multiple filters are not necessarily required as hardware. Multiple filters may be implemented by repeatedly using a single filter prepared as hardware. That is, while there is only one hardware filter including SRAM, arithmetic circuits, and registers, two consecutive filter processes can be implemented by gradually updating the filter coefficients and using the calculation results of one filter as input data for the next filter. The input layer 203 is illustrated as an example of such a layer. The intermediate processing layer 204 following the input layer 203 receives the calculation results of the input layer 203 and implements subsequent processing. Like the input layer 203, the intermediate processing layer 204 is also composed of multiple filters. By performing multiple filter processes in this way, the encoding unit 201 encodes the image data to be processed.

[0029] Similar to the encoding unit 201, the decoding unit 202 is also provided with multiple processing layers 205 for performing multiple filter processes. The final output from the processing layers 205 of the decoding unit 202 is uniquely determined by the activation function in the final layer 206 of the decoding unit 202. In this way, the probability of the attribute of the pixel of interest for the image data to be processed is determined. In this way, several layers are constructed by combining multiple neurons in a CNN, and encoding and decoding are performed by combining the multiple constructed layers. A feature vector is obtained through these processes.

[0030] (Filter configuration) FIG. 3 is a conceptual diagram showing the internal configuration of a filter constituting the inference unit 104 shown in FIG. 2. The filter 300 includes an SRAM 310 in which input data, output data, and a filter coefficient group are loaded, a coefficient register 322 in which filter coefficients used in convolution calculations are set, and an operation register 321 in which data of an operation range (operation window) is loaded. As described above, the inference unit 104 includes one or more filters 300 for extracting one or more features. Note that, to realize multiple filters, the inference unit 104 may include multiple filters 300 as hardware, or one filter 300 may be used repeatedly while changing the filter coefficients. In either case, the inference unit 104 may include the filters necessary to configure and implement the CNN shown in FIG. 2.

[0031] In the following description, the SRAM 310 has an input data area 311, a filter coefficient area 312 that stores filter coefficients, and an output data area 313 that stores the processing results of the convolution operation. In this embodiment, the input data area 311 stores target data extracted from image data stored in the DRAM 103 and reference data, which is a portion of margin data required for the convolution operation of the target data. This target data and reference data are referred to as input data. The target data is divided data extracted from image data (image data for inference) stored in the DRAM 103, but it does not necessarily have to be divided data. The image data stored in the DRAM 103 is image data with a page size as one unit, for example.

[0032] The target data is one block of image data obtained by dividing one page of image data (inference image data) into multiple blocks, and the division size is arbitrary. For example, one page of data may be divided into tiles vertically and horizontally, or into strips in only one direction, either vertically or horizontally. In the following description, the divided image blocks are referred to as target data. The inference unit 104 divides the image data stored in the DRAM 103 into multiple image blocks and expands them in the SRAM 310 of the filter 300, thereby reducing the capacity of the SRAM 310. In the example of FIG. 3, image blocks of 8 pixels x 9 pixels are expanded in the input data area 311 of the SRAM 310.

[0033] It is preferable that the division size be determined according to the capacity of the SRAM 310. A required amount of data is read from the DRAM 103 according to the processing unit, and is expanded in the SRAM 310. Although an image block of 8 pixels by 9 pixels is shown in FIG. 3, in practice, data of a larger size is often expanded in the input data area 311. Furthermore, in the present disclosure, the processing unit in the inference unit 104 is not necessarily limited to a divided image block, and may be image data in units of pages held in the DRAM 103.

[0034] Filter coefficients are stored in a filter coefficient area 312 of the SRAM 310. The filter coefficients are obtained from the ROM 102 and stored in the SRAM 310. FIG. 3 shows an example in which the size of the filter coefficients (hereinafter referred to as the filter size) is 3×3. Although FIG. 3 shows an example in which one filter coefficient is stored, the filter coefficient area 312 stores multiple filter coefficients used by each of the multiple filters that make up the CNN. During execution of a convolution operation, one filter coefficient to be used for the ongoing convolution operation from the multiple filter coefficients stored in the filter coefficient area 312 is set in the coefficient register 322. Note that the method of storing filter coefficients in the SRAM 310 is not limited to this example, and one filter coefficient may be stored per SRAM 310.

[0035] First, data within the calculation range of the filter size from the input data expanded in the input data area 311 of the SRAM 310 is set in the calculation register 321. One filter coefficient to be used for the convolution calculation from the filter coefficients held in the filter coefficient area 312 of the SRAM 310 is set in the coefficient register 322. In the convolution calculation, the data held in the calculation register 321 is multiplied by the coefficient held in the coefficient register 322, and updated to the value of the multiplication result. The sum of all the multiplication result values ​​held in the calculation register 321 is collected at the center pixel of the calculation range and output as the convolution calculation result. This convolution calculation result (output data) is held at the corresponding pixel position in the output data area 313 of the SRAM 310. This calculation process is repeated while sliding the calculation range in a predetermined sliding direction. The above process is called filtering of the calculation data. The filtering process will be described later.

[0036] 3 is an example, and data may be stored in other ways. For example, in FIG. 3, an example has been described in which the data in the calculation range and the data resulting from the multiplication are stored in the same calculation register 321, but this is not limiting, and the data in the calculation range and the data resulting from the multiplication may each be stored in a different register. Also, while the result of the convolution calculation is stored in the output data area 313 in the same SRAM 310 as the SRAM 310 in which the input data is stored, this is not limiting, and the result may be stored in a memory (SRAM or DRAM 103, etc.) separate from the input data area 311.

[0037] (Data processing process) 4 is a block diagram showing the functional configuration and data processing process of the inference unit 104. As shown in Fig. 4, the inference unit 104 has an acquisition unit 410, a convolution operation unit 430, a padding unit 440, an output unit 450, etc. The acquisition unit 410 has a data division unit 411. These functional units are realized, for example, by the CPU 101 executing a program stored in the ROM 102.

[0038] ROM 102 stores in advance the division conditions, reference data acquisition conditions, padding conditions, and filter coefficients for the target data acquired by inference unit 104. The division conditions are division position 401 and division size 402 shown in FIG. 4. The reference data acquisition condition is reference data position 403 shown in FIG. 4. The padding conditions are padding method 405 and padding position 406 shown in FIG. 4. Filter coefficients 404 include multiple filter coefficients used in all layers of inference unit 104, and are filter coefficients that have been learned by an external learning device.

[0039] The division position 401 is information indicating the position of an image block in an original image when dividing the original image into image blocks. In the case of image data, the division position is specified by, for example, coordinates in the original image. The division size 402 is information indicating the size of an image block. For image data, the vertical and horizontal sizes are specified. The division size 402 is arbitrary, but is preferably determined according to the capacity of the memory (SRAM 310) provided in the inference unit 104. Furthermore, the minimum size of the division size 402 is determined depending on the structure of the CNN built in the inference unit 104. Factors that determine the structure of the CNN include the filter size of the layer, the number of stages, the number of reductions and enlargements, etc.

[0040] The reference data position 403 is information for specifying the position (pixel) of the original image from which the acquisition unit 410 acquires reference data, and defines the position and data acquisition range relative to the target data 421. Specifically, it is assumed that information such as data for one line on the left and right, one line above and below, or one line above, below, left, right, and so on, is set every other pixel.

[0041] The padding method 405 is information specifying a method for filling in missing data in the calculation data, and is set, for example, to fixed value padding, mirror padding, average value padding, etc. Fixed value padding is a method of padding with "0" or any other real value. Mirror padding is a method of arranging data in an image block in a line-symmetrical manner with the edge of the image block as a boundary line, inverting the data. Average value padding is a method of padding with the average value of surrounding pixel values. Note that the padding method is not limited to the above example, and other methods may be used. In the following description, the data added by the padding unit 440 is referred to as padding data.

[0042] The padding position 406 is information for specifying the position where the padding unit 440 pads data, and for example, the position and range relative to the target data 421 are determined. In this embodiment, data excluding reference data, which is a portion of the margin data required for the convolution operation of the target data 421, is padded. Therefore, the information of the padding position 406 is data obtained by excluding the reference data from the margin data for the target data 421. Specifically, for example, if the margin data is within a range of one line on the top, bottom, left, or right, and "one line on the left or right" is set as the reference data position 403, the information of the padding position 406 is "one line on the top, bottom, left, or right." Furthermore, if "one line on the top, bottom, left, or right" is set as the reference data position 403, the information of the padding position 406 is "one line on the left or right." Furthermore, if "one line on the top, bottom, left, or right, data every other pixel (reference data acquisition)" is set as the reference data position 403, the information of the padding position 406 is "one line on the top, bottom, left, or right, data of pixels where no reference data is placed." The above information indicating the division conditions, reference data acquisition conditions, and padding conditions is an example, and other information may be used. Also, an example is shown in which the CPU 101 reads out the division conditions, reference data acquisition conditions, and padding conditions stored in the ROM 102 and sets them in the inference unit 104, but this is not limited to this method, and data acquisition and padding may be performed by a program so as to satisfy the above conditions.

[0043] Image data (original image) stored in DRAM 103 in units of pages is read into inference unit 104 by acquisition unit 410. Preferably, acquisition unit 410 divides the image data stored in DRAM 103 into image blocks of a predetermined size using data division unit 411 and reads the divided image data. Information on the division positions in the original image data of the image blocks and information on the division sizes are set in advance in ROM 102 as division positions 401 and division sizes 402. Note that FIG. 4 is a diagram assuming that image data is divided when inference unit 104 reads the image data from DRAM 103, but this is not limited thereto, and data of image blocks divided in advance by preprocessing may be stored in DRAM 103. Furthermore, the image blocks may be stored in DRAM 103 with reference data added to them.

[0044] 5 is a diagram illustrating the data for inference and the data groups surrounding the divided image blocks. In the example shown in FIG. 5, one page of image data, which is the data for inference, is divided into four blocks vertically and horizontally, for a total of 16 blocks. In the following description, the image data before division is referred to as the original image 501, and one block of image data after division is referred to as image blocks 502 and 504.

[0045] The acquisition unit 410 sequentially reads the image blocks divided by the data division unit 411 in the order of processing as target data 421, and expands the data in the input data area 311 of the SRAM 310. At this time, the acquisition unit 410 also acquires information on the division position 401. This is because the acquisition position of the reference data may change depending on the division position. In this embodiment, the target data 421 for one divided image block is expanded in the input data area 311. The acquisition unit 410 also acquires reference data 422 from a group of pixels that exist around the acquired image block (target data 421) separately from the target data 421, and expands the reference data 422 in the input data area 311 of the SRAM 310.

[0046] The reference data 422 is margin data necessary to obtain a predetermined size of calculation result data in the convolution calculation executed by the convolution calculation unit 430. In this embodiment, a portion of the margin data is acquired as the reference data 422 (see FIG. 10 ). The acquisition unit 410 can identify the position of the original image from which to acquire the reference data, based on information on the reference data position 403 set in the ROM 102 and information on the division position 401 and division size 402 of the acquired target data 421. The acquisition unit 410 acquires the reference data 422 from a data group existing around the acquired target data 421, i.e., from the original image 501 stored in the DRAM 103, and expands it in the input data area 311. Data other than the reference data 422 among the margin data is filled by, for example, padding.

[0047] The size of the margin data required for the convolution calculation is determined depending on the structure of the CNN model constructed in the inference unit 104. For example, in the case of a 3x3 filter, a range of one line (one column or one row) on each of the top, bottom, left, and right sides around the target data 421 is used as margin data for the calculation. In the case of a 5x5 filter, a range of two lines (two columns or two rows) on each of the top, bottom, left, and right sides around the target data 421 is used as margin data for the calculation.

[0048] 5, the data group in the gray range surrounding the image block 502 is margin data 503 required for the convolution operation. The margin data 503 of the image block 502 is data that overlaps with adjacent image blocks. In this embodiment, in order to reduce the SRAM capacity and prevent a decrease in the reliability of the operation results, the acquisition unit 410 acquires only a portion of the margin data 503, for example, data on the left and right of the target data in the margin data 503, from the DRAM 103 as reference data 422. The padding unit 440 pads the remaining data in the margin data 503, excluding the reference data 422, with a fixed value such as "0", for example.

[0049] 5, the margin data of image block 504 located at the edge of original image 501 is insufficient if it is only data 505 overlapping with adjacent image blocks. In this case, too, the padding unit 440 can pad the data of missing portion 506.

[0050] The acquisition unit 410 acquires the filter coefficients 404 from the ROM 102. For example, if the processing layer is configured with n stages of filters, the acquisition unit 410 acquires filter coefficients for n stages and stores them in the filter coefficient area 312 of the SRAM 310.

[0051] The convolution operation unit 430 performs a convolution operation using margin data including the image block (target data 421) acquired by the acquisition unit 410 and surrounding reference data 422. In the following description, the target data 421 and reference data 422 acquired by the acquisition unit 410 from the DRAM 103 are referred to as input data 420. This input data 420 is expanded in the input data area 311 of the SRAM 310. The input data 420 including padding data becomes operation data to be processed in the convolution operation.

[0052] The convolution calculation unit 430 reads out the filter coefficients of the processing target stage from the filter coefficient area 312 of the SRAM 310 and sets them in the coefficient register 322 of the register 320. The convolution calculation unit 430 also sets data within a predetermined calculation range from the calculation data including the input data 420 and padding data in the calculation register 321, and performs a product-sum calculation with the filter coefficients set in the coefficient register 322. The convolution calculation unit 430 performs calculations on all pixels of the calculation data while sliding the calculation range, and writes the calculation results to the output data area 313 of the SRAM 310. Details of the convolution calculation will be described later.

[0053] If all the margin data required for the convolution operation by the convolution operation unit 430 has not been expanded in the input data area 311 (if the acquisition unit 410 has not acquired it), the padding unit 440 pads the missing data. For example, if information for acquiring one line of data on the left and right of the target data 421 is set as the reference data position 403, the acquisition unit 410 acquires only one line of data on the left and right of the target data 421 from the DRAM 103 as reference data 422. The padding unit 440 pads the data of the margin data excluding the reference data 422 based on information on the padding method 405 and the padding position 406 that are preset in the ROM 102.

[0054] 5, for target data 421 that does not contain pixels that serve as original image margin data, the acquisition unit 410 cannot acquire part of the margin data. In this case as well, the padding unit 440 pads the data that is insufficient for the calculation.

[0055] As described above, when the calculation data processed by the convolution calculation unit 430 is insufficient, the padding unit 440 adds padding data to the missing data location. In this embodiment, primarily for the purpose of reducing the capacity of the SRAM 310, some of the margin data that can be acquired from the DRAM 103 is not acquired and is instead padded. That is, actual data of the original image stored in the DRAM 103 is referenced for some of the margin data, and padding data is added to the remaining portion. By using some of the actual data of the margin data included in the calculation data in this manner, a decrease in the accuracy of the CNN calculation results is suppressed, thereby suppressing a decrease in reliability. Furthermore, by using padding data for some of the margin data, the amount of data stored in the SRAM 310 is reduced, thereby reducing the required SRAM capacity. Therefore, even in an embedded device with limited hardware resources, such as the printer 100 shown in this embodiment, it is possible to perform inference using a CNN model while suppressing a decrease in reliability.

[0056] The layers that partially acquire the margin data as described above may be all layers constituting the CNN, or at least one layer. Also, at least one may be included in each of the layers. For example, it is preferable to configure the final layer of the CNN input layer (encoding unit 201) to partially acquire the margin data.

[0057] Furthermore, when the target data 421 is divided data, it is preferable that the position of the reference data in the margin data (the position to which the padding unit 440 adds padding data) be changed for each position of the target data 421 (image block). Specifically, it is preferable that the padding positions be set randomly, for example, so that no periodicity appears in the padding positions between image blocks. This makes it possible to prevent the phenomenon in which portions with reduced reliability periodically appear and become noticeable in the output image obtained as a result of the convolution operation.

[0058] The output unit 450 stores the calculation results from the convolution calculation unit 430 in the output data area 313 of the SRAM 310. The output unit 450 determines a feature vector 460 from the results of all the filter processes and outputs it as the inference result of the inference unit 104.

[0059] (inference) Next, the flow of inference will be described. Fig. 6 is a flowchart showing the flow of inference in this embodiment. The processing shown in this flowchart is written in a program stored in ROM 102. The program is called by CPU 101, expanded in a work area of ​​RAM 103, and executed by CPU 101. When an original image to be processed is transferred to DRAM 103 via data transfer I / F 105, CPU 101 starts the processing shown in Fig. 6. In the following description, the symbol "S" means step.

[0060] In S601, the acquisition unit 410 acquires input data 420 from the DRAM 103 in accordance with division conditions and reference data acquisition conditions stored in advance in the ROM 102. The division conditions are a division position 401 and a division size 402. The reference data acquisition condition is a reference data position 403. The padding conditions are a padding method 405 and a padding position 406. As described above, the input data 420 includes target data 421 for one image block and reference data 422, which is part of margin data required for the convolution operation of the target data 421. The acquisition unit 410 also acquires filter coefficients stored in advance in the ROM 102. If there are multiple stages to be processed by the filter 300, multiple filter coefficients are acquired. The acquisition unit 410 expands the received one or more filter coefficients into the filter coefficient area 312 of the SRAM 310.

[0061] In S602 , the acquisition unit 410 expands the acquired input data 420 in the input data area 311 of the SRAM 310 of the filter 300 .

[0062] In S603, the acquisition unit 410 sets the filter coefficient of the target stage, among one or more filter coefficients expanded in the SRAM 310, in the coefficient register 322. First, the filter coefficient of the first-stage filter is set in the coefficient register 322.

[0063] In S604, the convolution operation unit 430 executes a convolution operation on the input data 420 expanded in the input data area 311 using the filter coefficients set in the coefficient register 322. Details of the convolution operation will be described later (FIG. 8).

[0064] In S605, the inference unit 104 determines whether processing for the next stage remains. If processing for the next stage remains, the process returns to S603, where the filter coefficients of the second-stage filter are obtained from the SRAM 310 and set in the coefficient register 322, and the convolution operation of the second-stage filter is performed on the input data 420 expanded in the input data area 311. In this way, all convolution operations from the first-stage filter to the n-th-stage filter are performed, and if it is determined in S605 that the next stage does not remain, the process of this flowchart ends.

[0065] (Example of margin data acquisition; comparison example) Next, a method for acquiring margin data will be described. First, a method for acquiring margin data that is a comparative example to this embodiment will be described. Fig. 7 is a diagram showing a data acquisition method as a comparative example. Fig. 7 shows an example in which all margin data is acquired from the original image 501 in the DRAM 103.

[0066] 7, an area 702 indicated by a solid line in an input data area 701 in the SRAM 310 is an area in which target data 421 (image blocks after division) acquired by the acquisition unit 410 from the original image stored in the DRAM 103 is expanded. A range 703 indicated by a dashed line around the area 702 indicated by a solid line indicates the range of margin data required for the convolution operation. In the example shown in FIG. 7, the entire range of one line around the outer periphery of the target data 421 is acquired from the original image in the DRAM 103 as margin data (data indicated in gray in the figure) and stored in the SRAM 310.

[0067] In this way, when the margin data for the target data 421 is entirely composed of actual data, the amount of SRAM 310 used includes not only the target data 421 but all the margin data. This requires a large amount of capacity. Instead, actual data is used in the calculation, which increases the accuracy of the feature vector obtained as a result of inference. In the example of Figure 7, the range of the margin data is one line each on the top, bottom, left, and right, because the filter size used for the convolution calculation is 3x3. The larger the filter size, the larger the required margin data range 703. When the filter size is 5x5, the margin data range 703 is two lines each on the top, bottom, left, and right.

[0068] (Convolution operation processing; comparison example) Details of the convolution calculation process will be described with reference to FIGS. 8 and 9. FIG. 8 is a flowchart illustrating the flow of the convolution calculation process in the case of FIG. 7. The process shown in this flowchart is executed in S604 of FIG. 6. Before starting the flowchart shown in FIG. 8, it is assumed that the input data 420 is expanded in the SRAM 310 as shown in FIG. 7 and the filter coefficients are set in the coefficient register 322 by the processes of S601 to S603 of FIG. 6. FIG. 9 is a diagram illustrating an example of the convolution calculation when all the margin data is obtained from the original image as shown in FIG. 7.

[0069] In S801, the convolution calculation unit 430 acquires data in the calculation range one range at a time from the input data expanded in the SRAM 310 and sets the data in the calculation register 321. The data in the calculation range is data equivalent to the number of elements of the filter coefficient (3 × 3). For example, as shown in FIG. 9A, to obtain the value of pixel D1 in the output image (output data area 313), data in a calculation range 900, shown in gray in the figure, is required from the data expanded in the input data area 701 of the SRAM 310. The calculation range 900 in FIG. 9A includes nine pixels: o1, o2, o3, o5, d1, d2, o6, d4, and d5. Of the data in the calculation range 900, the data d1, d2, d4, and d5 are data of the image block (target data 421) read from the DRAM 103, and the data o1 to o6 are actual data read from the DRAM 103 as margin data (reference data 422).

[0070] In S802, the convolution calculation unit 430 performs a product-sum calculation on the acquired data in the calculation range 900 and the filter coefficients. The convolution calculation unit 430 multiplies the values ​​of each pixel o1, o2, o3, o5, d1, d2, o6, d4, and d5 in the calculation range 900 by the values ​​of corresponding elements c1 to c9 of the filter, and writes the multiplication results to each pixel r1 to r9 in the calculation register 321. Thereafter, the convolution calculation unit 430 collects and adds up the multiplication results r1 to r9 for the number of filter elements held in the calculation register 321, and aggregates the results into pixel r5.

[0071] The pixel r5 in the calculation range 900 corresponds to the pixel D1 in the output data area 313 (output image). Therefore, the value of the pixel D1 is D1=o1×c1+o2×c2+o3×c3 +o5×c4+d1×c5+d2×c6 +o6×c7+d4×c8+d5×c9 is determined by

[0072] In S803, the convolution operation unit 430 stores the value resulting from the sum-of-products operation at the corresponding pixel position in the output data area 313 of the SRAM 310. In the example of FIG. 9(a), the value resulting from the sum-of-products operation is held in pixel D1 of the output data area 313.

[0073] In S804, the convolution calculation unit 430 determines whether all of the calculation data, including the target data 421 and its margin data, held in the input data area 311 of the SRAM 310 has been processed. If there is unprocessed calculation data, the process returns to S801, and the convolution calculation unit 430 performs a product-sum calculation on the next calculation range. Specifically, as shown in FIG. 9(b), the convolution calculation unit 430 slides the calculation range 900 horizontally by one line. That is, of the calculation data already acquired from the SRAM 310, the values ​​of o1, o5, and o6 are discarded, and instead the values ​​of o4, d3, and d6 are loaded into the calculation register 321, and the above-mentioned product-sum calculation is performed.

[0074] The value of pixel D2 in the output data area 313 (output image) is D2=o2×c1+o3×c2+o4×c3 +d1×c4+d2×c5+d3×c6 +d4×c7+d5×c8+d6×c9 The determined value is written to pixel D2 in the output data area 313.

[0075] In this way, the convolution operation unit 430 repeatedly executes product-sum operations with the filter coefficients while sliding the operation range in order, and ends the processing of this flowchart when it has processed all of the operation data including the target data 421 and its margin data stored in the SRAM 310. When the processing of Figure 8 ends, the operation result data after the filter processing of the target stage is stored in the output data area 313 of the SRAM 310.

[0076] In the examples of FIGS. 7 to 9, the convolution calculation is performed using 3×3 filter coefficients, so the calculation range 900 is a 3×3 range. The result of the convolution calculation is collected at the pixel at the center of the calculation range as shown in the following equation (1). In equation (1), j is a number indicating the pixel position of the output image D, c i is the value of each element of the filter coefficient, and di is the value of the calculation data included in the calculation range. i is an identifier indicating the filter coefficient or the pixel position within the calculation range. The calculation result Dj of equation (1) is the value of the pixel located at the center of the calculation range 900.

[0077]

number

[0078] As mentioned above, convolutional operations require margin data in a range that depends on the filter size. Furthermore, a larger range of margin data is required depending on the number of filter stages. In other words, the range of margin data required to ultimately obtain an output image of a certain size is determined by the structure of the CNN, including at least the filter size and the number of filter stages (the number of convolutional layers).

[0079] (Margin Data Acquisition Method and Convolution Operation Processing According to the Present Embodiment) Next, a method for acquiring margin data and convolution calculation processing in this embodiment will be described with reference to Figs. 10 to 12. Fig. 10 is a diagram showing a method for acquiring peripheral data according to this embodiment. Fig. 11 is a flowchart explaining the flow of convolution calculation processing in the case of Fig. 10. Fig. 12 is a diagram explaining the calculation procedure of convolution calculation in the case of Fig. 10.

[0080] 10 , an area 1002 indicated by a solid line in an input data area 1001 in the SRAM 310 is an area in which target data 421 (divided image blocks) acquired by the acquisition unit 410 from an original image stored in the DRAM 103 is expanded. Areas 1003 and 1004 indicated by dashed lines on the left and right of the area 1002 (shown in gray in the figure) represent areas in which reference data 422 is stored. That is, in the example shown in FIG. 10 , of the margin data required for the convolution operation, only the data in the areas to the left and right of the target data 421 is fetched from the DRAM 103 as reference data 422 and expanded in the SRAM 310. Data in the areas above and below the target data 421 is not fetched, or is deleted after fetching. In this way, the inference unit 104 of this embodiment partially acquires the margin data required for the convolution operation from the DRAM 103 and expands it in the SRAM 310. In the case of FIG. 10, the amount of usage of the SRAM 310 is reduced by one line each above and below compared to the case of FIG.

[0081] In the case of a 3x3 single-stage filter, the margin data required for the convolution operation is one line each on the top, bottom, left, and right. However, according to the data acquisition method of this embodiment shown in FIG. 10, the margin data for one line each on the top and bottom is not expanded in the SRAM 310. In this case, the missing margin data is interpolated by padding. As the filter size increases, the width (range) of the required margin data also increases, and when a multi-stage configuration is used, even wider margin data is required. Therefore, as the filter size and number of layers increase, a greater memory reduction effect is achieved.

[0082] The convolution operation using the data acquired by the method shown in FIG. 10 will be described in detail with reference to FIGS. 11 and 12. The process shown in this flowchart is executed in S604 of FIG. 6. Before starting the flowchart shown in FIG. 11, it is assumed that the input data shown in FIG. 10 is held in the SRAM 310 and the filter coefficients are set in the coefficient register 322 by the processes of S601 to S603 of FIG. 6. The input data includes the divided image block (target data 421) and the reference data 422 of the left and right areas thereof. Note that in FIG. 10, identifiers d1 to d9 and o5 to o7 for identifying pixels are shown for only some of the pixels, but this is for the purpose of explaining the operation range in FIG. 12. In reality, data is captured for all pixels in the input data area 1001.

[0083] In S1101, the convolution calculation unit 430 acquires data in the calculation range one range at a time from the input data expanded in the SRAM 310 and sets the data in the calculation register 321. The data in the calculation range is data equivalent to the number of elements of the filter coefficient (3 × 3). For example, as shown in FIG. 12(a), in order to obtain the value of pixel D1 in the output image (output data area 313), data in the calculation range 1200 shown in gray in FIG. 12 is required from the calculation data expanded in the input data area 1001 of the SRAM 310. However, when data is acquired using the method shown in FIG. 10, data o5, d1, d2, o6, d4, and d5 in the calculation range 1200 are stored in the SRAM 310, but data for the three pixels above o5, d1, and d2 is not stored in the SRAM 310. In the calculation range 1200, data d1, d2, d4, and d5 are image block data read from the DRAM 103, and data o5 and o6 are actual data read from the DRAM 103 as reference data 422.

[0084] In S1102, the padding unit 440 adds padding data to the calculation register 321 for the three pixels above o5, d1, and d2 where data is missing. In this way, the padding unit 440 writes the padding data directly to the corresponding pixel position in the calculation register 321 before the multiply-and-accumulate operation. Alternatively, the padding unit 440 may write the padding data directly to the corresponding pixel position in the calculation register 321 after the multiply-and-accumulate operation. The padding data is a value set by the padding method 405. For example, it may be an arbitrary fixed value such as "0", mirror image data of the data in the image block, the average value of the actual data around the corresponding pixel, etc.

[0085] If the padding data is written directly to the calculation register 321 without being expanded in the SRAM 310, the padding process can be completed using only the processing within the register 320. In the example shown in Figure 12(a), data for three pixels located above o5, d1, and d2 in the calculation range 1200 is missing, so the padding unit 440 writes padding data Pd directly to the corresponding positions r1, r2, and r3 in the calculation register 321.

[0086] In S1103, the convolution calculation unit 430 performs a product-sum calculation on the data of the calculation range 1200 acquired in S1101 and the filter coefficients. That is, the convolution calculation unit 430 multiplies the values ​​of each pixel o5, d1, d2, o6, d4, and d5 in the calculation range 1200 by the values ​​of the corresponding elements c4 to c9 of the filter, and writes the results to r4 to r9 of the calculation register 321. Thereafter, the convolution calculation unit 430 collects and adds the multiplication results and padding data for the number of filter elements held in registers r1 to r9, and determines the result as the value of pixel D1.

[0087] The value of D1 in the output image is D1=Pd+Pd+Pd +o5×c4+d1×c5+d2×c6 +o6×c7+d4×c8+d5×c9 The value of D1 is written to the output data area 313. Note that this calculation example shows a case where the padding data Pd is written to r1, r2, and r3 of the calculation register 321 as values ​​after the product-sum operation. When padding data is added before the product-sum operation, the value of D1 of the output image is calculated as follows:

[0088] D1=Pd×c1+Pd×c2+Pd×c3 +o5×c4+d1×c5+d2×c6 +o6×c7+d4×c8+d5×c9

[0089] In S1104, the convolution operation unit 430 stores the result of the product-sum operation at the corresponding pixel position in the output data area 313 of the SRAM 310. In the example of FIG. 12(a), the value of the result of the product-sum operation is stored in pixel D1 of the output data area 313.

[0090] In S1105, the convolution calculation unit 430 determines whether all of the calculation data stored in the input data area 311 of the SRAM 310 has been processed. If there is unprocessed data in the calculation data, the process returns to S1101, and the convolution calculation unit 430 performs a product-sum calculation on the next calculation range. Specifically, as shown in FIG. 12(b), the calculation range 1200 is moved horizontally by one line. That is, of the calculation data already obtained from the SRAM 310, the values ​​of o5 and o6 are discarded, and instead the values ​​of d3 and d6 are loaded into the calculation register 321, and the above-mentioned product-sum calculation is performed.

[0091] Since data for three pixels above d1, d2, and d3 in the calculation range 1200 is insufficient, padding data Pd is added to the corresponding positions r1, r2, and r3 in the calculation register 321.

[0092] The value of pixel D2 in the output image is D2=Pd+Pd+Pd +d1×c4+d2×c5+d3×c6 +d4×c7+d5×c8+d6×c9 The value of pixel D2 is written to the output data area 313. Note that this calculation example shows a case where padding data Pd is written to r1, r2, and r3 of the calculation register 321 as values ​​after the product-sum operation. When padding data is added before the product-sum operation, the value of D1 of the output image is calculated as follows:

[0093] D1=Pd×c1+Pd×c2+Pd×c3 +d1×c4+d2×c5+d3×c6 +d4×c7+d5×c8+d6×c9

[0094] In this way, the convolution calculation unit 430 repeats the convolution calculation while sliding the calculation range and padding the missing data. When all the calculation data, including the target data 421 and margin data stored in the SRAM 310, has been processed, the processing of this flowchart ends. When the processing of Figure 11 is completed, the output image after the filter processing of the target stage is stored in the output data area 313 of the SRAM 310.

[0095] As shown in this embodiment, when the margin data reduction portion is the top and bottom lines of the image block, it is more preferable to slide the calculation range in the left-right direction. Similarly, when the margin data reduction portion is the left and right lines of the image block, it is more preferable to slide the calculation range in the up-down direction. That is, it is preferable to obtain reference data from a data group that exists in the same direction as the sliding direction of the calculation range of the convolution calculation for the target data. This is because the pixel positions where data is insufficient are the same before and after moving the calculation range, thereby reducing the number of times padding is performed. However, this does not prevent reference data from being obtained from a data group that exists in a direction perpendicular to the sliding direction of the calculation range of the convolution calculation for the target data. That is, when the margin data reduction portion is the top and bottom lines of the image block, it is also possible to slide the calculation range in the up-down direction. Furthermore, when the margin data reduction portion is the left and right lines of the image block, it is also possible to slide the calculation range in the left-right direction.

[0096] Furthermore, when the data in DRAM 103 is arranged horizontally, it is preferable to set the margin data reduction portion to the lines above and below the image block. This is because the frequency of access to DRAM 103 can be reduced when the acquisition unit 410 reads data from DRAM 103. However, the data in DRAM 103 may be arranged vertically. The frequency of access to DRAM 103 when the acquisition unit 410 reads data from DRAM 103 will be described below.

[0097] FIG. 13 is a diagram illustrating a method of accessing the DRAM when data is taken from the DRAM 103 to the SRAM 310. Image data stored in the DRAM 103 is shown. The acquisition unit 410 acquires 10 horizontal by 11 vertical image blocks, indicated by a bold frame, from the DRAM 103 and transfers them to the SRAM 310. The images in the DRAM 103 are arranged from left to right. When transferring data from the DRAM 103 to the SRAM 310, the acquisition unit 410 transfers data one by one, as indicated by the arrows in the frame. In the example of FIG. 13(a), the DRAM 103 is accessed 11 times to transfer data.

[0098] On the other hand, when the top and bottom margin data are reduced as in this embodiment, a 10 horizontal by 9 vertical image block is transferred from the DRAM 103 as shown in FIG. 13(b). Therefore, as shown in the bold frame, the number of accesses to the DRAM 103 is reduced by two, top and bottom, to nine accesses. This reduces the number of accesses compared to the case of FIG. 13(a). In this way, the frequency of accesses to the DRAM 103 can be reduced by the relationship between the data arrangement on the DRAM 103 and the position of the padding (the position where the margin data is reduced).

[0099] Due to its characteristics, the DRAM 103 is designed to read a lump of data for each data row. In the example of Figure 13, data for 10 consecutive pixels is read with each access. Therefore, if the left and right margin data are to be reduced, reading 8 consecutive pixels would be sufficient, but the system is designed so that 10 pixels are read from the DRAM 103. Therefore, the number of data accesses to the DRAM 103 is the same as when all the margin data is obtained, at 11 times (the 11 arrows in Figure 13(a)). In this case, the excess data that was read can be deleted after it has been obtained.

[0100] As described above, the inference unit 104 of the printer 100 of this embodiment retrieves a portion of the margin data required for the convolution calculation from the original image as reference data, and interpolates the remaining margin data excluding the reference data by padding. This reduces the memory capacity required for the filter process compared to when all margin data is retrieved. Furthermore, compared to when all margin data is padded, the actual data can be used for the calculation, which reduces the decrease in accuracy of the feature vector obtained as a result of the filter process. Therefore, it is possible to reduce the memory capacity required for the CNN convolution calculation while suppressing the decrease in reliability of the calculation results.

[0101] In the above example, the upper and lower regions of the margin data of the target data are not acquired, and the left and right regions are acquired as reference data. However, the present disclosure is not limited to this example. Margin data may be reduced for only the upper or lower line, or only the left or right line. Furthermore, the ratio of reference data to the total amount of margin data required for the convolution operation is arbitrary. Below, a modified example of the first embodiment will be described.

[0102] <Variation 1> Fig. 14 is a diagram showing a data acquisition method according to Modification 1 of this embodiment. In the example shown in Fig. 14, the margin data taken from DRAM 103 to SRAM 310 is reduced compared to the case of Fig. 7. What differs from the first embodiment shown in Fig. 10 is that, of the margin data required for the convolution operation, reference data is acquired for one line each above and below, and margin data on the left and right is not acquired. Areas 1403 and 1404 shown in gray in Fig. 14 indicate areas where the reference data taken from DRAM 103 is held.

[0103] 14, an area 1402 indicated by a solid line in an input data area 1401 in the SRAM 310 is an area in which an image block (target data 421) acquired from the DRAM 103 is expanded. Areas 1403 and 1404 indicated by dashed lines above and below the area 1402 indicated by a solid line indicate areas in which reference data 422 acquired from the DRAM 103 is stored. In the example shown in FIG. 14, of the margin data required for the convolution operation, only data in the areas above and below the image block is acquired from the DRAM 103 and expanded in the SRAM 310. Data in the areas on the left and right is not acquired, or is deleted after being acquired. In the example shown in FIG. 14, the margin data required for the convolution operation is also partially acquired from the DRAM 103 and expanded in the SRAM 310, thereby reducing the amount of SRAM 310 used compared to FIG. 7.

[0104] 14, the padding unit 440 also pads any missing data from the margin data required for the convolution operation. The padding unit 440 may write the padding data directly into the corresponding data area of ​​the operation register 321. As described above, the padding data may be any fixed value such as "0," mirror image data of the data in the image block, or the average value of the data surrounding the corresponding pixel. The processing procedure for the convolution operation is the same as in the above-described embodiment.

[0105] 14, when padding the left and right data, it is preferable to slide the calculation range vertically. Also, from the viewpoint of improving the efficiency of the frequency of access to the DRAM 103, it is preferable that the data on the DRAM 103 be arranged vertically.

[0106] FIG. 15 is a diagram illustrating the calculation procedure for a convolution operation using the calculation data loaded into SRAM 102 by the method shown in FIG. 14. This diagram illustrates an example in which the sliding direction of calculation range 1500 is vertical. As shown in FIG. 15(a), to obtain the value of pixel D1 in output data area 313 (output image), the data in calculation range 1500, shown in gray in the figure, is required from the input data expanded in input data area 1401 of SRAM 310. Data o2, o3, d1, d2, d4, and d5 in calculation range 1500 are stored in SRAM 310, but data for the three pixels to the left of o2, d1, and d4 are not stored in SRAM 310. Note that data d1, d2, d4, and d5 in calculation range 1500 are image block data read from DRAM 103, and data o2 and o3 are data read from DRAM 103 as reference data.

[0107] The padding unit 440 adds padding data to pixel positions where data is missing. The padding unit 440 may write the padding data directly to the corresponding data area of ​​the calculation register 321. As described above, the padding data may be, for example, an arbitrary fixed value such as "0," mirror image data of the data in the image block, or the average value of the data surrounding the corresponding pixel. In the example shown in FIG. 15(a), data for three pixels located to the left of o2, d1, and d4 in the calculation range 1500 is missing, so the padding unit 440 directly writes padding data Pd to the corresponding positions r1, r4, and r7 of the calculation register 321.

[0108] The convolution calculation unit 430 performs a product-sum calculation on the data in the calculation range 1500 and the filter coefficients. That is, the convolution calculation unit 430 multiplies the values ​​of each pixel o2, o3, d1, d2, d4, and d5 in the calculation range 1500 by the values ​​of the corresponding filter elements c2, c3, c5, c6, c8, and c9, and writes the results to r2, r3, r5, r6, r8, and r9 in the calculation register 321. Thereafter, the convolution calculation unit 430 collects and adds the multiplication results and padding data for the number of filter elements held in registers r1 to r9, and determines the result as the value of pixel D1.

[0109] The value of D1 in the output image is D1=Pd+o2×c2+o3×c3 +Pd+d1×c5+d2×c6 +Pd+d4×c8+d5×c9 The value of D1 is written to the output data area 313. Note that this calculation example shows a case where the padding data Pd is written to r1, r4, and r7 of the calculation register 321 as a value after the product-sum operation. When padding data is added before the product-sum operation, the value of D1 of the output image is calculated as follows:

[0110] D1=Pd×c1+o2×c2+o3×c3 +Pd×c4+d1×c5+d2×c6 +Pd×c7+d4×c8+d5×c9

[0111] Next, as shown in FIG. 15(b), the convolution calculation unit 430 slides the calculation range 1500 down one line (vertically). That is, of the calculation data already acquired from the SRAM 310, the values ​​of o2 and o3 are discarded, and instead the values ​​of d7 and d8 are acquired. Because there is a shortage of data for three pixels located to the left of d1, d4, and d7 in the calculation range 1500, padding data Pd is added to the corresponding positions r1, r4, and r7 in the register. Note that this padding data has already been written. In this case, there is no need to rewrite the padding data Pd.

[0112] The convolution operation unit 430 performs a product-sum operation on the acquired data in the operation range 1500 and the filter coefficients to determine the value of pixel D4.

[0113] The value of D4 in the output image is D4=Pd+d1×c2+d2×c3 +Pd+d4×c5+d5×c6 +Pd+d7×c8+d8×c9 The value of D1 is written to the output data area 313. Note that this calculation example shows a case where the padding data Pd is written to r1, r4, and r7 of the calculation register 321 as a value after the product-sum operation. When padding data is added before the product-sum operation, the value of D1 of the output image is calculated as follows:

[0114] D4=Pd×c1+d1×c2+d2×c3 +Pd×c4+d4×c5+d5×c6 +Pd×c7+d7×c8+d8×c9

[0115] As shown in Modification 1, when the margin data reduction areas are the left and right areas of the image block, it is preferable to slide the calculation range vertically. This is because the pixel positions where data is insufficient are the same before and after moving the calculation range, so the number of times padding data is added can be reduced. Furthermore, when the data arrangement direction in DRAM 103 is vertical, if the margin data reduction areas are the left and right areas of the image block as in Modification 1, the frequency of access to DRAM 103 can be reduced, which is efficient. Note that in the above example, the calculation range may be slid horizontally. Furthermore, the data arrangement direction in DRAM 103 may be horizontal. Furthermore, only the upper line or only the lower line may be acquired as reference data.

[0116] As described above, according to the first modification, the upper and lower data, which are part of the margin data required for the convolution operation, are taken from the original image as reference data, and the remaining data of the margin data excluding the reference data is interpolated by padding. This reduces the memory capacity required for the filter operation compared to when all the margin data is taken in. Furthermore, compared to when all the margin data is padded, the actual data can be used for the operation, making it possible to suppress a decrease in the accuracy of the feature vector obtained as a result of the filter operation. Therefore, it is possible to reduce the memory capacity required for the convolution operation of the CNN while suppressing a decrease in the reliability of the operation results.

[0117] <Variation 2> FIG. 16 is a diagram showing a data acquisition method according to Modification 2 of this embodiment. In the example shown in FIG. 16, the margin data taken from DRAM 103 to SRAM 310 is reduced compared to the case of FIG. 7. What differs from the first embodiment (FIG. 10) is that reference data is acquired discretely in the upper, lower, left, and right regions, which are margin data required for convolution calculations. A plurality of pixels 1603 shown in gray in FIG. 16 indicate the regions where the reference data taken from DRAM 103 is held.

[0118] 16, an area 1602 indicated by a solid line in an input data area 1601 in the SRAM 310 is an area in which an image block acquired from the DRAM 103 is loaded. Furthermore, a plurality of areas 1603 (indicated in gray) indicated by dashed lines and present above, below, left and right of the area 1602 indicate areas in which reference data is held. That is, in the example shown in FIG. 16, data on the image block and pixels surrounding the image block are fetched from the DRAM 103 for every other pixel and loaded in the SRAM 310. In the case of FIG. 16 as well, margin data required for the convolution operation is partially acquired from the DRAM 103 and loaded in the SRAM 310, so the amount of SRAM 310 used is reduced compared to FIG. 7.

[0119] 16, the padding unit 440 pads any missing data from the margin data required for the convolution operation. The padding unit 440 may write the padding data directly into the corresponding data area of ​​the operation register 321. As described above, the padding data may be any fixed value such as "0," mirror image data of the data in the image block, or the average value of the data surrounding the corresponding pixel. The processing procedure for the convolution operation is the same as in the above-described embodiment and modified example.

[0120] As described above, according to Modification 2, some of the margin data required for the convolution operation is taken from the original image as reference data, and the remaining margin data excluding the reference data is interpolated by padding. This reduces the memory capacity required for the filter operation compared to when all margin data is taken in. Furthermore, compared to when all margin data is padded, this allows actual data to be used for the operation, making it possible to suppress a decrease in the accuracy of the feature vector obtained as a result of the filter operation.

[0121] Furthermore, according to Modification 2, it is expected that the accuracy of the feature vector obtained as a result of the filtering process will be further improved compared to the cases of Fig. 10 and Fig. 14. This is because, even though the padding data is thinned out, the true value of the original image is used in the convolution operation without locally concentrating the padding data.

[0122] 16 shows an example in which reference data is acquired every other pixel, but the present invention is not limited to this example and may be arranged by skipping any number of pixels, such as every third pixel or every third pixel. Furthermore, as long as the arrangement is discrete, the reference data may be randomly arranged at any position. Furthermore, as shown in FIGS. 10 and 14, when acquiring reference data for at least one line of the margin data (top, bottom, left, or right), the reference data may be acquired discretely on that line.

[0123] <Variation 3> In the first embodiment and modifications 1 and 2 described above, the margin data width is one line on each side, but the present disclosure can also be applied to cases where the margin data spans multiple lines. For example, when using a 5x5 filter, the margin data width requires two lines on each side.

[0124] Fig. 17 is a diagram showing an example of margin data spanning multiple lines. In Fig. 17, an area 1702 indicated by a solid line in an input data area 1701 in the SRAM 310 is an area in which an image block acquired from the DRAM 103 is developed. Areas 1703 indicated by dashed lines above, below, left, and right of the solid line indicate the range of margin data required for convolution calculation. In the case of a 5x5 filter, this indicates that margin data covering a range of two lines (two rows and two columns) surrounding the image block is required. Furthermore, an area 1704 indicated by gray in Fig. 17 shows a state in which reference data o1 to o7 acquired from the DRAM 103 is held.

[0125] In the example of FIG. 17, data o1 to o7 acquired from the DRAM 103 are concentrated and stored in a portion of one of two lines of margin data (area 1703). The remaining margin data is padded. In this case, reference data is partially used, so the amount of SRAM used can be reduced, as in the above-described embodiments and modifications. However, the reference data is concentrated and stored locally, and is separated from the padding area. If reference data is acquired in this manner, some information from the actual data will be missing.

[0126] FIG. 18 is a diagram showing a data acquisition method according to Modification 3. The method shown in FIG. 18 is a more preferable data acquisition method than that shown in FIG. 17. The example of FIG. 18 also shows a case where the range of margin data required for convolution calculation spans multiple lines. Multiple areas 1803 shown in gray in FIG. 18 are areas where reference data acquired from DRAM 103 is held. The reference data o1 to o7 acquired from DRAM 103 are discretely arranged so as not to be concentrated locally. Specifically, the reference data o1 to o7 are arranged in a staggered pattern in SRAM 310.

[0127] In this way, when the margin data required for the convolution operation spans multiple lines, it is preferable to distribute the margin data obtained from the DRAM 103 so that it is not concentrated on one line. This makes it possible to prevent loss of information from the original image data in a certain line. As a result, it is possible to suppress a decrease in the accuracy of the processing results in the convolution operation using padding.

[0128] 18 shows an example of a staggered arrangement in which reference data is arranged every other pixel, but the present invention is not limited to this example and may be arranged by skipping any number of pixels, such as every third pixel or every third pixel. Also, any discrete arrangement is acceptable, and reference data may be arranged randomly at any position. Furthermore, when margin data required for convolution calculation spans multiple lines, this third modification does not prevent actual data from being captured only on the upper and lower lines as shown in FIG. 10 or only on the left and right lines as shown in FIG. 14.

[0129] In any case, in this embodiment, the reference data that the inference unit 104 acquires from the DRAM 103 and loads in the SRAM 310 is set to be less than all the margin data required for the convolution operation, but more than when all the margin data is padded. This makes it possible to reduce the capacity of the SRAM 310 required for the filter processing of the inference unit 104, while suppressing a decrease in the accuracy of the processing result.

[0130] Note that, when filter processing is performed across multiple layers due to the structure of the CNN used by the inference unit 104, it is not necessary to perform the margin data reduction of this embodiment in all layers; rather, one or more layers may be selected and the margin data reduction of this embodiment may be performed. In this case, the margin data may be reduced as in this embodiment for filters used in layers that have a high data reduction effect. An example of a layer that has a high data reduction effect is the last layer of the encoding unit 201. This layer is preferable because it has a low resolution and therefore requires a small amount of padding, and the resolution is increased by upsampling in subsequent layers.

[0131] Furthermore, the padding position (the position where margin data is reduced) may be changed depending on the position in the original image of the target data 421 (image block) fetched from DRAM 103. For example, when an image block at the top of the original image is obtained from DRAM 103, there are no pixels outside the top edge of the original image data in DRAM 103, so the padding position is set to the top line. Alternatively, it may be set to the top and bottom lines as shown in FIG. 10. Furthermore, when an image block at the left edge of the original image is obtained from DRAM 103, there are no pixels outside the left edge of the original image data in DRAM 103, so the padding position is set to only the left line. Alternatively, it may be set to the left and right lines as shown in FIG. 14.

[0132] The padding position may also be changed for each image block loaded from the DRAM 103. In this case, it is preferable that the padding position be changed randomly so as not to have any particular periodicity. For example, if padding is always performed at the same position for all image blocks, periodic accuracy degradation will occur. By dispersing the padding positions as much as possible among image blocks, periodic accuracy degradation can be prevented.

[0133] A program for determining which image block should be padded at which position may be stored in advance in ROM 102 of printer 100, and the CPU 101 may execute the program to determine the padding position for each image block. Alternatively, the CPU 101 may determine which image block should be padded at which position based on division positions and padding position information that is preset in ROM 102.

[0134] The configuration of the inference unit 104 in the printer 100 of the first embodiment and an example of inference have been described above. In this embodiment, part of the margin data required for the convolution operation is obtained as reference data, and the remaining part is padded. This makes it possible to perform inference using a CNN model in an embedded device with limited computing resources while suppressing a decrease in the accuracy of inference.

[0135] <Second embodiment> In the second embodiment, in the printer 100 shown in the first embodiment, the learning conditions for obtaining the filter coefficients used in inference are made common to the inference conditions in the inference unit 104. This improves the accuracy of the inference results.

[0136] The accuracy of the attribute probability ultimately obtained as a result of inference depends on the similarity between the image characteristics used in inference and the image characteristics assumed in learning. Learning is a process in which a CNN uses a large amount of learning data to optimize filter coefficients and determine filter coefficients suitable for extracting features. Therefore, in the second embodiment, as described in the first embodiment, inference conditions that take into account the hardware resource constraints of the printer 100 are also reflected in learning. The learned filter coefficients determined by learning are then set in the printer 100.

[0137] In particular, in this embodiment, the conditions for dividing the calculation data, the conditions for obtaining reference data, and the padding conditions used in inference and learning are common to both inference and learning. In addition, the conditions for determining the structure of the CNN model, such as the number of processing layers, the filter size, the number of reductions and expansions (hereinafter referred to as CNN conditions), are also common to both inference and learning.

[0138] (System Configuration) Next, the system configuration of the information processing system 1900 according to the second embodiment will be described. FIG. 19 is a diagram showing the relationship between the printer 100 as an inference device and a learning device 1901.

[0139] As shown in Figure 19(a), the second embodiment describes an example of an information processing system 1900 in which a learning device 1901 and a printer 100 are configured as separate devices. Note that the configuration of the information processing system 1900 is not limited to this example, and for example, the printer 100 may be configured to include both an inference unit and a learning unit. Also, as in the first embodiment, the printer 100 is an example of an inference device as a product, and is not limited to the printer 100 and may be other embedded devices.

[0140] In the following description, the printer 100 is the printer 100 shown in the first embodiment, and the hardware configuration and functional configuration are the same as those in the first embodiment, and the same parts are given the same reference numerals.

[0141] The learning device 1901 is a device that generates filter coefficients used in the inference performed by the inference unit 104 of the printer 100, and is configured, for example, by an information processing device such as a personal computer (PC). The hardware resources of the learning device 1901 are assumed to be higher performance in terms of calculation speed and storage capacity than those of the printer 100.

[0142] In the second embodiment, the learning conditions used by the learning device 1901 to generate filter coefficients reflect the inference conditions of the printer 100. Specifically, the conditions for acquiring learning data in the learning device 1901 reflect the conditions for acquiring calculation data in the inference unit 104 of the printer 100. The structure of the CNN model in the learning device 1901 is the same as the structure of the CNN model in the inference unit 104 of the printer 100. The arrows in FIG. 19(a) do not necessarily mean that the learning device 1901 and the printer 100 are communicatively connected. They indicate that the inference conditions set in the printer 100 are reflected in the learning device 1901, and that the learned filter coefficients generated by the learning device 1901 are reflected in the printer 100 and used for inference. The conditions and filter coefficients may be reflected manually by the user.

[0143] FIG. 19(b) is a diagram showing an example of conditions 1920 reflected in the learning device 1901. The conditions 1920 reflected in the learning device 1901 include division conditions for data used in learning (learning data), reference data acquisition conditions, padding conditions, and CNN conditions. The division conditions are division position 1921 and division size 1922. The reference data acquisition condition is reference data position 1923. The padding conditions are padding method 1924 and padding position 1925. These are the same as the division position 401, division size 402, reference data position 403, padding method 405, and padding position 406 set in the ROM 102 of the printer 100 in FIG. 4. The CNN conditions 1926 are conditions related to the structure of the CNN model, such as the number of processing layers, filter size, number of reductions, and enlargements. The CNN conditions 1926 are also reflected in the learning device 1901 in the same content as the conditions of the CNN model of the inference unit 104 of the printer 100 shown in FIG.

[0144] When the learning device 1901 performs learning, the learning data is divided and acquired based on the division position 1921 and the division size 1922, and reference data specified based on the reference data position 1923 is acquired and stored in the memory (RAM 2130) of the learning unit 2000 of the learning device 1901. In addition, padding data is added to any missing data among the margin data required for the convolution operation based on the padding method 405 and the padding position 406.

[0145] (Learning device configuration) 20 is a diagram showing an example of the hardware configuration of a learning device 1901. The learning device 1901 includes, for example, a CPU 2001, a learning unit 2000, a ROM 2002, a RAM 2003, a communication unit 2004, an input unit 2005, a display unit 2006, a storage unit 2007, and a data transfer I / F. Each of these units is connected to the CPU 2001 via a data bus 2009. Note that the configuration of the learning device 1901 is not limited to the example shown in FIG. 20, and various other configurations may be used as appropriate.

[0146] The CPU 2001 executes various processes using the RAM 2003 as a work area in accordance with programs stored in the ROM 2002 or the storage unit 2007. The RAM 2003 is a volatile storage area and is used as a work memory, etc. The ROM 2002 is a nonvolatile storage area and stores the programs according to this embodiment, an OS (Operating System), etc. The storage unit 2007 is a nonvolatile storage device such as an HDD or SSD and stores various data such as programs, data required for program execution, and image data used for learning.

[0147] The communication unit 2004 is an interface for communicating with a network such as a LAN, a WAN, or the Internet. The display unit 2006 includes a display and a display control circuit, and displays data input from the CPU 2001. The input unit 2005 includes input devices such as a keyboard, a mouse, or other pointing devices, and transmits data input by a user via the input devices to the CPU 2001. The data transfer I / F 2008 is an interface for transmitting and receiving data to and from an external device. For example, the connection method for the data transfer I / F 312 is not particularly limited, and may be, for example, USB, IEEE 1394, or the like. The connection method may be wired or wireless.

[0148] The learning unit 2000 includes a filter (a calculation circuit, memories such as RAM and registers) for executing learning using CNN, and executes learning according to set conditions 1920 .

[0149] 21 is a diagram showing the functional configuration and data processing process of the learning device 1901. As shown in Fig. 21, the learning device 1901 has a condition input unit 2101, a setting unit 2102, and a learning unit 2000. The learning unit 2000 has an acquisition unit 2110, a RAM 2130, a convolution calculation unit 2150, an output unit 2160, a comparison unit 2170, an update unit 2180, etc. These functional units are realized, for example, by the CPU 2001 executing a program stored in the ROM 102.

[0150] The RAM 2003 stores an image data group 2106, which is a plurality of pieces of training data. The image data group 2106 is input from an external device of the training device 1901 via the communication unit 2004 or from a portable storage medium via the data transfer I / F 2008, and is stored in the RAM 2003. Note that FIG. 20 is a diagram assuming that the training data is divided when the training unit 2000 reads the training data from the RAM 2003, but this is not limited thereto. Data of image blocks divided in advance by preprocessing may be stored in the RAM 2003. Furthermore, the image blocks may be stored in the RAM 2003 with reference data 2122 previously added to them.

[0151] The condition input unit 2101 accepts input of various conditions for the learning performed by the learning unit 2000. The input conditions include the above-mentioned condition 1920 shown in FIG. 19(b). Specifically, the conditions include conditions related to division of learning data (division position 1921, division size 1922), reference data acquisition conditions (reference data position 1923), padding conditions (padding method 1924, padding position 1925), and CNN conditions 1926.

[0152] The condition input unit 2101 also receives input of a teacher image 2103, an update count 2104, and a filter coefficient 2105. The teacher image 2103 is image data indicating the correct answer, and is compared with an output image obtained as a result of the convolution operation in the comparison unit 2170. The update count 2104 is an upper limit value for the number of times that the update unit 2180 repeats updating of the filter coefficient. An arbitrary value is set as the initial value of the filter coefficient 2105. For example, a random value is set as the initial value of the filter coefficient 2105, and the value is gradually updated and optimized by repeating learning. In the second embodiment, it is assumed that information is manually input to the condition input unit 2101 by a user. The condition input unit 2101 transfers the received information to the setting unit 2102.

[0153] The setting unit 2102 receives the learning data division conditions, reference data acquisition conditions, padding conditions, CNN conditions 1926, teacher images 2103, update count 2104, and filter coefficients 2105 accepted by the condition input unit 2101, and sets them in the learning unit 2000. The division conditions are a division position 1921 and a division size 1922. The reference data acquisition condition is a reference data position 1923. The padding conditions are a padding method 1924 and a padding position 1925.

[0154] Specifically, the setting unit 2102 sets the CNN conditions 1926 to the learning unit 2000. The learning unit 2000 constructs a CNN model in accordance with the CNN conditions 1926. The setting unit 2102 also sets a division position 1921 and a division size 1922 to the data division unit 2111. The setting unit 2102 also sets a reference data position 1923 to the acquisition unit 2110. The setting unit 2102 also sets a padding method 1924 and a padding position 1925 to the padding unit 2140. The setting unit 2102 also sets a teacher image (correct answer) to the comparison unit 2170, and sets the number of updates to the update unit 2180.

[0155] The acquisition unit 2110, data division unit 2111, padding unit 2140, and convolution operation unit 2150 of the learning unit 2000 are similar to the acquisition unit 410, data division unit 411, padding unit 440, and convolution operation unit 430 of the inference unit 104 in Fig. 4. The RAM 2130 has an input data area, a filter coefficient storage area, and an output data storage area, similar to the SRAM shown in the first embodiment and its modifications shown in Figs. 10, 14, and 16 to 18, for example.

[0156] The acquisition unit 2110 acquires learning data from the page-by-page image data group 2106 stored in the RAM 2003 based on the conditions 1920 set by the setting unit 2102. The acquisition unit 2110 divides the learning data into image blocks of a predetermined size using the data division unit 2111, sequentially reads the data in processing order as target data 2121, and stores the data in the input data area of ​​the RAM 2130 of the learning unit 2000. At this time, the acquisition unit 2110 may also acquire information on the division position. This is because the method of acquiring reference data may be changed depending on the division position. In the second embodiment, the divided target data 2121 for one image block is stored in the input data area. In addition, the acquisition unit 2110 acquires reference data 2122 from a group of pixels existing around the acquired image block (target data 2121) in the original image (learning data), and stores the reference data in the input data area of ​​the SRAM 310.

[0157] The acquisition unit 2110 can identify the position of the original image from which to acquire reference data, based on the information on the reference data position 1923 set in the setting unit 2102 and the information on the division position 1921 and division size 1922. The acquisition unit 2110 acquires reference data 2122 from a data group existing around the acquired target data 2121, i.e., from the original image 501 stored in the RAM 2003, and expands it in the input data area. Data other than the reference data 2122 out of the margin data is filled in by, for example, padding.

[0158] The padding unit 2140 pads the data that is insufficient for the convolution operation based on the padding method 1924 and padding position 1925 set by the setting unit 2102 .

[0159] As described above, the conditions 1920 set in the setting unit 1920 of the learning device 1901 are the same as the division position, division size, reference data position, padding method, and padding position set in the inference unit 104 shown in the first embodiment. Therefore, the input data 2120 expanded in the RAM 2130 of the learning unit 2000 is acquired using the same size and reference data acquisition method as the input data in the inference unit 104 of the printer 100. Padding data is also added in the same way as the padding data in the inference unit 104 of the printer 100. For example, like the input data for the inference unit 104 shown in FIG. 10 , the input data is stored in the RAM 2130 with some of the margin data removed.

[0160] Arbitrary initial values ​​are set for the filter coefficients 2105. For example, if the processing layer is configured with n stages of filters, the acquisition unit 2110 sets initial values ​​for the filter coefficients for each of the n stages and stores them in the filter coefficient area of ​​the RAM 2130.

[0161] The convolution calculation unit 2150 reads the filter coefficients of the processing target stage from the filter coefficient area of ​​the RAM 2130 and sets them in the coefficient register. The convolution calculation unit 2150 also sets data within a predetermined calculation range from the calculation data including the input data 2120 and padding data stored in the RAM 2130 in the calculation register, and performs a product-sum calculation with the filter coefficients set in the coefficient register. The convolution calculation unit 2150 performs calculations on all pixels of the calculation data while sliding the calculation range, and writes the output image (features) that are the calculation results in the output data area of ​​the RAM 2130. If there is a next layer, the output image of that layer is used as input and the convolution calculation is repeated. When the convolution calculations for all processing layers are completed, the output unit 2160 obtains a feature vector.

[0162] The comparison unit 2170 compares the feature vector output from the output unit 2160 with the teacher image 2103, which is the correct answer, and calculates the error.

[0163] The update unit 2180 propagates the error calculated by the comparison unit 2170 from the output layer side to the input layer side, and updates the filter coefficients in each neuron by calculating a gradient from this error. For the next input image, learning is performed using the updated filter coefficients. When the upper limit of the number of updates set by the setting unit 2102 is reached, the learning unit 2000 ends learning. The learning unit 2000 outputs learned filter coefficients 2190 as the result of learning.

[0164] (How to generate filter coefficients) Here, the process of generating filter coefficients (learning process) in the learning device 1901 will be described in more detail.

[0165] (generation environment) FIG. 22 is a schematic diagram of the vicinity of an input section of a typical CNN model. In this embodiment, a case where filter coefficients are generated using the learning device 1901 shown in FIG. 19 will be described. Data 2201 is data input to the learning device 1901. For example, when the input data is image data, three channels (R, G, and B) are prepared for each coordinate. Neurons 2211 to 2216 constitute the processing layer of the input data 2201. The neurons 2211 to 2216 are filters for convolving the input data 2201, and each neuron holds a different filter coefficient. This is because one filter extracts one characteristic; therefore, to extract multiple different features, multiple filter processes must be performed. It is assumed that the filter coefficients held in each filter (neurons 2211 to 2216) are for three channels (R, G, and B). As will be described later, in the initial state, the values ​​of the filter coefficients are variables to be generated. For example, neuron 2211 holds 3×3 filter coefficients for three channels, R, G, and B, for convolving input data 2201 .

[0166] In the example of Figure 22, six processing layers are provided. As a result, six features are extracted after the first stage of processing. Neurons 2221 to 2224 are the second stage of processing layer. The second stage of processing layer receives the results of neurons 2211 to 2216 in the first stage as input, and performs similar convolution operations using four processing layers (neurons 2221 to 2224). Next, we will explain the activation function.

[0167] FIG. 23 is a schematic diagram showing an overview of the processing unit in the processing layer. In the processing layer, input data is convolved by a convolution operation unit 2311, and the result is input to an activation function unit 2312. The activation function unit 2312 is a function with nonlinear characteristics. Specifically, a sigmoid function, a ReLU function, or the like is used. The activation function unit 2312 receives the calculation result from the convolution operation unit 2311, performs a function operation, and outputs the result. Depending on the input from the convolution operation unit 2311, the output from the activation function unit 2312 may be weak. In other words, whether information is transmitted from the activation function unit 2312 to the next layer depends on the coefficients held in the convolution operation unit 2311. This process is repeated in succession until the final stage of the CNN model (not shown) is reached, thereby generating features.

[0168] (Getting the error) Figure 24 is a schematic diagram showing the vicinity of the output section of a typical CNN model. After passing through the input in Figure 22 and reaching the final stage, features are output through an activation function 2401. In this way, the features of the input image are obtained. As described above, the CNN model obtains features from the input data using a huge number of filter calculations and activation functions.

[0169] The true feature values ​​that indicate the characteristics of the input image can be prepared by methods other than learning. For example, the values ​​can be determined by human judgment through visual inspection. Below, this value (true feature value) is referred to as the "correct answer" or training data. By taking the difference between the feature value obtained from the CNN model and the correct answer, the error of the input data can be obtained.

[0170] (Error propagation) The error between the output of each layer and the correct answer is propagated from the output layer side to the input layer side, and the gradient is calculated from this error, thereby updating the filter coefficients in each neuron. This is called the error backpropagation method. The error backpropagation method is a well-known technique, and is described, for example, in Japanese Patent Application Laid-Open No. 6-96046. As a result of this error propagation, the filter coefficients in all layers of the CNN are updated. Note that the error backpropagation method is just one example, and other methods may also be used to update the filter coefficients.

[0171] (Overall flow of the learning process) Next, the overall flow of the learning process executed by the learning device 1901 will be described. 25 is a flowchart illustrating the overall flow of the learning process executed by the learning device 1901. A program for executing the process shown in this flowchart is stored in the storage unit 2007 or ROM 2002 of the learning device 1901. The CPU 2001 calls this program and executes the process of this flowchart while using the RAM 2003 as a work area. For example, when the user issues an instruction to start learning via the input unit 2005, the CPU 2001 starts this flowchart.

[0172] In S2501, the CPU 2001 (condition input unit 2101) accepts the design of the structure of a CNN model for training by the user. The CPU 2001 accepts input of parameters related to the model structure. The parameters related to the model structure include the number of convolutional layers (filters), filter size, the number of reductions and expansions, etc., and correspond to the CNN conditions 1926 among the above-mentioned conditions 1920. Note that an existing model may be used instead of the parameters.

[0173] In S2502, the CPU 2001 (condition input unit 2101) accepts the setting of model conditions. The model conditions include, among the above-mentioned conditions 1920, division position 1921, division size 1922, reference data position 1923, padding method 1924, and padding position 1925. The CPU 2001 also accepts a teacher image 2103 and an update count 2104.

[0174] In S2501 and S2502, the user sets the same model structure (CNN conditions) and model conditions as those of the inference unit 104 of the printer 100 in the learning device 1901. At this time, the condition input unit 2101 (CPU 2001) of the learning device 1901 may display a UI (user interface) screen for the user to set the model structure and model conditions.

[0175] 26 shows an example of a UI screen 2600 for the user to set a model structure and model conditions. The UI screen 2600 includes a model structure setting area 2601, a learning start button 2602, a mode selection button 2603, and a layer selection button 2604.

[0176] In the model structure setting area 2601, the user sets parameters related to the model structure, such as the number of convolution layers, filter size, and number of reductions and enlargements. A model structure diagram according to the set parameters is displayed in the model structure setting area 2601. In the second embodiment, the user sets the same model structure (at least the number of layers, filter size, and number of reductions and enlargements) as the CNN model structure of the printer 100. Based on the model structure set here, the range of margin data required for convolution calculations in each filter is determined.

[0177] The mode selection button 2603 is an operation unit for selecting either an automatic mode in which the reference data to be acquired from the margin data is automatically set, or a manual mode in which the user manually sets the reference data. When the user selects the automatic mode, the CPU 2001 automatically sets the reference data position for the entire model. Note that in the automatic mode, the CPU 2001 may allow the user to set the ratio of reference data to margin data. A setting screen 2607 for this purpose may be displayed as a pop-up. The ratio of reference data to margin data can be set to any value between 0% and 100%. When 100% is set, all pixels of the margin data are composed of reference data (actual data), and when 0% is set, all pixels of the margin data are padded. The user simply sets the same ratio of reference data to margin data as that set by the inference unit 104 of the printer 100 in the setting screen 2607.

[0178] The ratio set on the setting screen 2607 is set uniformly for the entire model in automatic mode. The CPU 2001 determines the reference data position for each layer based on the ratio set by the user. The processing of the CPU 2001 in automatic mode will be described in the third embodiment.

[0179] When the user selects manual mode, the CPU 2001 accepts input for selecting a layer and detailed settings for the reference data position in the selected layer. A layer is selected when the user designates one of the layers of the model structure displayed in the model structure setting area 2601 while pressing the layer selection button 2604. For the selected layer, the CPU 2001 accepts a user operation to select either "referencing actual data" or "padding." When radio button 2605 is set to ON, "referencing actual data" is selected, and when radio button 2606 is set to ON, "padding" is selected.

[0180] If "Refer to actual data" is selected, the settings will be such that all margin data for the selected layer is actual data (data of the original image). If “padding” is selected, the CPU 2001 further accepts settings for a thinning position 2608 and a padding value 2609 .

[0181] The thinning position 2608 can be selected from one or more of top, bottom, left, and right. For example, if top and bottom are selected by the user, actual data is not taken into the top and bottom regions of the margin data required for the convolution operation, but padding is performed, as shown in FIG. 10. That is, actual data is taken into the left and right regions of the margin data as reference data, and padding data is added to the top and bottom regions. If right and left are selected by the user for the thinning position 2608, actual data is not taken into the right and left regions of the margin data required for the convolution operation, but padding is performed, as shown in FIG. 14. That is, actual data is taken into the top and bottom regions of the margin data as reference data, and padding data is added to the right and left regions. Note that the thinning position may be selected from only top, only bottom, only left, or only right.

[0182] In the example of FIG. 26 , either the pixel value or mirror image can be selected as the padding value 2609. If the pixel value is selected by the user, the CPU 2001 accepts input of any real value into the input field 2610. In this case, the CPU 2001 uses the input value as padding data to fill in the pixels lacking data. The example of FIG. 26 shows a state in which "0" is input into the input field 2610. If mirror image is selected by the user, the CPU 2001 mirror-reflects a portion of the input data input to the learning unit 2000 to fill in the pixels lacking data. Note that, although the pixel value and mirror image can be selected as the padding value 2609 in the example of FIG. 26 , other padding methods may be selected without being limited thereto. For example, the average value of the pixel values ​​of multiple pixels surrounding the padding position may be used.

[0183] The thinning position set by the thinning position 2608 on the UI screen 2600 is set in the setting unit 2102 as the padding position 1925. Of the margin data required for the convolution operation, pixels excluding those at the position set by the thinning position 2608 are set in the setting unit 2102 as the reference data position 1923. In addition, the padding value 2609 set on the UI screen 2600 is set in the setting unit 2102 as the padding method 1924.

[0184] In the second embodiment, the user selects manual mode on the UI screen 2600 and sets the thinning position 2608 and padding value 2609 so that they are the same as the margin data conditions of the printer 100. As a result, of the model conditions that need to be set in the learning device 1901, the reference data position 1923, padding method 1924, and padding position 1925 are set.

[0185] In the example of the UI screen 2600 in FIG. 26, the options for the thinning position 2608 do not include a discrete arrangement or a staggered arrangement, but these may be included as options. Furthermore, different padding values ​​may be set for each channel and each processing layer. Among the model conditions, the settings for the division position 1921 and division size 1922 may also be included in the UI screen 2600. In this embodiment, the user sets the image division position 1921 and division size 1922 so that they are the same as the image data division conditions of the printer 100. The same data division program may also be executed on the printer 100 and the learning device 1901. As a result, the division position 1921 and division size 1922 are set among the conditions that need to be set on the learning device 1901.

[0186] Setting conditions using the UI screen 2600 is one example, and conditions may be set by other methods. For example, the user may directly specify the model structure (CNN conditions), padding method, padding position, division position, division size, reference data position, etc. for the program code. Furthermore, with regard to setting the model structure (setting the CNN conditions 1926), the learning device 1901 may obtain a sharable CNN model published on the Web via the communication unit 2004 and reflect it in the program code.

[0187] As described above, when the model design and model condition setting are completed and the user operates the learning start button 2602 on the UI screen 2600, the process proceeds to S2503.

[0188] In S2503, the CPU 2001 initializes the filter coefficients. The CPU 2001 sets the filter coefficients of the filter size set in S2501 to arbitrary values. For example, the CPU 2001 sets random values.

[0189] In S2504, the CPU 2001 sets the CNN model, model conditions, and filter coefficients set in S2501 to S2503 in the learning unit 2000, and starts learning. Learning will be described later. When learning is complete, the process proceeds to S2505.

[0190] In S2505, the CPU 2001 outputs the learned filter coefficients, which are the learning results. This flowchart ends with the above processing.

[0191] (Learning process) The learning process executed in S2504 will be described. Fig. 27 is a flowchart showing the flow of the learning process. Fig. 28 is a diagram explaining the division and padding of learning data. When the CPU 2001 of the learning device 1901 completes the processing up to S2503 in Fig. 25, it then starts the processing of the flowchart shown in Fig. 27. It is assumed that a learning image data group 2106 has been input to the learning device 1901 and stored in the RAM 2003 before this flowchart starts.

[0192] In S2701, the CPU 2001 of the learning device 1901 acquires any one image data (hereinafter referred to as the original image 2801) from the image data group 2106 and transfers it to the learning unit 2000. The data dividing unit 2111 of the learning unit 2000 divides the original image 2801 in accordance with the division position 1921 and division size 1922 set in the setting unit 2102, to obtain multiple image blocks 2802 shown in FIG. 28. Position information in the original image is added to each of the multiple image blocks 2802. The position information in the original image is the information of the division position 1921.

[0193] In S2702, the CPU 2001 may increase the image block 2802 by an augmentation process. The augmentation process results in an augmented image group 2803 consisting of a plurality of augmented images. The augmented image group 2803 is a collection of a plurality of augmented image data generated by processing the image block 2802. Processing of the image block 2802 may include, for example, mirror-inversion duplication, partial overwriting of any image element such as a photograph, text, or graphics, etc. Position information in the original image 2801 is added to each of the augmented image data. The augmented image group 2803 may be stored in the RAM 2130 of the learning unit 2000, or in any storage area such as the RAM 2003, memory unit 2007, or external storage of the learning device 1901. The processing of S2701 and S2702 may be executed by the learning unit 2000.

[0194] In S2703, the acquisition unit 2110 of the learning unit 2000 sequentially acquires one arbitrary piece of augmented image data from the augmented image group 2803 as target data 2121, and performs learning using a CNN model. In this embodiment, the acquisition unit 2110 of the learning unit 2000 acquires one piece of augmented image data and loads it in RAM 2130. At this time, the acquisition unit 2110 also acquires reference data 2122 surrounding the acquired augmented image data based on the information of the reference data position 1923 set in the setting unit 2102, and loads it in RAM 2130.

[0195] For example, when the thinning positions 2608 are set to the top and bottom on the UI screen 2600 of FIG. 26 , the acquisition unit 2110 acquires data on the right and left of the margin data from the pixel group surrounding the augmented image data (target data 2121) in the original image 2801. The reference data 2122 and augmented image data (target data 2121) acquired in this manner are expanded in the input data area of ​​the RAM 2130 as the input data 2120 shown in FIG. 21 . The acquisition unit 2110 also acquires filter coefficients that are stored in advance in the ROM 106. When there are multiple stages to be processed by the filter, the acquisition unit 2110 acquires multiple filter coefficients. The acquisition unit 2110 stores the received one or more filter coefficients in the filter coefficient area of ​​the RAM 2130.

[0196] The convolution calculation unit 2150 sets the filter coefficients of the target stage in the coefficient register, and sequentially sets the data in the calculation range from the input data expanded in the input data area of ​​the RAM 2130 in the calculation register to perform the convolution calculation. Note that, at the beginning of learning, the filter coefficients are set to arbitrary random values. Furthermore, if there is insufficient data in the calculation range, the padding unit 2140 performs padding. The padding unit 2140 assigns padding data to the calculation register based on the padding method and padding position information set in the setting unit 1920. The convolution calculation unit 2150 performs the convolution calculation while sliding the calculation range, and writes the calculation result to the corresponding pixel in the output data area. When the convolution calculation is completed for the input data expanded in the input data area, an output image indicating feature quantities is obtained.

[0197] If there is a next stage, the convolution operation unit 2150 sets the filter coefficients of the target stage in the coefficient register, sequentially performs convolution operations on the input data expanded in the input data area, and writes the operation results to the corresponding pixels in the output data area. The above processing is performed for all pixels of the input data. If there is a further next layer, CNN processing is performed on the next layer using the output of the previous layer as input. As a result of the CNN processing, the output unit 2160 obtains a feature vector of the input augmented image data.

[0198] In S2704, the output unit 2160 stores the feature vector obtained in S2603 in the RAM. In S2705, the output unit 2160 determines whether processing has been completed for all the augmented image data. If processing has not been completed, the process returns to S2603. If processing has been completed for all the augmented images, the process proceeds to S2706.

[0199] In S2706, the output unit 2160 adds up all of the feature vectors obtained in the processes up to this point. The added feature will be referred to as the “total feature” hereinafter. The output unit 2160 transfers the total feature to the comparison unit 2170. In S2707, the comparison unit 2170 compares (takes a difference from) the total feature amount acquired from the output unit 2160 with the teacher image 2103 to calculate an error. The teacher image 2103 is a correct vector that has been added the same number of times as the number of padding processes, and is set by the setting unit 2102.

[0200] In S2708, the update unit 2180 propagates the error calculated in S2707 to the CNN model to update the filter coefficients. As a result of this error propagation, the filter coefficients for all layers used in the CNN model are determined.

[0201] In S2709, the learning unit 2000 determines whether the processes from padding to error propagation (S2702 to S2708) have been completed for all image blocks. If the processes have not been completed, the process returns to S2702 and starts with padding for the next image block. Note that the next learning uses filter coefficients that reflect the results of the error propagation performed in the previous process. By repeatedly performing the above learning process, the filter coefficients are successively optimized.

[0202] If the processes from padding to error propagation (S2702 to S2708) have been completed for all divided images for the number of updates set by the setting unit 2102, the process proceeds to S2710.

[0203] In S2710, the CPU 2001 determines whether the above-described learning has been completed for all of the input image data group 2106. If not, the process returns to S2701 and repeats the processes of S2701 to S2709. If learning has been completed for all of the input image data group 2106, the process of this flowchart ends. Note that in this embodiment, padding is performed after dividing the image, but this is not limited to this. The original image may be padded first, and then divided.

[0204] The filter coefficients obtained by learning can be output as parameters. The CPU 2001 of the learning device 1901 outputs the filter coefficients obtained by learning to a predetermined output destination in accordance with a user's instruction. The output destination may be, for example, a storage medium connected to the data transfer I / F 2008, a printer 100 connected via the data transfer I / F 2008 or the communication unit 2004, or an external device.

[0205] The printer 100 acquires filter coefficients from a storage medium or a communication-connected learning device 1901. The acquired filter coefficients are stored in the ROM 106 of the printer 100. This allows the inference unit 104 of the printer 100 to perform inference using the filter coefficients generated by the learning device 1901. The filter coefficients are determined by learning performed under conditions similar to the CNN conditions, division conditions, reference data acquisition conditions, and padding conditions in the inference performed by the printer 100. This improves the attribute accuracy obtained as a result of the inference. This is because, when the filter coefficients are optimized in learning, the filter coefficients are determined taking into account the reliability of the "part where margin data has been reduced and padded."

[0206] Specifically, in the inference process in the printer 100, as shown in FIG. 10, the upper and lower margin areas of the margin data required for the convolution calculation of the target data are not acquired. In this case, the data learned by the learning unit 2000 also does not acquire the upper and lower margin areas, but is added by padding. Learning in this manner causes the filter coefficients to converge to filter coefficients that are less affected by the position of the "upper and lower margin areas," due to the difficulty of converging feature value errors in the "upper and lower margin areas." When the filter coefficients generated in this manner are implemented in the ROM 102 of the printer 100, the inference unit 104 of the printer 100 also outputs feature values ​​obtained using calculation data with padded upper and lower margin areas. Therefore, feature values ​​with underestimated reliability can be obtained. This increases the reliability of the feature values ​​resulting from the inference process.

[0207] In the present embodiment, as an example corresponding to the data acquisition method shown in FIG. 10 , when acquiring padded image data for learning, data in the upper and lower regions of the margin data is thinned (not acquired) and padded. However, the present embodiment is not limited to this example. If the printer's inference unit 104 thins out left and right margin data as shown in FIG. 14 , data in the left and right regions of the margin data may be thinned (not acquired) and padded when acquiring padded image data for learning. Furthermore, if the printer's inference unit 104 thins out data in a staggered pattern as shown in FIG. 16 , when acquiring padded image data for learning, the margin data may be thinned out in a staggered pattern and the missing data may be padded. Furthermore, the range of the margin data is not limited to one line and may include multiple lines. In this case, if the printer's inference unit 104 thins out margin data in a staggered pattern as shown in FIG. 18 , when acquiring padded image data for learning, the margin data may be thinned out (reduced) in a staggered pattern and the missing data may be padded. In any case, the learning device 1901 acquires and performs learning on calculation data under the same division conditions, reference data acquisition conditions, and padding conditions as the inference unit 104 of the printer 100. This increases the accuracy of the feature values ​​that are ultimately obtained.

[0208] <Third embodiment> Next, the automatic mode will be described as the third embodiment. The automatic mode is selected by operating the mode selection button 2603 on the UI screen 2600 shown in FIG.

[0209] FIG. 29 is a flowchart showing the overall flow of the learning process in the third embodiment. Note that S2901 and S2904 to S2907 in the flowchart in FIG. 29 correspond to S2501 and S2502 to S2505 in FIG. 25, respectively. That is, the process shown in FIG. 29 differs from the second embodiment in that the processes of S2902 and S2903 are added after S2501 in FIG. 25. The following explanation will focus on the differences from the second embodiment. Note that in the third embodiment, as in the second embodiment, the learning conditions for obtaining filter coefficients used in inference are the same as the inference conditions in the inference unit 104. This improves the accuracy of the attribute probabilities finally obtained as the result of inference.

[0210] (System configuration and functional configuration) The system configuration and functional configuration of an information processing system 3000 of the third embodiment will be described. FIG. 30 is a diagram showing the system configuration and functional configuration of the information processing system 3000 of the third embodiment. As shown in FIG. 30, the information processing system 3000 of the third embodiment includes an inference device 3010 and a learning device 3001, which are communicatively connected via their respective data transfer units or communication units. The hardware configuration of the learning device 3001 is the same as that of the learning device 1901 shown in FIG. 20, and the inference device 3010 is an embedded device having an inference unit, such as the printer 100 shown in FIG. 1, with the hardware configuration being the same as that of the printer 100 shown in FIG. 1.

[0211] In the third embodiment, a learning device 3001 includes a learning unit 2000, a device condition acquisition unit 3002, a model condition determination unit 3003, a model construction unit 3004, and a mode selection unit 3005. An inference device 3010 includes an inference unit 104, a use case setting unit 3011, and a device condition transfer unit 3012. Speed ​​conditions 3013, SRAM capacity 3014, and padding conditions 3015 are stored in the ROM 102 or DRAM 103 of the inference device 3010. These functional units are realized in each device by, for example, the CPU executing a program stored in the ROM.

[0212] The learning unit 2000 of the learning device 3001 is similar to the learning unit 2000 of the second embodiment shown in FIG. When the automatic mode is selected by the mode selection unit 3005, the device condition acquisition unit 3002 requests the inference device 3010 to acquire the device conditions, and receives the device conditions transferred from the inference device 3010. The device conditions include a speed condition 3013, an SRAM capacity 3014, and a padding condition 3015. When the inference device 3010 is a printer, the speed condition 3013 is, for example, an upper limit value for the printing speed, and is determined by the use case setting unit 3011 and stored in the RAM 102. The SRAM capacity 3014 is the capacity (circuit scale) of the SRAM used for the filter of the inference unit 104, and is stored in advance in the ROM 102.

[0213] The model condition determination unit 3003 determines the model conditions of the learning unit 2000 based on the device conditions of the inference device 3010 acquired by the device condition acquisition unit 3002, and sets them in the model construction unit 3004. Specifically, the model condition determination unit 3003 determines the model conditions, particularly the reference data acquisition conditions, so as not to exceed the values ​​of the speed condition 3013 and SRAM capacity 3014 of the inference device 3010. The method for determining the model conditions will be described later.

[0214] The model construction unit 3004 constructs a CNN model based on information for constructing a CNN model acquired from the inference device 3010. In the third embodiment, the device condition acquisition unit 3002 acquires the device conditions of the inference device 3010 from the inference device 3010 as information for constructing a CNN model. The model condition determination unit 3003 determines model conditions based on the device conditions, and the model construction unit 3004 constructs a CNN model based on the determined model conditions.

[0215] The mode selection unit 3005 accepts a user's selection of whether to use an automatic mode in which the conditions for acquiring reference data are set automatically or a manual mode in which the conditions are set manually. For example, either the automatic mode or the manual mode is selected by operating the mode selection button 2603 on the UI screen 2600 shown in Fig. 26. In the third embodiment, the automatic mode is selected.

[0216] The inference unit 104 of the inference device 3010 is similar to the inference unit 104 of the first embodiment shown in FIG. The use case setting unit 3011 accepts the setting of a use case that can be set in the inference device 3010 .

[0217] FIG. 31 is a diagram showing an example of a UI screen 3100 for print settings as an example of a use case. The use case setting unit 3011 displays, for example, the UI screen 3100 for setting the use case shown in FIG. 31 on the operation panel 106 of the printer and accepts input of multiple parameter values ​​by the user. The parameters include, for example, paper size, paper type, color / monochrome selection, single-sided / double-sided selection, and print quality. The use case setting unit 3011 sets speed conditions that are not rate-determining for the required paper transport speed and print head operating speed based on the input parameter values. Information on the speed conditions may be stored in ROM 102 in advance for each use case, or the CPU 101 may acquire the required paper transport speed and print head operating speed according to the set use case and determine the speed conditions.

[0218] When the device condition transfer unit 3012 receives a request to acquire device conditions from the learning device 3001, it transfers the device conditions, including speed conditions 3013, SRAM capacity 3014, and padding conditions 3015, stored in the ROM 102 or DRAM 103, to the learning device 3001. Note that the learning device 3001 may acquire the speed conditions via PC software installed in the learning device 3001, such as a printer driver.

[0219] (Overall flow of the learning process) The flow of the learning process executed by the learning device 3001 of the third embodiment will be described with reference to Figure 29. A program for executing the process shown in this flowchart is stored in the storage unit 2007 or ROM 2002 of the learning device 3001. The CPU 2001 calls this program and executes the process of this flowchart while using the RAM 2003 as a work area. For example, when the user issues an instruction to start the learning process via the input unit 2005, the CPU 2001 starts this flowchart.

[0220] In S2901, the CPU 2001 of the learning device 3001 accepts the user's design of the structure of a CNN model for learning. The CPU 2001 accepts input of parameters related to the model structure. The parameters related to the model structure include the number of convolutional layers (filters), the filter size, the number of reductions and expansions, etc. In S2901, the user sets a model structure (CNN conditions) similar to that of the inference unit 104 of the printer 100 in the learning device 1901. At this time, the CPU 2001 of the learning device 3001 may display a UI screen 2600 for the user to set the model structure.

[0221] In S2902, the CPU 2001 determines whether the automatic mode has been selected by the user. For example, either the automatic or manual mode is set by operating the mode selection button 2603 on the UI screen 2600. If the manual mode has been selected, the process proceeds to S2904. The processes of S2904 to S2907 are the same as the processes of S2502 to S2505 in the second embodiment. If the automatic mode has been selected, the process proceeds to S2903.

[0222] In S2903, the CPU 2001 (device condition acquisition unit 3002) of the learning device 3001 requests the inference device 3010 to acquire the device conditions, and receives the device conditions transferred from the inference device 3010. As described above, the device conditions include a speed condition 3013, an SRAM capacity 3014, and a padding condition 3015. The speed condition 3013 is the upper limit of the printing speed, is determined by the use case setting unit 3011, and is stored in the RAM 102. The SRAM capacity 3014 is the capacity (circuit size) of the SRAM used for the filter of the inference unit 104, and is stored in advance in the ROM 102.

[0223] In S2904, the CPU 2001 (model condition determination unit 3003) of the learning device 3001 determines model conditions based on the device conditions of the inference device 3010 acquired in S2903, and sets them in the model construction unit 3004. Since the design of the CNN model is completed in S2901, in order to determine the model conditions so as not to exceed the value of SRAM capacity 3014, the CPU 2001 simply adjusts the image division size and the amount of reference data in the margin data (hereinafter abbreviated as reference data amount).

[0224] Dividing the original image into smaller pieces reduces the amount of image data transferred to and stored in the RAM of the learning unit 2000, but the number of margin data processed increases by the number of divided images, slowing down the processing speed. Replacing margin data with padding reduces the amount of image data transferred to and stored in the RAM of the learning unit 2000, but the increased amount of padding reduces the reliability of learning, resulting in a decrease in the accuracy of attribute probability determination in inference. Therefore, the CPU 2001 (model condition determination unit 3003) of the learning device 3001 determines a division size and reference data amount that satisfy the speed condition 3013 and achieves a determination accuracy higher than a predetermined standard. If multiple candidates are determined, the CPU 2001 of the learning device 3001 may execute steps S2905 to S2907 for each of the multiple candidates, and set the candidate (division size and reference data amount) that produced the highest performance results as the model condition.

[0225] Of the model conditions, the padding method 1924 and padding position can be set by setting the padding conditions 3015 (padding method, padding position) of the inference device 3010 acquired in S2903. The division position 1921 is determined based on the division size and original image size determined in S2904. The reference data position 1923 is determined based on the margin data range and padding position determined based on the CNN conditions designed in S2901, and the reference data amount determined in S2904.

[0226] As described above, when the model design and model condition setting are completed and the user operates the learning start button 2602 on the UI screen 2600, the process proceeds to S2905.

[0227] In S2905, the CPU 2001 initializes the filter coefficients. The CPU 2001 sets the filter coefficients of the size set by the model design to arbitrary values. For example, random values ​​are set.

[0228] In S2906, the CPU 2001 sets the model conditions and filter coefficients in the learning unit 2000 and starts learning. The learning is the same as in the second embodiment. When the learning is completed, the process proceeds to S2907. In S2907, the CPU 2001 outputs the learned filter coefficients, which are the learning results. This flowchart ends with the above processing.

[0229] The inference device 3010 acquires learned filter coefficients from the learning device 3001. The acquired learned filter coefficients are stored in the ROM 102 of the printer 100. This allows the inference unit 104 of the printer 100 to perform inference using the learned filter coefficients generated by the learning device 3001. The filter coefficients reflect the CNN conditions for inference and are determined by learning performed under model conditions determined based on the device conditions of the inference device 3010. The model conditions are determined based on the speed conditions and SRAM capacity constraints required of the inference device 3010, and so as to maintain determination accuracy above a predetermined standard. Therefore, even in an inference device 3010 that is limited by speed conditions and SRAM capacity, such as an embedded device, inference can be performed and a decrease in the reliability of the inference results is suppressed.

[0230] <Modification of the third embodiment> In the third embodiment described above, an example has been described in which the learning device 3001 acquires the device conditions (speed conditions, SRAM capacity, and padding conditions) of the inference device 3010 and determines the model conditions (division size and reference data amount) for learning. However, this embodiment is not limited to this example, and the inference device 3010 may determine the reference data acquisition conditions based on its own device conditions and the model structure of the inference unit (CNN conditions) and transfer them to the learning device.

[0231] (System configuration and functional configuration) The system configuration and functional configuration of an information processing system 3200 according to a modified example of the third embodiment will be described. FIG. 32 is a diagram showing the system configuration and functional configuration of the information processing system 3200 according to a modified example of the third embodiment. As shown in FIG. 32, the information processing system 3200 includes an inference device 3210 and a learning device 3201, and the inference device 3210 and the learning device 3201 are communicatively connected via their respective data transfer units or communication units. The hardware configuration of the learning device 3201 is the same as that of the learning device 1901 shown in FIG. 20. The inference device 3210 is an embedded device having an inference unit 104. For example, if the inference device 3210 is the printer 100 shown in FIG. 1, the hardware configuration is the same as that of the printer 100 shown in FIG. 1.

[0232] In the modification of the third embodiment, a learning device 3201 includes a learning unit 2000, an acquisition unit 3203, a model construction unit 3004, and a mode selection unit 3005. The learning unit 2000, the model construction unit 3004, and the mode selection unit 3005 are the same as those in FIG. The inference device 3210 has an inference unit 104, a use case setting unit 3011, a device condition acquisition unit 3212, and a reference data acquisition condition setting unit 3213. In addition, the ROM 102 or DRAM 103 of the inference device 3210 holds a speed condition 3013, an SRAM capacity 3014, and a padding condition 3015. The inference unit 104 and the use case setting unit 3011 are the same as those in FIG. 30.

[0233] When automatic mode is selected, the acquisition unit 3203 of the learning device 3201 requests the inference device 3210 to acquire reference data acquisition conditions. The acquisition unit 3203 also receives the reference data acquisition conditions transmitted from the inference device 3210 and sets them in the model construction unit 3004.

[0234] When the reference data acquisition condition setting unit 3213 of the inference device 3210 receives a request to acquire reference data acquisition conditions from the learning device 3001, it transmits the reference data acquisition conditions of the inference device 3210 to the learning device 3201. The reference data acquisition condition setting unit 3213 first acquires device conditions including speed conditions 3013, SRAM capacity 3014, and padding conditions 3015 stored in ROM 102 or DRAM 103 by the device condition acquisition unit 3212.

[0235] The reference data acquisition condition setting unit 3213 of the inference device 3210 determines the reference data acquisition conditions in the inference unit 104 based on the device conditions (speed condition 3013, SRAM capacity 3014) of the inference device 3210, and transmits them to the learning device 3201. Specifically, the reference data acquisition condition setting unit 3213 determines the model conditions, particularly the reference data acquisition conditions, so that they do not exceed the values ​​of the speed condition 3013 and SRAM capacity 3014 of the inference device 3210.

[0236] The method for determining the model conditions is the same as in the above-described third embodiment. The reference data acquisition condition setting unit 3213 determines the reference data acquisition conditions that do not exceed the value of the SRAM capacity 3014, satisfy the speed condition 3013, and ensure that the accuracy of the inference determination is higher than a predetermined standard. The reference data acquisition condition setting unit 3213 transmits the determined reference data acquisition conditions (division size and reference data amount) to the learning device 3201.

[0237] The acquisition unit 3203 of the learning device 3201 acquires the reference data acquisition conditions sent from the inference device 3210 and sets them in the model construction unit 3004 .

[0238] Of the model conditions set in the learning device 3201, the padding method 1924 and padding position 1925 are preferably acquired from the inference device 3210, as in the third embodiment. The division position 1921 is determined based on the division size and original image size acquired as the reference data acquisition conditions. The reference data position 1923 is determined based on the range and padding position of margin data required for convolution calculations, which are determined from the CNN conditions, and the margin amount acquired as the margin condition.

[0239] As described above, once the model design and model condition setting are completed, the learning unit 2000 executes learning and optimizes the filter coefficients.

[0240] In this modification, the generated filter coefficients are transmitted from the learning device 3001 to the inference device 3210. The filter coefficients acquired by the inference device 3210 are stored in the ROM 102 of the printer 100. This allows the inference unit 104 of the inference device 3210 to perform inference using the filter coefficients generated by the learning device 3201. The filter coefficients reflect the CNN conditions for inference and are determined by learning performed under model conditions (particularly, reference data acquisition conditions) determined based on the device conditions of the inference device 3210. Furthermore, the model conditions are determined based on the speed conditions and SRAM capacity constraints required of the inference device 3210, and so as to maintain a predetermined accuracy. Therefore, even in an inference device 3210 that is constrained by speed conditions and SRAM capacity, such as an embedded device, inference can be performed, and a decrease in the reliability of the inference results is suppressed.

[0241] In the third embodiment, the inference device and the learning device are communicatively connected via a data transfer I / F or a communication unit to transmit and receive data, but the present disclosure is not limited to this. The inference device and the learning device may be offline, and data may be transferred using a portable storage medium.

[0242] The inferences shown in the first to third embodiments are utilized in pattern recognition of two-dimensional image data. That is, features are extracted from the two-dimensional image data by CNN, and attribute probabilities of the type of image the two-dimensional image data represents are determined and output as pattern recognition results. The inferences and learnings shown in the first to third embodiments may also be utilized in processes other than pattern recognition.

[0243] In the first to third embodiments, the data to be processed is two-dimensional image data, and a two-dimensional filter is used for the filter coefficients. However, the present disclosure is not limited to this example. For example, the present disclosure can also be applied to a case where a one-dimensional filter is used for one-dimensional time-series data, such as audio data. In general, the present disclosure can be similarly applied to data of any dimension by creating a suitable configuration according to the dimension of the feature.

[0244] Furthermore, the screen configurations of the UI screens, the contents displayed on the screens, the operation procedures, the operation methods, etc. shown in the embodiments are merely examples, and the present disclosure is not limited thereto. Furthermore, although the functions of the inference unit and the learning unit in the above-described embodiments are described as being realized by a CPU executing processing according to a program, the present disclosure is not limited thereto, and an information processing device (a processor such as a GPU) different from the CPU may be provided.

[0245] While preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that those skilled in the art can conceive of various modifications or alterations within the scope of the technical ideas disclosed herein, and it is understood that such modifications also fall within the technical scope of the present disclosure.

[0246] <Other embodiments> The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0247] The disclosure of the above-described embodiment includes the following configurations.

[0248] (Configuration 1) An information processing device that performs inference using a convolutional neural network, an acquisition means for acquiring target data from inference data input to the information processing device; a calculation means for performing a convolution operation using calculation data including the target data acquired by the acquisition means and margin data, which is data other than the target data and is necessary to obtain calculation result data of a predetermined size, and outputting the calculation result data; The information processing device is characterized in that the acquisition means acquires first data, which is a portion of the margin data, from a group of data in the inference data that exists around the target data and is separate from the target data, and does not acquire second data, which is part of the margin data but excludes the first data, from the group of data.

[0249] (Configuration 2)

[0250] 2. The information processing device according to configuration 1, further comprising padding means for padding second data, which is data excluding the first data, from the margin data.

[0251] (Configuration 3) 3. The information processing device according to configuration 1 or 2, wherein the first data is stored in a storage area together with the target data, and the second data is not stored in the storage area.

[0252] (Configuration 4) 3. The information processing device according to configuration 2, wherein the padding means writes the second data directly into a register used by the calculation means for calculation.

[0253] (Configuration 5) 5. The information processing device according to configuration 2 or 4, wherein the padding means pads the second data with an arbitrary fixed value.

[0254] (Configuration 6) 6. The information processing device according to any one of configurations 2, 4, and 5, wherein the padding means pads the second data with data based on the target data.

[0255] (Configuration 7) 7. The information processing device according to any one of configurations 1 to 6, wherein the acquisition means acquires the first data from the data group that exists in the same direction as a sliding direction of an operation range of the convolution operation with respect to the target data.

[0256] (Configuration 8) 7. The information processing device according to any one of configurations 1 to 6, wherein the acquisition means acquires the first data from the data group that exists in a direction perpendicular to a sliding direction of an operation range of the convolution operation for the target data.

[0257] (Configuration 9) 9. The information processing device according to any one of configurations 1 to 8, wherein the acquiring means acquires the first data discretely from the data group.

[0258] (Configuration 10) 10. The information processing apparatus according to configuration 9, wherein the acquiring means acquires the first data from the data group at predetermined data intervals.

[0259] (Configuration 11) The information processing device according to any one of configurations 1 to 10, characterized in that when the target data is two-dimensional data and the range of the margin data spans multiple lines, the acquisition means acquires the first data from the data group so that the first data is distributed across all of the multiple lines.

[0260] (Configuration 12) 12. The information processing device according to any one of configurations 1 to 11, wherein in at least one layer constituting the convolutional neural network, the acquiring means acquires a part of the margin data from the data group as the first data.

[0261] (Configuration 13) 12. The information processing device according to any one of configurations 1 to 11, wherein the layer in which the acquisition means acquires a portion of the margin data from the data group as the first data is a final layer of an encoder unit of the convolutional neural network.

[0262] (Configuration 14) 14. The information processing device according to any one of configurations 1 to 13, wherein the target data is divided data obtained by dividing the inference data into predetermined unit sizes.

[0263] (Configuration 15) When the target data is the divided data, 15. The information processing apparatus according to configuration 14, wherein the acquiring means changes the position of the data group from which the first data is acquired for each of the target data.

[0264] (Configuration 16) 16. The information processing device according to any one of configurations 1 to 15, wherein the target data is image data.

[0265] (Configuration 17) An embedded device having the information processing device according to configuration 1.

[0266] (Configuration 18) An inference method using a convolutional neural network, comprising: an acquisition step of acquiring target data from input inference data; a computing step of performing a convolution operation using computation data including the target data acquired in the acquiring step and margin data, which is data other than the target data and is necessary to obtain computation result data of a predetermined size, and outputting the computation result data, An inference method in which, in the acquisition step, first data, which is a portion of the margin data, is acquired from a group of data that exists around the target data in the inference data, separate from the target data, and second data, excluding the first data, of the margin data, is not acquired from the group of data.

[0267] (Configuration 19) A program for causing a computer to execute an inference method using a convolutional neural network, an acquisition step of acquiring target data from input inference data; a computing step of performing a convolution operation using computation data including the target data acquired in the acquiring step and margin data, which is data other than the target data and is necessary to obtain computation result data of a predetermined size, and outputting the computation result data, A program in which, in the acquisition step, first data, which is a portion of the margin data, is acquired from a data group that exists around the target data in the inference data separately from the target data, and second data, excluding the first data, of the margin data is not acquired from the data group.

Claims

1. An information processing device that performs inference using a convolutional neural network, an acquisition means for acquiring target data from inference data input to the information processing device; a calculation means for performing a convolution operation using calculation data including the target data acquired by the acquisition means and margin data, which is data other than the target data and is necessary to obtain calculation result data of a predetermined size, and outputting the calculation result data; The information processing device is characterized in that the acquisition means acquires first data, which is a portion of the margin data, from a group of data in the inference data that exists around the target data separately from the target data, and does not acquire second data, excluding the first data, from the group of data.

2. 2. The information processing apparatus according to claim 1, further comprising: padding means for padding second data, which is data excluding the first data, from the margin data.

3. 3. The information processing apparatus according to claim 2, wherein the first data is stored in a storage area together with the target data, and the second data is not stored in the storage area.

4. 3. The information processing apparatus according to claim 2, wherein said padding means writes said second data directly into a register used by said calculation means for calculation.

5. 3. The information processing apparatus according to claim 2, wherein the padding means pads the second data with an arbitrary fixed value.

6. 3. The information processing apparatus according to claim 2, wherein the padding means pads the second data with data based on the target data.

7. 2. The information processing apparatus according to claim 1, wherein the acquiring means acquires the first data from the data group that exists in the same direction as a sliding direction of an operation range of the convolution operation with respect to the target data.

8. 2. The information processing apparatus according to claim 1, wherein the acquiring means acquires the first data from the data group that exists in a direction perpendicular to a sliding direction of the calculation range of the convolution calculation with respect to the target data.

9. 2. The information processing apparatus according to claim 1, wherein the acquiring means acquires the first data discretely from the data group.

10. 10. The information processing apparatus according to claim 9, wherein the acquiring means acquires the first data from the data group at predetermined data intervals.

11. The information processing device described in claim 1, characterized in that when the target data is two-dimensional data and the range of the margin data spans multiple lines, the acquisition means acquires the first data from the data group so that the first data is distributed across all of the multiple lines.

12. 2. The information processing apparatus according to claim 1, wherein in at least one layer constituting the convolutional neural network, the acquiring means acquires a part of the margin data from the data group as the first data.

13. 2. The information processing apparatus according to claim 1, wherein the layer from which the acquisition means acquires the part of the margin data as the first data from the data group is a final layer of an encoder unit of the convolutional neural network.

14. 2. The information processing apparatus according to claim 1, wherein the target data is divided data obtained by dividing the inference data into a predetermined unit size.

15. When the target data is the divided data, 15. The information processing apparatus according to claim 14, wherein the acquiring means changes a position of the data group from which the first data is acquired for each of the target data.

16. 2. The information processing apparatus according to claim 1, wherein the target data is image data.

17. An embedded device comprising the information processing device according to claim 1.

18. An inference method using a convolutional neural network, comprising: an acquisition step of acquiring target data from input inference data; a computing step of performing a convolution operation using computation data including the target data acquired in the acquiring step and margin data, which is data other than the target data and is necessary to obtain computation result data of a predetermined size, and outputting the computation result data, In the acquisition step, first data, which is a portion of the margin data, is acquired from a group of data that exists around the target data in the inference data, separate from the target data, and second data, excluding the first data, of the margin data is not acquired from the group of data.

19. A program for causing a computer to execute an inference method using a convolutional neural network, an acquisition step of acquiring target data from input inference data; a computing step of performing a convolution operation using computation data including the target data acquired in the acquiring step and margin data, which is data other than the target data and is necessary to obtain computation result data of a predetermined size, and outputting the computation result data, A program in which, in the acquisition step, first data, which is a portion of the margin data, is acquired from a data group that exists separately from the target data and around the target data in the inference data, and second data, excluding the first data, of the margin data, is not acquired from the data group.

Images