Image processing device and image processing method

Abstract

To provide a technique that reduces a computational cost for blending pixel values when combining demosaicing inference results for each pixel block in an input image.SOLUTION: A padding region is defined as a region where an error due to padding performed in demosaicing inference for a pixel block erodes from an edge of the pixel block. Pixel values are blended for an overlapping region where a first corresponding region corresponding to a padding region of an inference result block of one of neighboring pixel blocks and a second corresponding region corresponding to the padding region of the inference result block of the other pixel block overlap each other.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a demosaic inference technique for a mosaic image. [Background technology]

[0002] An image sensor used in a digital imaging device such as a digital camera is equipped with a color filter having, for example, an RGB array, and is configured to allow light of a specific wavelength to enter each pixel. Specifically, color filters having, for example, a Bayer array are often used. A captured image of the Bayer array is a so-called mosaic image in which only a pixel value corresponding to one of the RGB colors is set for each pixel. The development processing unit of the digital imaging device performs various signal processing on this mosaic image, such as demosaic processing that interpolates pixel values ​​of the remaining two colors, to generate and output a color image. As a conventional method of demosaic processing, there is a method in which a linear filter is applied to the pixel values ​​of sparse RGB colors, linear interpolation is performed on the surrounding pixel values ​​of the same color, and the RGB colors corresponding to each pixel are calculated and set. Since this method has low interpolation accuracy, many nonlinear interpolation methods have been proposed so far. However, in all of these methods, there is a problem that false colors and artifacts occur in image areas where each method is weak.

[0003] In recent years, a data-driven interpolation method that applies deep learning technology has been proposed. Non-Patent Document 1 discloses a method for training a CNN-based demosaic network. In this method, a large number of teacher images in RGB format are first acquired, and the CNN is trained using these images. Once training is complete, a mosaic image (input image) is input to the CNN to perform inference (a regression task for the input data) that converts the image into an RGB image. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Deep Joint Demosaicking and Denoising, Siggraph Asia 2016. Summary of the Invention [Problem to be solved by the invention]

[0005] In the above method, the size of the image that can be input to the GPU at one time is limited due to the memory capacity of the GPU. Therefore, when the size of the input image is large, a measure is required such as dividing the input image into multiple pixel blocks and inputting each divided pixel block to the GPU for inference. Then, after each inference result is obtained, they are combined to reconstruct the final output image.

[0006] In this case, in order to smooth the boundaries between pixel blocks, it is possible to divide the pixel blocks so that their edges overlap each other, and then mix the overlapping areas by blending them when combining them. However, there is a problem in that the calculation costs are high because the areas to be blended are large.

[0007] The present invention provides techniques for reducing the computational cost of blending pixel values ​​that is performed when combining demosaicing inference results for each pixel block in an input image. [Means for solving the problem]

[0008] One aspect of the present invention relates to a method for dividing an input image into a plurality of pixel blocks such that adjacent pixel blocks overlap each other; an acquisition means for acquiring an inference result block which is an inference result of demosaic for each of the plurality of pixel blocks; a generating means for generating a combined image by combining the inference result blocks acquired by the acquiring means such that the inference result blocks of adjacent pixel blocks overlap each other; Equipped with The generating means includes: From the edge of the pixel block , which is determined by the width according to the number of layers of the convolutional neural network used in the demosaic inference for the pixel block. A region is defined as a padding region, and pixel values ​​are blended in an overlap region where a first corresponding region corresponding to the padding region in an inference result block of one of adjacent pixel blocks overlaps with a second corresponding region corresponding to the padding region in an inference result block of the other pixel block. It is characterized by: Effect of the Invention

[0009] The configurations of the present invention can reduce the computational cost of blending pixel values ​​when combining demosaicing inference results for each pixel block in an input image. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing device 100. [Diagram 2] FIG. 1 is a diagram for explaining an outline of a first embodiment. [Diagram 3] Diagram showing network architecture. [Figure 4] FIG. 1 is a block diagram showing an example of the functional arrangement of an image processing device 100. [Diagram 5] 3 is a flowchart of a process performed by the image processing device 100. [Figure 6] FIG. 2 is a block diagram showing an example of the functional configuration of the system. [Figure 7] 4 is a flowchart showing the operation of the system. [Figure 8] FIG. [Figure 9] FIG. 11 is a diagram showing an example of setting an overlap width. [Figure 10] FIG. 13 is a diagram showing an example of dividing an input image. [Figure 11] FIG. 1 is a diagram for explaining a convolution operation. [Figure 12] FIG. 11 is a diagram showing an example of a method for determining an alpha map. [Figure 13] FIG. 13 is a diagram for explaining a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] [First embodiment] In this embodiment, as shown in Fig. 2, an input image 201 is divided to generate a plurality of pixel blocks 202. The input image 201 is a so-called mosaic image in which only a pixel value corresponding to one of the RGB colors is set for each pixel, and is, for example, a well-known RAW image. Then, a plurality of "RGB format inference result blocks 203" are generated by performing demosaic inference on each of the plurality of pixel blocks 202. Then, each inference result block 203 is combined to generate a combined image 204 of the same size as the input image 201, and during the combination, an area for "blending pixel values" is defined, and "blending pixel values" is performed only on the defined area.

[0013] (Configuration of image processing device) First, a hardware configuration example of an image processing device 100 according to this embodiment will be described with reference to the block diagram of Fig. 1. The image processing device 100 according to this embodiment can be applied to computer devices such as a PC (personal computer), a tablet terminal device, and a smartphone.

[0014] The CPU 101 executes various processes using computer programs and data stored in the RAM 102 and the ROM 103. As a result, the CPU 101 controls the operation of the entire image processing device 100, and executes or controls each process that will be described as being performed by the image processing device 100.

[0015] The RAM 102 has an area for storing computer programs and data loaded from the ROM 103, the secondary storage device 104, the external storage device 108, etc., and an area for storing information such as an input image (RAW image) output from the imaging device 111. The RAM 102 also has a work area used when the CPU 101 and the GPU 110 execute various processes. In this way, the RAM 102 can provide various areas as appropriate.

[0016] The ROM 103 stores setting data for the image processing device 100, computer programs and data related to the startup of the image processing device 100, computer programs and data related to the basic operations of the image processing device 100, and the like.

[0017] The secondary storage device 104 is a non-volatile memory such as a hard disk drive. The secondary storage device 104 stores an OS (operating system), computer programs and data for causing the CPU 101 and the GPU 110 to execute or control various processes described as being performed by the image processing device 100, and the like. The computer programs and data stored in the secondary storage device 104 are loaded into the RAM 102 as appropriate under the control of the CPU 101, and become targets for processing by the CPU 101 and the GPU 110. Note that the secondary storage device 104 can be various storage devices such as a hard disk drive, an optical disk drive, a flash memory, and the like.

[0018] The GPU 110 operates based on computer programs and data loaded into the RAM 102, thereby performing various arithmetic operations on data received from the CPU 101, and notifies the CPU 101 of the results of the operations.

[0019] The imaging device 111 has an imaging element equipped with a color filter having an array such as a Bayer array, and outputs a RAW image output from the imaging element to the system bus 107.

[0020] The input interface 105 is a serial bus interface such as USB or IEEE 1394. The image processing apparatus 100 acquires data, commands, and the like from the outside via the input interface 105.

[0021] The output interface 106 is a serial bus interface such as USB or IEEE1394, similar to the input interface 105. The output interface 106 may be a video output terminal such as DVI or HDMI (registered trademark). The image processing device 100 outputs data and the like to the outside via the output interface 106.

[0022] The CPU 101 , RAM 102 , ROM 103 , secondary storage device 104 , GPU 110 , imaging device 111 , input interface 105 , and output interface 106 are all connected to a system bus 107 .

[0023] The operation unit 112 is a user interface such as a keyboard, a mouse, a touch panel, etc., and the user can input various instructions to the CPU 101 via the input interface 105 by operating it.

[0024] The external storage device 108 is a memory device, such as a hard disk drive, a memory card, a CF card, an SD card, or a USB memory, that is connected to / attached to the image processing device 100. Computer programs and data read from the external storage device 108 are input to the image processing device 100 via an input interface 105 and stored in the RAM 102 or the secondary storage device 104. In addition, the computer programs and data to be saved in the external storage device 108 are written to the external storage device 108 via an output interface 106.

[0025] The display device 109 has a liquid crystal screen or a touch panel screen, and displays images, text, etc., as the results of processing by the CPU 101 and the GPU 110. The display device 109 may also be a projection device such as a projector that projects images and text.

[0026] Note that the configuration shown in Fig. 1 is an example of a device configuration capable of implementing each process described below, and the configuration capable of implementing each process described below is not limited to the configuration shown in Fig. 1. For example, in Fig. 1, the imaging device 111 is incorporated in the image processing device 100 as a device built into the image processing device 100. However, this is not limiting, and for example, such an imaging device 111 may be connected to the input interface 105 as an external device of the image processing device 100.

[0027] In this embodiment, the image processing device 100 operates as follows by executing an image processing application. That is, the image processing device 100 divides an input image (RAW image) output from the imaging device 111 to generate a plurality of pixel blocks, and generates a plurality of "RGB format inference result blocks 203" by performing demosaic inference on each of the plurality of pixel blocks. The image processing device 100 then combines each of the inference result blocks 203 to generate a "combined image 204 of the same size as the input image 201," and during the combination, it determines an area in which "pixel value blending" is to be performed, and performs "pixel value blending" only on the determined area.

[0028] (About CNN) In this embodiment, demosaic inference for pixel blocks is performed using a Convolutional Neural Network (CNN). Here, a CNN that is used in general image processing techniques that apply deep learning techniques, such as Non-Patent Document 1, will be described.

[0029] CNN is a learning-type image processing technology that repeatedly performs nonlinear calculations after convolving an image with a filter generated through training. The filter is also called a local receptive field (LRF). The image obtained by convolving the filter with an image and then performing nonlinear calculations is called a feature map. Furthermore, learning is performed using training images (or data sets) consisting of pairs of input and output images. Simply put, learning is the process of generating filter values ​​from training data that can convert an input image into a corresponding output image with high accuracy. More details on this will be provided later.

[0030] When an image has RGB color channels or when a feature map is composed of multiple images, the filter used for convolution also has multiple channels accordingly. In other words, a convolution filter is expressed as a four-dimensional array that includes the number of channels in addition to the vertical and horizontal sizes and number of images. The process of performing nonlinear operations after convolving a filter with an image (or feature map) is expressed in units of layers. For example, it is called the nth layer feature map or the nth layer filter. Also, for example, a CNN that repeats filter convolution and nonlinear operations three times is said to have a three-layer network structure. This process can be formulated as shown in the following equation (1).

[0031]

number

[0032] In formula (1), Wn is the nth layer filter, bn is the nth layer bias, G is the nonlinear operator, Xn is the nth layer feature map, and * is the convolution operator. Note that the (l) on the right indicates that this is the lth filter or feature map. The filters and biases are generated by learning, which will be described later, and are collectively called network parameters. For example, a sigmoid function or ReLU (Rectified Linear Unit) is used for the nonlinear operation. ReLU is given by the following formula (2).

[0033]

number

[0034] In other words, it is a nonlinear process in which negative elements of the input vector X are set to zero and positive elements are left as they are. Next, we will explain CNN learning. CNN learning is performed by minimizing the objective function, generally expressed by the following formula (3), for learning data consisting of a pair of input learning images (student images) and corresponding output learning images (teacher images).

[0035]

number

[0036] Here, L is a loss function that measures the error between the correct answer and its estimation. Yi is the i-th output training image, and Xi is the i-th input training image. F is a function that collectively expresses the equation (1) performed in each layer of the CNN. θ is the network parameter (filter and bias).

[0037]

number

[0038] is the L2 norm, or more simply, the square root of the sum of the squares of the elements of vector Z. Additionally, n is the total number of training data used for learning. However, since the total number of training data is generally large, in Stochastic Gradient Descent (SGD), a portion of the training images are randomly selected and used for learning. This reduces the computational load when learning using a large amount of training data. Additionally, various methods are known for minimizing (optimizing) objective functions, such as the momentum method, AdaGrad method, AdaDelta method, and Adam method. The Adam method is given by the following equation (4).

[0039]

number

[0040] In equation (4), θi t is the i-th network parameter at iteration t, and g is θi t is the gradient of the loss function L with respect to . Additionally, m and v are moment vectors, α is the base learning rate, β1 and β2 are hyperparameters, and ε is a small constant. Note that since there are no guidelines for selecting an optimization method in learning, essentially any method can be used, but it is known that differences in convergence between methods will result in differences in learning time.

[0041] Well-known examples of networks using CNN include ResNet in the field of image recognition and its application RED-Net in the field of super-resolution. In both cases, CNNs are multi-layered and filter convolutions are performed multiple times to increase the accuracy of processing. For example, ResNet features a network structure with paths that shortcut the convolutional layer, which allows for a multi-layered network with 152 layers, achieving highly accurate recognition approaching the recognition rate of humans. The reason that multi-layered CNNs increase the accuracy of processing is, simply put, because they can express nonlinear relationships between input and output by repeating nonlinear calculations multiple times.

[0042] The CNN according to this embodiment is a trained CNN that has been trained to output, when a pixel block is input, an inference result (inference result block) of demosaic for the pixel block.

[0043] (Example of functional configuration of image processing device) An example of the functional configuration of the image processing device 100 is shown in the block diagram of Fig. 4. The process performed by the image processing device 100 to generate a combined image that is an inference result of demosaic on an input image will be described with reference to the flowchart of Fig. 5.

[0044] 4 will be described below as the subject of processing, but in reality, the functions of the functional units are realized by the CPU 101 or the GPU 110 executing a computer program for causing the CPU 101 or the GPU 110 to execute the functions of the functional units. Note that one or more of the functional units shown in FIG. 4 may be implemented in hardware.

[0045] In step S501, the input image acquisition unit 401 acquires an input image to be subjected to demosaic processing. The method of acquiring the input image is not limited to a specific method.

[0046] For example, the input image acquisition unit 401 may control the imaging device 111 and acquire a RAW image captured by the imaging device 111 as an input image through the control. Also, for example, the input image acquisition unit 401 may acquire a RAW image stored in the secondary storage device 104 as an input image, or may acquire a RAW image stored in the external storage device 108 as an input image via the input interface 105. Also, when the image processing device 100 is connected to a network (a wired / wireless network such as the Internet or a LAN), the input image acquisition unit 401 may acquire a RAW image as an input image from an external device via the network.

[0047] Also, here, the RAW image is assumed to be a RAW image captured by an imaging device that employs a Bayer array as a color filter array, but the color filter array is not limited to the Bayer array and other color filter arrays such as X-Trans may be used.

[0048] In step S502, the inference environment acquisition unit 402 acquires inference environment information, which is information necessary for performing demosaic inference on an image. The inference environment information includes two items: a "model used for inference" and a "memory amount of the GPU 110 used for inference."

[0049] A "model used for inference" is a combination of "CNN network parameters" and "CNN network architecture." "CNN network parameters" are parameters that define CNN, including parameters such as weight coefficients and bias values. "CNN network architecture" is information that indicates the structure of CNN, such as what layers are connected and in what order to configure CNN.

[0050] The "amount of memory of the GPU 110 used for inference" refers to the amount of memory available to the GPU 110 in memories such as the RAM 102, the secondary storage device 104, the external storage device 108, and the internal memory of the GPU 110.

[0051] The method of acquiring the inference environment information is not limited to a specific method. For example, the inference environment acquisition unit 402 may acquire the analogy environment information stored in the secondary storage device 104, or may acquire the inference environment information stored in the external storage device 108 via the input interface 105. For example, the inference environment acquisition unit 402 may acquire the inference environment information input by the user operating the operation unit 112. In addition, when the image processing device 100 is connected to a network (a wired / wireless network such as the Internet or a LAN), the inference environment acquisition unit 402 may acquire the inference environment information from an external device via the network.

[0052] In step S503, the width calculation unit 403 calculates the width of the area (padding area) where the error due to padding performed in the demosaic inference for the pixel block encroaches on the edge of the pixel block (the width (padding width) where the effect of the padding encroaches on the edge of the pixel block), based on the "CNN network architecture" included in the inference environment information.

[0053] CNN consists of a stack of convolution filters, each of which applies a convolution operation to a feature map. An example is shown in Figure 11. For simplicity, the size of the convolution filter is 3x3.

[0054] 11(a) shows a scene where a convolution filter is applied to a 3x3 region 1102 in a feature map (image) 1101. The values ​​displayed in the region 1102 are feature values ​​(input values), and in the 3x3 convolution filter, the input values ​​are multiplied by the filter coefficients corresponding to the input values, and the sum of the multiplication results is obtained as the filtering result.

[0055] FIG. 11(b) shows a scene where this convolution filter is applied to the pixel in the upper right corner of the feature map 1101. In this case, the region 1102 to which the convolution filter is applied includes the outside of the feature map 1101 (part where no input value exists). Therefore, when applying a convolution filter to such a region 1102, the outside pixels are generally padded with appropriate values ​​before applying the convolution filter to the region 1102. A representative padding method is zero padding, which pads missing values ​​(values ​​in parts where no input value exists) with 0. A convolution filter is applied to the padded region, but the filtering result is a value different from the original value due to the mixing of 0 with the input value. A similar phenomenon occurs in all pixels on the outer periphery (edge) of the feature map, and an error is superimposed on the filtering result (output value).

[0056] When applying the convolution filter of the next layer to the output feature map, the filtering result with error (rectangle marked with "*") is used as the input value of the convolution, as shown in Figure 11(c). Therefore, the filtering result for the pixel at a distance of "2" from the edge of the image (a distance of two pixels from the edge of the image toward the center of the image) also has error superimposed on it. If this is repeated, the error will erode one pixel from the edge of the image for each convolution layer. For example, in the case of a CNN consisting of 50 3x3 convolution layers, an error due to padding will occur in the area within 50 pixels from the edge of the image in the inference result image. The width of the area eroded by the error due to padding is called the padding width (in this example, the padding width = 50). For example, if the padding area is located along the top edge of the image, the number of pixels in the padding area in the direction from the top edge of the image toward the center of the image is the "padding width of the padding area".

[0057] The padding width is a unique value for each network architecture, and is calculated by the width calculation unit 403. As an example, a method for calculating the padding width for a network in Non-Patent Document 1 will be shown.

[0058] The network architecture of Non-Patent Document 1 is as shown in FIG. 3. First, the pixel block 202 is converted into a missing image 306 with three channels (R channel, G channel, and B channel). The R channel of the missing image contains only pixels of the R component of the student image, and the pixel values ​​of the other pixels are set to missing values ​​(0). The G channel of the missing image contains only pixels of the G component of the student image, and the pixel values ​​of the other pixels are set to missing values ​​(0). The B channel of the missing image contains only pixels of the B component of the student image, and the pixel values ​​of the other pixels are set to missing values ​​(0). Note that the missing values ​​may be interpolated by a method such as bilinear interpolation. Next, the spatial resolution of the missing image 306 is reduced by half for each RGB channel in the subsampling layer 301. Next, after passing through the convolution layer 302, the size of the feature map is doubled in the upsampling layer 303 to have the same resolution as the pixel block 202. The obtained feature map is connected to the pixel block 202 itself in the channel direction in the connection layer 304, and then passed through the convolution layer 305 to output the inference result block 203 corresponding to each channel.

[0059] If a 3x3 convolution filter is applied to an area where the feature map has half the spatial resolution, the error due to padding will erode 2 pixels per layer at the scale of the input block. Also, if the convolution filter is larger, such as 5x5, the number of pixels erode will be larger. If the filter size of the i-th layer in a CNN is li, and the resolution of the feature map at that position is 1 / ki of the input block, then the padding width p for that CNN is calculated as follows:

[0060]

number

[0061] It should be noted that, if the width by which the effect of padding on a pixel block encroaches on an end of the pixel block can be calculated as the "padding width," the method for calculating the padding width is not limited to the above method.

[0062] In step S504, the division unit 404 divides the input image acquired in step S501 into a plurality of pixel blocks so that adjacent pixel blocks overlap each other, based on the padding width p calculated in step S503 and the inference environment information acquired in step S502. In order to perform such division, it is necessary to determine the size of the pixel blocks and the overlap width between the pixel blocks.

[0063] Regarding the size of the pixel blocks, the division unit 404 refers to the "amount of memory of GPU 110 used for inference" contained in the inference environment information, and sets the size to the maximum size (the maximum size that does not cause memory overflow) within the range in which all pixel blocks after division will fit into the memory used by GPU 110.

[0064] The division unit 404 sets the overlap width based on the padding width p obtained in step S503, as shown in FIG. 8. In the example of FIG. 8, the area within the padding width 802 (p pixels) from the end of the pixel block 801 is the padding area 803. On the other hand, in the example of FIG. 8, the area not affected by padding (the remaining area in the pixel block 801 excluding the padding area 803) is the non-padding area 804. Since the non-padding area is always not affected by padding errors, it is guaranteed that the inference result does not depend on the block division method. Using this property, the division unit 404 sets the overlap width so that the non-padding area 911 in the pixel block 901 and the non-padding area 912 in the pixel block 902 adjacent to the pixel block 901 are adjacent to each other, as shown in FIG. 9(a). That is, the division unit 404 sets the overlap width (overlapping width)=2p. As described above, the inference result for the non-padding area does not depend on the block division method, so the inference result for the joints between the non-padding areas is smooth.

[0065] The result of dividing an input image into a plurality of pixel blocks with such settings is shown in Fig. 10. In Fig. 10, an input image 201 is divided into six pixel blocks, and the non-padding areas of adjacent pixel blocks are adjacent to each other. Note that the block division method is not limited to the above method. For example, as shown in Fig. 9(b), the overlap width may be set so that a non-padding area 911 and a non-padding area 912 are spaced apart by a fixed distance (fixed number of pixels).

[0066] In step S505, the calculation unit 405 acquires position information of the area (blending area) where pixel values ​​are blended, using the padding width p obtained in step S503. In the following, for each pixel block, an "inference result block indicating the inference result of demosaic for the pixel block" of the same size as the pixel block is generated. Then, each inference result block is overlapped with an overlap width = 2p between adjacent inference result blocks, and arranged and combined according to the order of corresponding pixel blocks to generate a "combined image of the same size as the input image." In such a combined image, pixel value blending is not required for areas corresponding to non-padding areas (non-blending area). On the other hand, in such a combined image, "corresponding areas corresponding to overlap areas where the padding area of ​​one pixel block overlaps with the padding area of ​​the other pixel block in adjacent pixel blocks" are blending areas. Therefore, "blending of pixel values" is performed using the pixel values ​​of the inference result block of the one pixel block and the pixel values ​​of the inference result block of the other pixel block to determine the pixel values ​​of the blending area.

[0067] Therefore, in step S505, the calculation unit 405 determines, in adjacent pixel blocks, an overlapping region where a padding region in one pixel block (a region defined by a padding width p from an end of the one pixel block) overlaps with a padding region in the other pixel block (a region defined by a padding width p from an end of the other pixel block), as a blending region, and specifies position information of the blending region. The "position information of the blending region" may be, for example, image coordinates of the four corners of the blending region, image coordinates of each of the opposing corners of the blending region, or a set including image coordinates at the center of the blending region and the vertical and horizontal sizes of the blending region.

[0068] In step S506, the inference unit 406 inputs each pixel block obtained by the division in step S504 as an input block to the CNN defined by the "model used for inference" included in the inference environment information. The inference unit 406 then obtains the output of the CNN to which the pixel blocks have been input as an "inference result block that is the inference result of demosaic for the pixel block." As an example, the inference result block is obtained by the method described above with reference to FIG. 3.

[0069] In step S507, the map calculation unit 407 obtains a map of alpha values ​​(alpha map) used for blending pixel values ​​of each inference result block that overlaps in a blending area in the combined image (an area in the combined image specified by the above-mentioned "position information of the blending area"). An example of a method for obtaining an alpha map will be described with reference to FIG. 12.

[0070] As one example, a sloped alpha map 1201 is calculated such that the alpha value in the blending region 1001 of the left inferred result block of the inferred result blocks that overlap in the blending region 1001 decreases from the left end to the right end. This alpha map 1201 represents the alpha value at each horizontal position in the blending region 1001 of the left inferred result block. Whiter parts represent higher alpha values, and darker parts represent lower alpha values.

[0071] The horizontal axis of graph 1202 indicates the horizontal position in blending region 1001 of the inference result block on the left side, and the vertical axis indicates the alpha value at that position, which shows that the alpha value decreases linearly from the left end to the right end. Note that the decrease in alpha value does not have to be linear, nor does it have to decrease monotonically. Note that the decrease direction is not limited to from the left end to the right end, and may be from the right end to the left end. In other words, the alpha map is configured so that the alpha value decreases from one end of the alpha map (blend region) to the other end.

[0072] The alpha value may also be determined based on the inference error of the pixel. For example, the inference error ε (difference between pixel values) = |q-q'| at the pixel at position (x, y) is calculated using the pixel value q of the R pixel at position (x, y) in the pixel block and the pixel value q' of the R pixel at position (x, y) in the inference result block of the pixel block (within the inference result block). Then, based on the calculated inference error, the alpha value α(x, y) at position (x, y) is calculated as follows:

[0073]

number

[0074] In equation (6), the larger the inference error ε, the lower the reliability of the pixel is set (=the degree to which the value of the pixel is prioritized during blending is reduced, i.e., the alpha value is set to a smaller value), and this setting is realized by setting the inverse of the inference error as the alpha value. Also, normalization is performed to keep the alpha value within the range of [0,1]. Note that in equation (6), the inverse of the absolute value of the difference between q and q' is calculated as the alpha value, but the alpha value may be calculated using another equation. In other words, any method may be used as long as it is a method of calculating an alpha value corresponding to a pixel based on the difference between the pixel value of the pixel in a pixel block and the pixel value of the pixel in an inference result block of the pixel block, and the method of calculating an alpha value by a calculation such that the alpha value becomes smaller as the difference becomes larger.

[0075] Here, equation (6) is shown on the assumption that the pixel at position (x, y) is an R channel pixel, but this equation can also be applied when it is a G channel pixel or a B channel pixel. As a result, an alpha map 1203 is obtained, and the alpha value does not monotonically decrease from the left end to the right end in the blending region, as shown in graph 1204.

[0076] Although the method of determining the alpha map has been described by using a map with a given slope and calculating the alpha map based on the inference error of the pixel, other methods may be used.Also, a method may be used in which alpha maps determined by a plurality of methods are combined by weighted addition or the like to determine the final alpha map.

[0077] In step S508, the combination unit 408 combines each of the inference result blocks in the order of the corresponding pixel blocks, while overlapping adjacent inference result blocks with an overlap width of 2p (combining them to have a configuration similar to that of Figure 10), thereby generating a "combined image of the same size as the input image."

[0078] At this time, the combining unit 408 determines (obtains) the pixel value of the blending area in the combined image, which is defined by the "position information of the blending area", by blending the pixel values ​​of the inference result blocks that overlap in the blending area using the above-mentioned alpha map. More specifically, the combining unit 408 determines the pixel value of the pixel position (x, y) in the combined image based on the following conditions.

[0079] When a pixel position (x, y) in the combined image is a pixel position included in a single inference result block, the combining unit 408 sets the pixel value at the pixel position (x, y) of the single inference result block as the pixel value of the pixel position (x, y) in the combined image. For example, when the pixel position (x, y) in the combined image corresponds to pixel position 1002 in Fig. 10, the pixel value of the inference result block obtained for pixel position 1002 is set to the pixel position (x, y) in the combined image.

[0080] When a pixel position (x, y) in the combined image is a pixel position in an overlapping area where a corresponding area A corresponding to a non-padding area in one inference result block and a corresponding area B corresponding to a padding area in the other inference result block overlap, the combining unit 408 sets the pixel value at the pixel position (x, y) in the corresponding area A as the pixel value of the pixel position (x, y) in the combined image. For example, when the pixel position (x, y) in the combined image corresponds to pixel position 1003 in Fig. 10, the pixel value of the non-padding area at pixel position 1003 is set to pixel position (x, y) in the combined image.

[0081] When a pixel position (x, y) in the combined image is a pixel position in an overlapping area where a corresponding area C corresponding to a padding area in one inference result block and a corresponding area D corresponding to a padding area in the other inference result block overlap each other, the combining unit 408 sets the pixel value of the pixel position (x, y) in the combined image to a pixel value q* that is a blending result obtained by blending a pixel value q1 of the pixel position (x, y) in the corresponding area C and a pixel value q2 of the pixel position (x, y) in the corresponding area D according to an alpha value corresponding to the pixel position (x, y). An example of this blending is shown in the following formula (7).

[0082]

number

[0083] The combining unit 408 then outputs the combined image thus generated as a "final demosaic result for the input images." The output destination of the combined image is not limited to a specific output destination. For example, the combining unit 408 may store the combined image in the secondary storage device 104, may store the combined image in the external storage device 108 via the input interface 105, or may display the combined image on the display device 109 via the output interface 106. In addition, when the image processing device 100 is connected to the above-mentioned network, the combining unit 408 may transmit the combined image to an external device via the network.

[0084] The above-mentioned processing in steps S501 to S508 is performed for all input images for which demosaic inference is to be performed, and when demosaic inference has been performed for all input images, the processing according to the flowchart in FIG. 5 ends.

[0085] In this manner, according to the present embodiment, when an input image is divided into a plurality of pixel blocks, and an inference result block is generated for each pixel block and combined to generate a combined image, pixel values ​​are blended only in areas where padding areas overlap, thereby reducing the calculation cost for obtaining the demosaic result of the input image.

[0086] [Second embodiment] In the following embodiments and modifications including this embodiment, the differences from the first embodiment will be described, and unless otherwise specified below, they will be the same as the first embodiment. In this embodiment, an example will be described in which a plurality of input images are sequentially demosaiced under the condition that the GPU used for inference is predetermined and the size of the input images is constant.

[0087] An example of the functional configuration of the system according to this embodiment is shown in the block diagram of Fig. 6. As shown in Fig. 6, the system according to this embodiment has a calculation unit 601 and an image processing device 690, and the calculation unit 601 and the image processing device 690 are connected via a network such as a LAN or the Internet. This connection may be wired or wireless, and the connection form is not limited to a specific connection form.

[0088] In this embodiment, the calculation unit 601 and the image processing device 690 will be described as being computer devices having the hardware configuration shown in Fig. 1. However, the present invention is not limited to this, and the calculation unit 601 and the image processing device 690 may adopt a hardware configuration other than the hardware configuration shown in Fig. 1. Furthermore, the calculation unit 601 and the image processing device 690 may adopt different hardware configurations.

[0089] The operation of the system according to this embodiment (operation for acquiring the result of demosaic for an input image) will be described with reference to the flowchart of Fig. 7. The process in step S701 is a process performed by the calculation unit 601.

[0090] In step S701, the calculation unit 601 acquires system information. The system information includes the above-mentioned inference environment information, padding width p, and position information of the blending region, which are invariant if the memory amount of the GPU 110 and the size of the input image are fixed. Therefore, in this embodiment, as a preprocessing for inference, only the system information is calculated and acquired in a lump. Furthermore, the calculation unit 601 acquires the size of the input image and transmits the size of the input image to the image processing device 690. The system information and the size of the input image transmitted to the image processing device 690 are stored in the storage unit 602 in the image processing device 690. In the following, it is assumed that the size of the input image is included in the system information and stored in the storage unit 602.

[0091] Then, the processes of steps S501 to S508 are performed as described above in the image processing device 690. If the memory capacity of the GPU 110 to be used, the size of the input image, or the like has changed, the process proceeds to step S701, where system information and the size of the input image are collected again. On the other hand, if the memory capacity of the GPU 110 to be used, the size of the input image, or the like has not changed, and if there are input images remaining for which demosaic inference should be performed, the process proceeds to step S501, and if there are no input images remaining, the process according to the flowchart in FIG. 7 ends.

[0092] This embodiment is also applicable when the memory amount of the GPU 110 and the size of the input image are not uniquely determined in advance. In that case, the calculation unit 601 lists a number of combinations of the memory amount of the GPU 110 and the size of the input image, calculates system information for various cases, and registers the system information in the storage unit 602. The image processing device 690 then refers to the memory amount of the GPU 110 and the size of the input image at the time of processing the input image, reads out the corresponding system information from the storage unit 602, and uses the system information.

[0093] In addition, either the memory amount of the GPU 110 or the size of the input image may not be fixed. In this case, the calculation unit 601 outputs only the information that can be calculated from the fixed information among the system information. The storage unit 602 calculates and outputs the remaining information among the system information as needed.

[0094] In this way, according to this embodiment, fixed information is used without recalculation unless the situation changes, thereby eliminating the need to recalculate unnecessary information and further reducing the computational costs associated with demosaicing for each input image.

[0095] [Third embodiment] In this embodiment, an example of reducing discontinuity of alpha values ​​at the end of a blending region with respect to the alpha map generated in step S507 will be described. FIG. 13(a) is an enlarged view of the periphery of a blending region 1001. The blending region 1001 is adjacent to a left block 1301 and a right block 1302. FIG. 13(b) is a graph showing alpha values ​​on a straight line 1300 sliced ​​at a certain height y=y0 for these regions. The alpha values ​​in the blending region 1001 are as shown in alpha map 1303, which indicates that the closer the alpha value is to 1, the more priority the left block 1301 has during blending, and the closer the alpha value is to 0, the more priority the right block 1302 has.

[0096] Here, no blending is performed in the left region 1304, and only the inference result of the left block 1301 is used. This can be expressed as setting the alpha value to α=1 in the left region 1304. Similarly, it can be said that α=0 in the right region 1305. In such a case, the alpha value may change suddenly and discontinuously at the left edge 1306 and right edge 1307 of the blending region 1001. This may result in unnatural edges in the combined image, which may be perceived by the human eye as artifacts.

[0097] To deal with this, processing is performed so that the alpha values ​​are continuous at the ends of the blending region. If the blending region is in the range of 0≦x≦xmax (x is the horizontal pixel position in the input image / combined image), the alpha value is corrected for an area of ​​width Δ from the end of the blending region. As a result, based on α(x,y) which is the alpha map 1303, α'(x,y) is calculated as follows, which is the corrected alpha map 1308.

[0098]

number

[0099] The correction of formula (8a) is applied to the left end of the blending region, and the correction of formula (8b) is applied to the right end. In either formula, the alpha value of the end in the alpha map is used to correct the alpha value of the partial region including that end. As a result, as shown in FIG. 13(c), the alpha value does not change suddenly at the end of the blending region 1001, resulting in an alpha map 1308 in which the alpha value is continuously connected. Note that the formula for this correction is not limited to the above formula, and other correction methods can also be applied.

[0100] <Modification> The configurations shown in Fig. 4 and Fig. 6 can be modified / altered as appropriate. For example, one functional unit may be divided into multiple functional units according to function, or two or more functional units may be integrated into one functional unit. The configuration shown in Fig. 4 may be configured by two or more devices, and the configuration shown in Fig. 6 may be configured by three or more devices. In that case, the devices are connected via a circuit or a wired or wireless network, and perform data communication with each other to perform cooperative operations, thereby realizing the processing described in each of the above embodiments.

[0101] Furthermore, the numerical values, processing timing, processing order, processing subject, data (information) structure / destination / source / storage location, etc. used in each of the above embodiments and variant examples are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0102] In addition, some or all of the embodiments and modified examples described above may be used in appropriate combination. In addition, some or all of the embodiments and modified examples described above may be used selectively.

[0103] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0104] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0105] 401: Input image acquisition unit 402: Inference environment acquisition unit 403: Width calculation unit 404: Division unit 405: Calculation unit 406: Inference unit 407: Map calculation unit 408: Integration unit

Claims

1. a dividing means for dividing an input image into a plurality of pixel blocks such that adjacent pixel blocks overlap each other; an acquisition means for acquiring an inference result block which is an inference result of demosaic for each of the plurality of pixel blocks; a generating means for generating a combined image by combining the inference result blocks acquired by the acquiring means such that the inference result blocks of adjacent pixel blocks overlap each other; Equipped with The generating means includes: A region determined from an end of a pixel block by a width corresponding to the number of layers of a convolutional neural network used in demosaic inference for the pixel block is defined as a padding region, and pixel values ​​are blended in an overlapping region where a first corresponding region corresponding to the padding region in an inference result block of one of adjacent pixel blocks overlaps with a second corresponding region corresponding to the padding region in an inference result block of the other pixel block.

13. An image processing device comprising:

2. 2 . The image processing device according to claim 1 , wherein the dividing means divides the input image into a plurality of pixel blocks such that adjacent pixel blocks overlap each other with an overlap width corresponding to a width of the padding area.

3. 3. The image processing apparatus according to claim 1, wherein the dividing means divides the input image into a plurality of pixel blocks such that non-padding areas of adjacent pixel blocks are adjacent to each other.

4. 3. The image processing apparatus according to claim 1, wherein the dividing means divides the input image into a plurality of pixel blocks such that the non-padding areas in adjacent pixel blocks are spaced apart by a fixed distance.

5. 5. The image processing device according to claim 1, wherein the acquisition means acquires an inference result block of the pixel block output from the convolutional neural network by inputting each of the plurality of pixel blocks to the convolutional neural network.

6. 6. The image processing apparatus according to claim 1, wherein the generating means determines pixel values ​​of the overlapping region by blending pixel values ​​in the first corresponding region and pixel values ​​in the second corresponding region.

7. 7. The image processing apparatus according to claim 6, wherein the generating means performs the blending using an alpha map that represents an alpha value at each position in the overlapping region.

8. 8. The image processing apparatus according to claim 7, wherein the alpha map represents a decrease in alpha value from one end of the overlap region to the other end.

9. 9. The image processing device according to claim 7, wherein the generating means determines an alpha value corresponding to the pixel based on a difference between a pixel value of the pixel in the pixel block and a pixel value of the pixel in an inference result block of the pixel block.

10. 10. The image processing apparatus according to claim 9, wherein the generating means determines the alpha value by a calculation such that the alpha value becomes smaller as the difference increases.

11. 11. The image processing apparatus according to claim 10, wherein the generating means obtains an inverse of the difference as the alpha value.

12. moreover, 12. The image processing apparatus according to claim 7, wherein an alpha value of an end portion in the alpha map is used to correct an alpha value of a partial region including the end portion.

13. 13. The image processing device according to claim 1, wherein the generating means sets pixel values ​​of a corresponding area corresponding to a padding area in the inference result block of one of adjacent pixel blocks and a corresponding area corresponding to a non-padding area in the inference result block of the other pixel block in an overlapping area of ​​the corresponding area corresponding to a non-padding area in the inference result block of the other pixel block.

14. 14. The image processing apparatus according to claim 1, wherein the generating means sets pixel values ​​in a single inference result block to pixel values ​​in the single inference result block for pixel values ​​in an area included in the single inference result block.

15. 15. The image processing apparatus according to claim 1, wherein the input image is a RAW image.

16. An image processing method performed by an image processing device, comprising: a division step in which a division means of the image processing device divides an input image into a plurality of pixel blocks such that adjacent pixel blocks overlap each other; an acquisition step in which an acquisition means of the image processing device acquires, for each of the plurality of pixel blocks, an inference result block which is an inference result of demosaic for the pixel block; a generating step in which a generating means of the image processing device generates a combined image by combining the inference result blocks acquired in the acquiring step so that the inference result blocks of adjacent pixel blocks overlap each other; Equipped with In the generating step, A region determined from an end of a pixel block by a width corresponding to the number of layers of a convolutional neural network used in demosaic inference for the pixel block is defined as a padding region, and pixel values ​​are blended in an overlapping region where a first corresponding region corresponding to the padding region in an inference result block of one of adjacent pixel blocks overlaps with a second corresponding region corresponding to the padding region in an inference result block of the other pixel block.

13. An image processing method comprising:

17. A computer program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 15.

Images