Information processing device and its control method

Abstract

Improve the performance of the demosaicing processing unit. [Solution] The information processing device includes: acquisition means for acquiring a first mosaic image and a second mosaic image generated based on the same original image; estimation means for inputting the first mosaic image and the second mosaic image to a demosaicing processing unit and generating a first estimated image and a second estimated image with pixel value interpolation; calculation means for calculating a first reconstruction error in the first estimated image and a second reconstruction error in the second estimated image; and update means for updating the parameters of the demosaicing processing unit based on the first and second reconstruction errors. The first mosaic image is a mosaic image in which the thinning pattern for generating the mosaic image and the original image are in a first positional relationship, and the second mosaic image is a mosaic image in which the thinning pattern and the original image are in a second positional relationship.

Description

[Technical Field]

[0001] This invention relates to image processing technology using machine learning. [Background technology]

[0002] In digital cameras, an image of a subject, formed by an optical system such as a lens, is captured as image information by measuring the amount of light using an image sensor with multiple pixels arranged spatially. Generally, the pixels of an image sensor themselves cannot distinguish colors, so a color filter with different colors arranged in a mosaic pattern is used to configure each pixel to transmit light of a specific color. For example, by using a color filter with three colors, red (R), green (G), and blue (B), arranged in a mosaic pattern, a mosaic image is obtained in which each pixel corresponds to one of the RGB colors. The camera's development processing unit performs signal processing on each pixel of this mosaic image, including demosaicing to interpolate the pixel values ​​of the remaining two missing colors (missing pixel values), to generate and output a color image (RGB image).

[0003] Generally, demosaicing calculates the pixel values ​​of the remaining two missing colors using rule-based interpolation with the pixel values ​​of surrounding pixels (1 or more). However, rule-based interpolation has the challenge of difficulty in reproducing high spatial frequency components. In recent years, machine learning-based demosaicing, such as neural networks (NNs) as exemplified in Non-Patent Document 1, has been studied. In machine learning-based demosaicing, inference is performed to convert a mosaic image to an RGB image using a pre-trained model that has been trained on a large number of RGB images. Compared to rule-based methods, machine learning-based methods have improved performance in reproducing high spatial frequency components and can estimate high-quality color images. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Gharbi et al., "Deep Joint Demosaicking and Denoising", SIGGRAPH Asia 2016 [Overview of the project] [Problems that the invention aims to solve]

[0005] When shooting with the camera tilted 90 degrees, if the subject is stationary and the same, the demosaicing result should ideally simply be a 90-degree rotation, resulting in the same color and image quality. However, machine learning-based methods have the challenge of easily changing the demosaicing result (color and image quality) due to the asymmetry in the arrangement of each color in the color filter. For example, when the camera rotates (rolls), the relative positional relationship between the pixel arrangement of each color in the color filter and the subject changes before and after the rotation. Therefore, for example, the color of the edges on the subject may change before and after the rotation.

[0006] This invention has been made in view of these problems and aims to provide a technology for improving the performance of a demosaicing unit. [Means for solving the problem]

[0007] To solve the above-mentioned problems, the information processing apparatus according to the present invention has the following configuration. That is, the information processing apparatus is An acquisition means for acquiring a first mosaic image and a second mosaic image generated based on the same source image, Estimation means inputs the first mosaic image and the second mosaic image into a demosaicing unit to generate a first estimated image and a second estimated image with pixel value interpolation, A calculation means for calculating the first reconstruction error in the first estimated image and the second reconstruction error in the second estimated image, An update means for updating the parameters of the demosaicing processing unit based on the first reconstruction error and the second reconstruction error, Equipped with, The first mosaic image is a mosaic image in which the thinning pattern for generating the mosaic image and the original image are in a first positional relationship, and the second mosaic image is a mosaic image in which the thinning pattern and the original image are in a second positional relationship different from the first positional relationship. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a technology that can improve the performance of the demosaicing processing unit. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows the hardware configuration of an information processing device. [Figure 2] This diagram shows the functional configuration of the information processing device (first embodiment). [Figure 3] This is a flowchart of the inference process. [Figure 4] This is a flowchart of the learning process (first embodiment). [Figure 5] This diagram illustrates the creation of a mosaic image based on a thinning pattern. [Figure 6] This diagram illustrates the processing flow of a demosaicing neural network. [Figure 7] This figure shows the functional configuration of the information processing device (second embodiment). [Figure 8] This is a flowchart of the learning process (second embodiment). [Figure 9] This figure shows the functional configuration of the information processing device (third embodiment). [Figure 10] This is a flowchart of the learning process (third embodiment). [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] (First Embodiment) As a first embodiment of the information processing device according to the present invention, an information processing device for training a neural network for demosaicing (demosaic NN) will be described below as an example. In the following description, demosaicing will be described using demosaicing to generate a normal color image (RGB 3 component image) from a mosaic image of a Bayer (RGGB) sequence as an example, but the sequence and color configuration are not limited to those described above.

[0012] <Overview> In demosaicing, the relative positional relationship between the pixel arrangement of each color in the original image and the mosaic image changes, and when the color to be thinned changes, the difficulty of interpolating the pixel values ​​of the missing pixels in demosaicing changes. As a result, even with the same original image, the demosaicing result (color and image quality) will change. Therefore, the demosaicing neural network is trained to mitigate such changes in the demosaicing result.

[0013] Specifically, two or more mosaic images are created from the original image, each with a different arrangement (for example, the relative positional relationship between the pixel arrangement of each color in the original image and the mosaic image has changed). Then, in backpropagation, the parameters of the demosaicing neural network are updated so that the reconstruction error between the demosaicing result (inference result) from each mosaic image and the original image is minimized.

[0014] <Device configuration> Figure 1 shows the hardware configuration of an information processing device that performs training of a demosaicing neural network and / or inference using the demosaicing neural network. The information processing device may be configured as, for example, a computer. The CPU 101 controls the entire device by executing a control program stored in the read-only memory (ROM) 102. The random access memory (RAM) 103 temporarily stores various data from each component. The RAM 103 is also used as a work area where the CPU 101 loads and uses programs.

[0015] The storage unit 104 stores the data to be processed in the first embodiment. For example, it stores data used for learning processing and data to be used for inference processing. A hard disk drive (HDD), flash memory, optical media, etc., can be used as the storage medium for the storage unit 104.

[0016] <Functional configuration and operation during inference processing> Figure 2(a) shows the functional configuration of the information processing device during inference processing. Here, the inference processing is demosaicing, which generates an RGB image from a mosaic image. The information processing device has a storage unit 201, an image acquisition unit 202, and a demosaicing processing unit 203. Each functional unit is realized by the CPU 101 executing a predetermined program. For example, the demosaicing processing unit 203 is realized by the CPU 101 executing a program to perform inference processing using a demosaicing neural network.

[0017] In the inference process, the memory unit 201 stores the parameters for the machine learning-based inference process used by the demosaicing processing unit 203. The memory unit 201 also stores the mosaic image that is to be demosaiced.

[0018] The image acquisition unit 202 acquires the mosaic image to be demosaiced from the storage unit 201. The mosaic image may be read from one that has been captured and stored in advance, or it may be acquired from one captured by the imaging device. In this example, the mosaic image is acquired from the storage unit 201, but it may also be acquired directly from the imaging device without going through the storage unit 201.

[0019] A mosaic image is an image in which each of the multiple pixels that make up the mosaic image has pixel value information for one of the three colors (RGB). In a mosaic image, each of the three colors (RGB) is arranged in a predetermined spatial arrangement (e.g., a Bayer array). Hereafter, the arrangement pattern of each of the three colors (RGB) in a mosaic image will be called the decimation pattern. An example of the relationship between the decimation pattern and the mosaic image is shown in Figure 5. Note that Figure 5(a) shows a pixel arrangement called a Bayer array, but the pixel arrangement is not limited to this. As shown in Figure 5(b), the mosaic image 504 is a composite of the decimation patterns (component maps 501-503) for each of the three colors (RGB). In component maps 501-503, only the pixel values ​​of pixels shown in white are included (pixels shown in gray are missing pixel values).

[0020] The demosaicing processing unit 203 sets the parameters obtained from the storage unit 201 into the demosaicing NN. Then, it inputs the mosaic image into the demosaicing NN and performs inference processing (demosaic processing), outputting an estimated image (RGB image). Here, the estimated image refers to an RGB image (where each pixel has pixel values ​​for the RGB3 components) in which the missing pixels of each color in the component maps 501 to 503 have been interpolated.

[0021] Figure 6 illustrates the flow of the inference process using a demosaicing neural network (NN). While this example uses Demosaicing NN 602, other machine learning-based methods may also be used. The structure of the demosaicing NN may be as shown in Non-Patent Document 1, but is not limited to this; U-Net, GAN (Generative Adversarial Network), Transformer, etc., may also be used. Furthermore, the demosaicing NN includes nonlinear transformations such as convolution and ReLU (Rectified Linear Unit), but the nonlinear transformations are not limited to these. Based on the input mosaic image 601, Mosaic NN 602 estimates component maps 603-605 of the RGB 3 components with interpolated missing pixels, and outputs an RGB image containing component maps 603-605 as the estimated image.

[0022] Figure 3 is a flowchart of the inference process. However, the information processing device does not necessarily have to perform all the steps described in this flowchart.

[0023] In S301, the image acquisition unit 202 acquires the mosaic image to be demosaiced from the storage unit 201. In S302, the demosaicing processing unit 203 inputs the mosaic image acquired in S301 to the demosaicing NN602 and outputs an estimated image.

[0024] <Functional configuration and operation during the learning process> Figure 2(b) shows the functional configuration of the information processing device during the learning process. Here, the learning process is the process of updating the parameters of the demosaicing neural network that will be used in the inference process described above. The information processing device has a storage unit 201, an image creation unit 204, a mosaic image creation unit 205, a demosaicing processing unit 203, a reconstruction error calculation unit 206, and a parameter update unit 207. Each functional unit is realized by the CPU 101 executing a predetermined program. For example, the demosaicing processing unit 203 is realized by the CPU 101 executing a program to perform inference processing using the demosaicing neural network.

[0025] During the learning process, the memory unit 201 stores the parameters of the demosaicing processing unit 203 and the source image used for learning. The source image is assumed to be an RGB image with no missing pixel values ​​(i.e., each pixel has RGB components).

[0026] The image creation unit 204 retrieves the original image from the storage unit 201 and creates a first image and a second image from a single original image. As will be described in detail later, the first image and the second image are images that have undergone different geometric transformations from the original image.

[0027] The mosaic image creation unit 205 creates a first mosaic image and a second mosaic image by downsampling pixels / pixel values ​​from the first image and the second image obtained from the image acquisition unit 202 according to a predetermined downsampling pattern.

[0028] The demosaicing processing unit 203 inputs the first mosaic image created by the mosaic image creation unit 205 into the pre-trained demosaicing NN602, and the first estimated image I 1 It outputs the second mosaic image. Similarly, the second mosaic image is input to the demosaicing NN602, and the second estimated image I 2 Output the first estimated image I. 1 , Second estimated image I 2 You can estimate either one first, or you can estimate them in parallel.

[0029] The reconstruction error calculation unit 206 calculates the loss for the estimation result by the demosaicing processing unit 203. Specifically, the first image obtained by the image creation unit 204 is used as GroundTruth (GT), and the first estimated image I estimated by the demosaicing processing unit 203 is calculated as follows: 1 and the first image G 1 The difference is calculated. Hereafter, this difference will be referred to as the first reconstruction error. Second estimated image I estimated by demosaicing processing unit 203 2 Similarly, the second image G obtained by the image creation unit 204 is also obtained. 2The difference from is calculated as the second reconstruction error. Details of the calculation method will be described later.

[0030] Based on the first reconstruction error and the second reconstruction error obtained by the reconstruction error calculation unit 206, the parameter update unit 207 updates the parameters (for example, the weight coefficients of the filter) of the demosaicking NN 602 and stores them in the storage unit 201.

[0031] FIG. 4 is a flowchart of the learning process in the first embodiment.

[0032] In S401, the image creation unit 204 acquires an original image, which is an RGB image without defects, from the storage unit 201.

[0033] In S402, the image creation unit 204 uses one original image acquired in S401 to create a first image G 1 and a second image G 2 with different relative positional relationships. Here, "different relative positional relationships" means a state in which corresponding points of two images in which the same subject is reflected are in pixels at different positions. Specifically, a first image G 1 obtained by applying a geometric transformation to the original image, and a second image G 1 which is an image obtained by applying a geometric transformation different from that of the first image G 2 to the original image are output.

[0034] Since the first image G 1 and the second image G 2 are created from the same original image, the relationship between the two images is an image with different relative positional relationships in which a certain pixel of one image is mapped to a pixel at a different position of the other image. Here, the geometric transformation indicates rotation, translation, vertical inversion, horizontal inversion, enlargement, etc. of the image, and is a transformation that generally maintains the structure of the subject reflected in the image. Also, although it is assumed that the rotation transformation is a 90-degree series rotation transformation (rotation of 90×n degrees (n is an integer)), any rotation angle may be used. Also, the first image G 1 and the second image G 2Either of the geometric transformations used to create it may be an identity map (a rotation with zero rotation angle).

[0035] In S403, the mosaic image creation unit 205 creates a first mosaic image and a second mosaic image. Specifically, the first image G created in S402 1 and the second image G 2 For each of these, a first mosaic image and a second mosaic image are created by downsampling pixels / pixel values ​​according to a predetermined downsampling pattern.

[0036] Figure 5 illustrates the creation of a mosaic image based on a thinning pattern. That is, the first image G 1 Then, component maps 501-503 shown in Figure 5(b) are generated according to the thinning pattern 500 shown in Figure 5(a), and a mosaic image 504 (first mosaic image) is created. Similarly, a second image G is created. 2 From this, create mosaic image 504 (second mosaic image).

[0037] In S404, the demosaicing processing unit 203 inputs the first mosaic image created in S403 to the pre-trained demosaicing NN602, and the first estimated image I 1 The output is generated. Similarly, the second mosaic image is input to the demosaicing NN602, and the second estimated image I 2 Outputs.

[0038] In S405, the reconstruction error calculation unit 206 calculates the loss for the estimation result by the demosaicing processing unit 203. Specifically, the first image G 1 Using GroundTruth (GT), the first estimated image I 1 and the first image G 1 Calculate the difference between this and the first reconstruction error Loss. r 1 This will be the notation. Similarly, the second estimated image I 2 and the second image G 2 The difference between this and the second reconstruction error Loss r 2It is calculated as follows.

[0039] The formula for calculating the reconstruction error in this embodiment is shown in equation (1). Equation (1) calculates the L1 distance (Manhattan distance), which is the sum of the absolute differences between each element (pixel value) of the estimated image (denoted as I) and GT (denoted as G). Here, the type of loss is not limited to the L1 distance; for example, the L2 distance (Euclidean distance) or other calculation methods can be used.

[0040]

number

[0041] In S406, the parameter update unit 207 calculates the loss based on the weighted sum of the first reconstruction error and the second reconstruction error. total Calculate the loss. total Based on this, the parameters of the demosaicing NN602 are updated using backpropagation. The parameters to be updated here refer to the weight coefficients of the filters in the convolutional layers that make up the demosaicing NN602, as shown in Figure 3(a). Loss total An example of the calculation method is shown in formula (2).

[0042]

number

[0043] λ 1 and λ 2 represents the weighting coefficient. Here, λ 1 and λ 2 This value is set to be non-zero. As a result, in a single backpropagation, the parameter is updated so that the reconstruction errors of both images mentioned above are small, and learning proceeds in a way that balances the image quality of both.

[0044] In S407, the parameter update unit 207 saves the updated demosaicing NN602 parameters to the storage unit 201. Then, the filter weight coefficients are loaded into the demosaicing NN602. This entire process constitutes one iteration of training.

[0045] In S408, the parameter update unit 207 determines whether to terminate the learning process. The termination of the learning process is determined when the loss value obtained by equation (2) falls below a predetermined threshold. It is also determined to terminate the process after a predetermined number of learning sessions have been performed, preventing the learning time from becoming excessively long.

[0046] Alternatively, the processing in S401 to S405 may be performed on multiple source images to calculate the reconstruction error based on equation (1) for each, and the parameters of the demosaicing NN602 may be updated by backpropagation in S406. In this embodiment, in S402, two images are created by applying different geometric transformations to one source image, and these are designated as the first and second images, but three or more images may be created. In that case, in the processing in S403 to S405, the reconstruction error is calculated for each inferred image obtained by inputting the mosaic image corresponding to these three or more images into the demosaicing NN602. Then, in S406, the parameters are updated by backpropagation based on the weighted sum of all of these.

[0047] As described above, according to the first embodiment, in the training process of the demosaicing neural network, training is performed using two or more mosaic images created from two or more images obtained by performing different geometric transformations on the same original image. Specifically, in one backpropagation, the parameters are updated so that the reconstruction error corresponding to each mosaic image is reduced. This makes it possible to train the demosaicing neural network in such a way that changes in the demosaicing result (color and image quality) are mitigated even when the relative positional relationship between the pixel arrangement of each color in the original image and the mosaic image changes.

[0048] (Variation 1) Modification 1 describes another method for obtaining the two mosaic images used for training. In the first embodiment, the first image G is obtained by applying different geometric transformations to the original image. 1 and the second image G 2 Two mosaic images were created by creating and applying the same thinning pattern to each. In the first variation 1, the first image G 1 and the second image G 2 Apply a different thinning pattern to each.

[0049] Note that the following explanation refers to the first image G. 1 and the second image G 2 Although we will describe them as the same image, as in the first embodiment, they may be two images created by applying different geometric transformations to the original image. Here, we will only describe the parts that differ from the first embodiment (S402 and S403).

[0050] In S402, the image creation unit 204 creates a first image G by applying the same geometric transformation to one original image acquired in S401. 1 and the second image G 2 To create the first image G, as mentioned above, 1 and the second image G 2 These will be identical images. Here, the geometric transformation can also be the identity map.

[0051] In S403, the mosaic image creation unit 205 creates a first mosaic image and a second mosaic image. Specifically, the first image G created in S402 1 and the second image G 2 For each, a first mosaic image and a second mosaic image are created by downsampling pixels / pixel values ​​according to different downsampling patterns.

[0052] Therefore, the information processing device stores multiple different decimation patterns, and the mosaic image creation unit 205 switches between them for use. Here, different decimation patterns are assumed to be patterns obtained by performing geometric transformations such as 90-degree rotation, translation, vertical inversion, and horizontal inversion on a given decimation pattern. In other words, they are assumed to be patterns that match when geometric transformations such as 90-degree rotation, translation, vertical inversion, and horizontal inversion are performed.

[0053] Figure 5(c) shows examples of different thinning patterns 505 to 508. Thinning pattern 505 is the same as thinning pattern 500 used in the first embodiment. Thinning patterns 506 to 508 are obtained by rotating thinning pattern 505 90 degrees to the left, 90 degrees to the right, and 180 degrees, respectively.

[0054] The two mosaic images obtained by switching between these thinning patterns will have the same characteristics as the two mosaic images in the first embodiment. Therefore, by performing the same learning as in the first embodiment using these two mosaic images, the same effects as in the first embodiment can be obtained.

[0055] (Second Embodiment) In the second embodiment, during the training (parameter update) of the demosaicing NN, the difference in the demosaicing results of the Bayer sequence (= degree of transformation influence) is used as the loss. total We will now explain the form used in the calculation of [the result]. This will enable the training of a demosaicing neural network that further improves robustness to geometric transformations.

[0056] Figure 7 shows the functional configuration of the information processing device in the second embodiment. Note that the hardware configuration of the information processing device is the same as in the first embodiment (Figure 1), so its explanation is omitted. Also, the functional configuration and operation of the information processing device during inference processing shown in Figure 7(a) are the same as in the first embodiment (Figures 2(a) and 3), so their explanation is omitted.

[0057] <Functional configuration and operation during the learning process> Figure 7(b) shows the functional configuration of the information processing device during the learning process. The information processing device includes a storage unit 701, an image creation unit 704, a mosaic image creation unit 705, a demosaicing processing unit 703, a reconstruction error calculation unit 707, a parameter update unit 708, and a conversion influence calculation unit 706. The following describes the parts that differ from the first embodiment (the conversion influence calculation unit 706 and the parameter update unit 708).

[0058] The conversion influence calculation unit 706 calculates the first estimated image I calculated by the demosaicing processing unit 703. 1 and the second estimated image I 2 Based on the difference, the degree of conversion influence is calculated. The parameter update unit 708 updates the parameters of demosaicing NN602 based on the first reconstruction error and the second reconstruction error obtained by the reconstruction error calculation unit 707, and the degree of conversion influence calculated by the degree of conversion influence calculation unit 706, and stores them in the storage unit 701.

[0059] Figure 8 is a flowchart of the learning process in the second embodiment.

[0060] In S801, the image creation unit 704 obtains the original image, which is an RGB image without any missing parts, from the storage unit 701.

[0061] In S802, the image creation unit 704, similar to the first embodiment (S402), uses one source image acquired in S801 to create a first image G with a different relative positional relationship. 1 and the second image G 2 Create.

[0062] In S803, the mosaic image creation unit 705 creates a first mosaic image and a second mosaic image, similar to the first embodiment (S403).

[0063] In S804, the demosaicing unit 703 inputs the first mosaic image created in S803 to the pre-trained demosaicing NN602, and the first estimated image I 1The output is generated. Similarly, the second mosaic image is input to the demosaicing NN602, and the second estimated image I 2 Outputs.

[0064] In S805, the conversion influence calculation unit 706 calculates the first estimated image I 1 and the second estimated image I 2 Based on the difference, the conversion impact loss t Calculate the conversion impact (Loss). t An example of the calculation method is shown in equation (3). In equation (3), the first estimated image I 1 The image obtained by inputting into function f, and the second estimated image I 2 The function g takes the input to obtain the image, and calculates the L1 distance, which is the sum of the absolute differences between each element (pixel value) of the image. However, the L2 distance can be used instead of the L1 distance, and the function is not limited to these two.

[0065]

number

[0066] A pixel in the original image corresponds to the first image G. 1 , second image G 2 Each corresponds to a specific pixel. The degree of transformation represents the change in the pixel value of those corresponding pixels. However, since each image has undergone geometric transformations such as rotation from the original image, the corresponding points have moved to different positions. Therefore, by applying functions f and g to each image, the image f(I 1 ), image g(I 2 ) The corresponding pixels on the image are transformed so that they have the same coordinates. Functions f and g are the first image G in S802. 1 , second image G 2 When creating it, it may be determined as the inverse function of the function applied to the original image, but is not limited to this.

[0067] In S806, the reconstruction error calculation unit 707 calculates the loss for the estimation result by the demosaicing processing unit 703, similar to the first embodiment (S405).

[0068] In S807, the parameter update unit 708 calculates the loss, which is a weighted sum, based on the first reconstruction error, the second reconstruction error, and the degree of transformation. total Calculate the loss. total Based on this, the parameters of the demosaicing NN602 are updated using backpropagation. The parameters to be updated here refer to the weight coefficients of the filters in the convolutional layers that make up the demosaicing NN602. Loss total An example of the calculation method is shown in formula (4).

[0069]

number

[0070] Loss r 1 The first reconstruction error is Loss r 2 This is the second reconstruction error, Loss t represents the degree of transformation influence. Also, λ 1 and λ 2 , and λ t This represents the weighting coefficient.

[0071] Note that in the first embodiment, equation (2) is the λ in equation (4). t This corresponds to the case where is zero. In the second embodiment, λ t By setting this value to non-zero, the effect of reducing the difference between the first estimated image and the second estimated image can be enhanced.

[0072] In other words, in the first embodiment, by independently bringing the first estimated image and the second estimated image closer to the GT, the difference between the two is implicitly reduced, which has the effect of suppressing image quality changes due to subject movement or camera rotation. In addition, in the second embodiment, the degree of transformation influence is used in the calculation of the loss function to explicitly reduce the change in image quality. This enhances the effect of reducing the difference between the first estimated image and the second estimated image.

[0073] Note that λ 1 and λ 2may be non - zero or zero, and its value may be adjusted during the process. For example, learn with a non - zero value as in the first embodiment, and after the learning converges, set λ 1 and λ 2 to zero, and set λ t to zero and learn. With this control, the generalization performance of the demosaicking NN can be improved to a certain extent and then adjusted to prioritize the suppression performance of image quality changes. For example, it is effective when the priority of suppressing image quality changes due to subject movement or camera rotation is high among the image qualities required by the user for the demosaicking result.

[0074] In S808, the parameter update unit 708 stores the parameters of the updated demosaicking NN602 in the storage unit 701. Then, it reads the filter weight coefficients into the demosaicking NN602. The above - mentioned process is regarded as one - iteration learning.

[0075] In S809, the parameter update unit 708 determines whether to end the learning. The determination of ending the learning may be made to end when the value of the loss obtained by the formula (4) is smaller than a predetermined threshold. Also, it may be determined to end when learning has been performed a predetermined number of times.

[0076] As described above, according to the second embodiment, in the calculation of the loss Loss total used for error backpropagation, the conversion influence degree Loss t is further considered. By using the conversion influence degree in the calculation of the loss and explicitly reducing the image quality change, the geometric transformation robustness of the demosaicking NN is improved.

[0077] (Modification Example 2) In modification example 2, a method of using the conversion influence degree calculated by the conversion influence degree calculation unit 7 as the weighting coefficient of each element of the reconstruction error will be described. Here, only the parts different from the second embodiment (S805 and S806) will be described.

[0078] In S805, the conversion influence degree calculation unit 706 calculates the conversion influence degree Loss it Calculate it. Specifically, according to Equation (5), calculate the conversion influence degree for each element i of the image. Note that Equation (5) corresponds to the value before taking the sum of all elements in the conversion influence degree Loss t in Equation (3).

[0079] [Number]

[0080] In S806, the reconstruction error calculation unit 707 calculates the first reconstruction error Loss r 1 and the second reconstruction error Loss r 2 The calculation is shown in Equation (6). In Equation (6), for the difference of each element i between the second image G 2 and the second estimated image I 2 weight each element using the conversion influence degree calculated by Equation (5).

[0081] [Number]

[0082] By the above processing, it becomes possible to perform learning with a stronger weight of the reconstruction error for pixels with a large difference between the first estimated image and the second estimated image.

[0083] (Third Embodiment) In the third embodiment, a form using mini-batch learning in which a small number of samples are grouped together (mini-batch) and the parameters are updated in that unit will be described. Specifically, the image that becomes the GT of the demosaicing and the mosaic image that becomes the input to the demosaicing NN are accumulated in the storage unit as a candidate data group, and the image data constituting the mini-batch is selected from the candidate data group.

[0084] In the first embodiment described above, if the gradient direction of the high-frequency components in the original image is concentrated in a certain direction, the pair of images before and after conversion may not be able to cover a variety of gradient directions. In that case, differences in image quality may occur depending on the gradient direction of the high-frequency components in the image being processed. Therefore, in the third embodiment, pairs are selected to increase the variation in the gradient directions of the images constituting the minibatch. As a result, demosaicing performance is improved in various gradient directions, and differences in image quality between images with different gradient directions are reduced.

[0085] Figure 9 shows the functional configuration of the information processing device in the third embodiment. The hardware configuration of the information processing device is the same as in the first embodiment (Figure 1), so its explanation is omitted. Also, the functional configuration and operation of the information processing device during inference processing shown in Figure 9(a) are the same as in the first embodiment (Figures 2(a) and 3), so their explanation is omitted.

[0086] <Functional configuration and operation during the learning process> Figure 9(b) shows the functional configuration of the information processing device during the learning process. The information processing device includes a storage unit 901, an image creation unit 904, a mosaic image creation unit 905, an image selection unit 906, a demosaicing processing unit 903, a reconstruction error calculation unit 907, and a parameter update unit 908. The following describes the parts that differ from the first embodiment.

[0087] The image creation unit 904 creates the first image G using the same procedure as the image creation unit 204. 1 and the second image G 2 The first estimated image I is stored in the memory unit 901. The mosaic image creation unit 905 creates the first estimated image I using the same procedure as the mosaic image creation unit 205. 1 and the second estimated image I 2 This is stored in the memory unit 901.

[0088] The image selection unit 906 selects the first image G stored in the storage unit 901. 1 , second image G 2 , First estimated image I 1 and the second estimated image I 2The image selection unit 906 then acquires the candidate data set. The image selection unit 906 then outputs a subset of images (image groups) selected from the candidate data set as a minibatch. Details of the image selection criteria will be described later.

[0089] The demosaicing processing unit 903 applies the demosaicing neural network (NN) to all mosaic images included in the minibatch and outputs an estimated image. The reconstruction error calculation unit 907 calculates the corresponding reconstruction error for all images included in the minibatch. The parameter update unit 908 calculates a loss value using all the reconstruction errors calculated by the reconstruction error calculation unit 907 and updates the parameters of the demosaicing neural network using backpropagation based on that loss value.

[0090] Figure 10 is a flowchart of the learning process in the third embodiment.

[0091] In S1001, the image creation unit 904 obtains the original image, which is an RGB image without any missing parts, from the storage unit 901.

[0092] In S1002, the image creation unit 204 applies a geometric transformation to the original image acquired in S1001, similar to the first embodiment (S402), to create a first image G with a different relative positional relationship. 1 and the second image G 2 Create.

[0093] In S1003, the mosaic image creation unit 905 mosaics according to the thinning pattern, similar to the first embodiment (S403). In S1002, the mosaic image creation unit 905 creates a first mosaic image and a second mosaic image from the first image and the second image.

[0094] In S1004, the image selection unit 906 selects the first image G stored in the storage unit 901. 1 , second image G 2First, the first and second mosaic images are obtained as candidate data sets. Then, multiple images selected from the candidate data sets are output as minibatches. The selection is made such that the variation in the "gradient direction" of the high-frequency components (edges) of the images included in the minibatches is greater than a threshold (i.e., not biased in any particular direction).

[0095] For example, the gradient intensity and gradient direction are determined for all pixels in both the vertical and horizontal directions. The direction from 0 to 180 degrees is then divided into several sections, the number of pixels corresponding to each direction is counted, and a histogram is created. When this histogram is created for all pixels in the images included in the minibatch, the gradient variance obtained from this histogram should be greater than a predetermined threshold. For example, from a list of candidate image combinations, a combination with a gradient variance greater than the threshold or the combination with the maximum gradient variance may be selected as the minibatch. The combination of images may be selected randomly.

[0096] In S1005, the demosaicing unit 903 inputs all the first mosaic images included in the minibatch output by S1004 to the demosaicing NN602, and the first estimated image I 1 It outputs the following. Similarly, all second mosaic images included in the minibatch are input to the demosaicing NN602, and the second estimated image I 2 Outputs.

[0097] In S1006, the reconstruction error calculation unit 907 calculates the first reconstruction error and the second reconstruction error for all first and second images included in the minibatch. Specifically, the first reconstruction error is calculated as the difference between the first estimated image and the first image, similar to the first embodiment (S405). Similarly, the second reconstruction error is calculated as the difference between the second estimated image and the second image. The first and second reconstruction errors can each be calculated according to formula (1), but the calculation method is not limited to this.

[0098] In S1007, the parameter update unit 908 calculates the loss based on the first reconstruction error and the second reconstruction error. total Calculate the loss. total Based on this, the parameters of the demosaicing NN602 are updated using backpropagation. Loss total The calculation method can be calculated according to formula (2), but the calculation method is not limited to this.

[0099] In S1008, the parameter update unit 908 saves the updated parameters of the demosaicing NN602 to the storage unit 901. Then, the filter weight coefficients are loaded into the demosaicing NN602. This entire process constitutes one iteration of training.

[0100] In S1009, the parameter update unit 908 determines whether to terminate the learning process. The determination of whether to terminate the learning process is based on the loss calculated in S1007. total The process may be considered complete when the value falls below a predetermined threshold. Alternatively, it may be considered complete when a predetermined number of learning sessions have been completed.

[0101] As described above, according to the third embodiment, minibatches are generated such that the variation in the "gradient direction" of the high-frequency components of the image is greater than a threshold, and the reconstruction error is calculated and learned using the minibatches as processing units. This allows high-frequency components with various gradient directions to be included in the same minibatches, thus enabling learning that balances image quality in various edge directions. As a result, the edge directions that are difficult to handle due to the asymmetry of the pixel arrangement of each color in the mosaic image can be overcome, and a demosaicing NN that generates similar images for geometrically transformed input data is obtained.

[0102] (Variation 3-1) Modification 3-1 describes an image selection method that considers the balance between the degree of image degradation due to geometric transformation and the variation in the gradient direction of the image.

[0103] In the image creation unit 904, if an arbitrary rotation other than a 90-degree rotation is performed as a geometric transformation, image interpolation (pixel interpolation) is necessary. As a result, degradation such as blurring of the image may occur compared to the original image, potentially reducing the performance of the resulting demosaicing neural network. On the other hand, arbitrary rotation can increase the variation in the gradient direction of the image. Therefore, a mini-batch is selected considering the balance between image degradation and variation in gradient direction.

[0104] Specifically, in S1002, the image creation unit 204 performs frequency analysis on the original image, the first image, and the second image. Based on the difference in intensity of frequency components greater than a certain predetermined frequency, it calculates the degree of degradation of the first image relative to the original image, and the degree of degradation of the second image relative to the original image.

[0105] Subsequently, in S1004, the image selection unit 906 selects images from among those whose image degradation level calculated in S1002 is below a predetermined threshold, such that the variation in the gradient direction included in the images constituting the mini-batch is greater than a predetermined threshold. The balance may be adjusted by adjusting each threshold.

[0106] As explained above, according to modification 3-1, it becomes possible to perform learning that balances the image degradation that may occur in geometric transformations with the increase in variations in the gradient direction.

[0107] (Variation 3-2) Modification 3-2 describes a method for determining whether to select both images in a pair of images (the first image and the second image) created from the same source image, or to select only one of them. In particular, image selection is performed considering the balance between the variation in content (subject) included in the images constituting the mini-batch and the variation in the gradient direction. Here, the variation in content refers to the number of source images that will be included in the mini-batch.

[0108] To train a demosaicing neural network that can handle various inputs, it is important to increase the variety of content in the training images. However, in the method of the third embodiment, since multiple images are generated from a single source image, the variety of content in the mini-batch (the number of source images) may be limited. For example, image pairs (first image and second image) are generated based on the same source image, and multiple image pairs corresponding to multiple source images are selected to form a mini-batch. In this method, if there is an upper limit on the number of images that make up a mini-batch (let's say M images) due to memory constraints of the learning device, the number of source images included in the mini-batch will be limited to M / 2 images.

[0109] Therefore, in S1004, the image selection unit 906 ensures that the type of original image exceeds a predetermined threshold. In other words, for some of the multiple image pairs included in the candidate data set, only one of the image pairs (either the first or second image) is included in the mini-batch. However, the selection is made such that the variation in the gradient direction of the high-frequency components exceeds a predetermined threshold. Note that the balance between content variation and gradient direction variation may be changed by adjusting each threshold.

[0110] As explained above, according to modification 3-2, it becomes possible to perform learning that balances the increase in content variations with the increase in gradient direction variations.

[0111] The disclosures herein include the following information processing devices, control methods, and programs. (Item 1) An acquisition means for acquiring a first mosaic image and a second mosaic image generated based on the same source image, Estimation means inputs the first mosaic image and the second mosaic image into a demosaicing unit to generate a first estimated image and a second estimated image with pixel value interpolation, A calculation means for calculating the first reconstruction error in the first estimated image and the second reconstruction error in the second estimated image, An update means for updating the parameters of the demosaicing processing unit based on the first reconstruction error and the second reconstruction error, Equipped with, The first mosaic image is a mosaic image in which the thinning pattern for generating the mosaic image and the original image are in a first positional relationship, and the second mosaic image is a mosaic image in which the thinning pattern and the original image are in a second positional relationship different from the first positional relationship. An information processing device characterized by the following: (Item 2) The acquisition means acquires a first image obtained by applying a first geometric transformation to the original image, a second image obtained by applying a second geometric transformation different from the first geometric transformation to the original image, and a first mosaic image and a second mosaic image obtained by applying the same decimation pattern to the first image and the second image. The calculation means calculates the first reconstruction error, which is the difference between the first estimated image and the first image, and the second reconstruction error, which is the difference between the second estimated image and the second image. The information processing device described in item 1, characterized by the features described herein. (Item 3) The first geometric transformation and the second geometric transformation are rotation transformations in which the rotation angles are different from each other. The information processing device described in item 2, characterized in that it is an information processing device. (Item 4) The acquisition means acquires a first image and a second image obtained by applying the same geometric transformation to the original image, a first mosaic image obtained by applying a first thinning pattern to the first image, and a second mosaic image obtained by applying a second thinning pattern different from the first thinning pattern to the second image. The calculation means calculates the first reconstruction error, which is the difference between the first estimated image and the first image, and the second reconstruction error, which is the difference between the second estimated image and the second image. The information processing device described in item 1, characterized by the features described herein. (Item 5) The update means updates the parameters of the neural network by backpropagation so that the weighted sum of the first reconstruction error and the second reconstruction error is less than or equal to a predetermined threshold. An information processing device according to any one of items 1 to 4, characterized by the above. (Item 6) The system further includes an influence calculation means for calculating the conversion influence, which is the difference between the first estimated image and the second estimated image. The update means further updates the parameter based on the degree of transformation influence. An information processing device according to any one of items 1 to 5, characterized by the above. (Item 7) A storage means for storing a plurality of first images obtained by applying a first geometric transformation to a plurality of mutually different original images, a plurality of second images obtained by applying a second geometric transformation different from the first geometric transformation to the plurality of original images, and a plurality of first mosaic images and a plurality of second mosaic images obtained by applying the same decimation pattern to the plurality of first images and the plurality of second images. A selection means for selecting an image group which is a subset of the plurality of first images, the plurality of second images, the plurality of first mosaic images, and the plurality of second mosaic images from the storage means, Furthermore, The selection means selects the image group such that the variation in the gradient direction of the high-frequency components in the image group is greater than a threshold. The acquisition means acquires the image group. An information processing device according to any one of items 1 to 6, characterized by the above. (Item 8) The selection means selects from the plurality of first images and the plurality of second images an image whose degree of degradation from the original image is below a predetermined threshold. The information processing device described in item 7, characterized by the features described herein. (Item 9) The selection means selects images such that the number of types of original images corresponding to the plurality of first images or the plurality of second images exceeds a predetermined threshold. The information processing device described in item 7, characterized by the features described herein. (Item 10) The demosaicing processing unit is composed of a neural network (NN), and the parameters are the weight coefficients of the filter. An information processing device according to any one of items 1 to 9, characterized by the above. (Item 11) The aforementioned thinning pattern is a Bayer sequence. An information processing device according to any one of items 1 to 10, characterized by the above. (Item 12) A method for controlling an information processing device that learns a machine learning-based demosaicing process, An acquisition process to obtain a first mosaic image and a second mosaic image generated based on the same source image, An estimation step in which the first mosaic image and the second mosaic image are input to a demosaicing unit to generate a first estimated image and a second estimated image obtained by pixel value interpolation, A calculation step for calculating the first reconstruction error in the first estimated image and the second reconstruction error in the second estimated image, An update step of updating the parameters of the demosaicing processing unit based on the first reconstruction error and the second reconstruction error, Includes, The first mosaic image is a mosaic image in which the thinning pattern for generating the mosaic image and the original image are in a first positional relationship, and the second mosaic image is a mosaic image in which the thinning pattern and the original image are in a second positional relationship different from the first positional relationship. A control method characterized by the following: (Item 13) A program to cause a computer to execute the control method described in item 12.

[0112] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0113] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0114] 201 Memory Unit; 202 Image Acquisition Unit; 203 Demosaicing Processing Unit; 204 Image Creation Unit; 205 Mosaic Image Creation Unit; 206 Reconstruction Error Calculation Unit; 207 Parameter Update Unit

Claims

1. An acquisition means for acquiring a first mosaic image and a second mosaic image generated based on the same source image, Estimation means inputs the first mosaic image and the second mosaic image into a demosaicing unit to generate a first estimated image and a second estimated image with pixel value interpolation, A calculation means for calculating a first reconstruction error in the first estimated image and a second reconstruction error in the second estimated image, An update means for updating the parameters of the demosaicing processing unit based on the first reconstruction error and the second reconstruction error, Equipped with, The first mosaic image is a mosaic image in which the thinning pattern for generating the mosaic image and the original image are in a first positional relationship, and the second mosaic image is a mosaic image in which the thinning pattern and the original image are in a second positional relationship different from the first positional relationship. An information processing device characterized by the following:

2. The acquisition means acquires a first image obtained by applying a first geometric transformation to the original image, a second image obtained by applying a second geometric transformation different from the first geometric transformation to the original image, and a first mosaic image and a second mosaic image obtained by applying the same decimation pattern to the first image and the second image. The calculation means calculates the first reconstruction error, which is the difference between the first estimated image and the first image, and the second reconstruction error, which is the difference between the second estimated image and the second image. The information processing apparatus according to feature 1.

3. The first geometric transformation and the second geometric transformation are rotation transformations in which the rotation angles are different from each other. The information processing apparatus according to feature 2.

4. The acquisition means acquires a first image and a second image obtained by applying the same geometric transformation to the original image, a first mosaic image obtained by applying a first thinning pattern to the first image, and a second mosaic image obtained by applying a second thinning pattern different from the first thinning pattern to the second image. The calculation means calculates the first reconstruction error, which is the difference between the first estimated image and the first image, and the second reconstruction error, which is the difference between the second estimated image and the second image. The information processing apparatus according to feature 1.

5. The update means updates the parameters of the neural network by backpropagation so that the weighted sum of the first reconstruction error and the second reconstruction error is less than or equal to a predetermined threshold. The information processing apparatus according to feature 1.

6. The system further includes an influence calculation means for calculating the conversion influence, which is the difference between the first estimated image and the second estimated image. The update means further updates the parameter based on the degree of transformation influence. The information processing apparatus according to feature 1.

7. A storage means for storing a plurality of first images obtained by applying a first geometric transformation to a plurality of mutually different original images, a plurality of second images obtained by applying a second geometric transformation different from the first geometric transformation to the plurality of original images, and a plurality of first mosaic images and a plurality of second mosaic images obtained by applying the same decimation pattern to the plurality of first images and the plurality of second images. A selection means for selecting an image group which is a subset of the plurality of first images, the plurality of second images, the plurality of first mosaic images, and the plurality of second mosaic images from the storage means, Furthermore, The selection means selects the image group such that the variation in the gradient direction of the high-frequency components in the image group is greater than a threshold. The acquisition means acquires the image group. The information processing apparatus according to feature 1.

8. The selection means selects from the plurality of first images and the plurality of second images an image whose degree of degradation from the original image is below a predetermined threshold. The information processing apparatus according to feature 7.

9. The selection means selects images such that the number of types of original images corresponding to the plurality of first images or the plurality of second images exceeds a predetermined threshold. The information processing apparatus according to feature 7.

10. The demosaicing processing unit is composed of a neural network (NN), and the parameters are the weight coefficients of the filter. The information processing apparatus according to feature 1.

11. The aforementioned thinning pattern is a Bayer sequence. The information processing apparatus according to feature 1.

12. A method for controlling an information processing device that learns a machine learning-based demosaicing process, An acquisition step to obtain a first mosaic image and a second mosaic image generated based on the same source image, An estimation step in which the first mosaic image and the second mosaic image are input to a demosaicing unit to generate a first estimated image and a second estimated image with pixel value interpolation, A calculation step of calculating a first reconstruction error in the first estimated image and a second reconstruction error in the second estimated image, An update step of updating the parameters of the demosaicing processing unit based on the first reconstruction error and the second reconstruction error, Includes, The first mosaic image is a mosaic image in which the thinning pattern for generating the mosaic image and the original image are in a first positional relationship, and the second mosaic image is a mosaic image in which the thinning pattern and the original image are in a second positional relationship different from the first positional relationship. A control method characterized by the following:

13. A program for causing a computer to execute the control method described in claim 12.

Images