Image processing device, image processing method

Abstract

To provide technology for efficiently learning image quality enhancement processing. [Solution] A first loss is calculated based on the pixel values ​​corresponding to the patterns of image quality defects in the resulting image obtained by image quality enhancement processing of the input image, and the pixel values ​​corresponding to the patterns of image quality defects in the target image representing the correct answer for the input image. The parameters of the image quality enhancement processing are updated based on the first loss.

Description

Technical Field

[0001] The present invention relates to learning technology.

Background Art

[0002] For each pixel of an image sensor used in a digital imaging device such as a digital camera, light of a specific wavelength enters through a color filter. For example, when using a color filter having a Bayer array, a mosaic image having pixel values corresponding to any of the RGB colors is obtained from the image sensor. Then, by performing a high-image-quality processing such as a demosaicing process for obtaining pixel values corresponding to the remaining two colors for each pixel of the mosaic image, a high-image-quality color image having pixel values corresponding to each of a plurality of colors is obtained for each pixel. For this demosaicing process, there are methods of performing linear or non-linear interpolation based on pixel values of the same color in the surroundings, and methods of applying learning of a CNN-based neural network model (hereinafter, NN model).

[0003] On the other hand, when generating a color image by performing a demosaicing process on a mosaic image, an image quality defect pattern may occur at a specific position. For example, near the edge boundary of the mosaic image, it may be impossible to interpolate the missing pixel values of each pixel in the Bayer array with correct pixel values, and a false pattern of a checkerboard pattern may occur. There is a demand for improving the accuracy of high-image-quality processing for a mosaic image in which interpolation by such a demosaicing process is difficult.

[0004] Regarding this technology, Patent Document 1 discloses a technology in which a specific shape in a mosaic image is detected, and interpolation of pixel values in the detection region is sequentially performed from the G component with a small missing rate, so that the color image after the demosaicing process is interpolated with correct pixel values. By detecting the edge component of the mosaic image and controlling the order of pixel value interpolation near the detected edge boundary to perform the demosaicing process, it is possible to reduce image quality defects and improve the accuracy of high-image-quality processing.

Prior Art Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2015-154308 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] However, the technology disclosed in Patent Document 1, mentioned above, does not consider whether image quality degradation occurs as a result of demosaicing in order to detect a specific shape from a mosaic image. Therefore, in the technology disclosed in Patent Document 1, if unexpected image quality degradation occurs in the image after demosaicing, the degradation cannot be reduced. The present invention provides a technology for efficiently learning high-image-quality processing. [Means for solving the problem]

[0007] One aspect of the present invention is characterized by comprising: a calculation means for calculating a first loss based on pixel values ​​corresponding to patterns related to image quality defects in a result image obtained by image quality enhancement processing of an input image, and pixel values ​​corresponding to patterns related to image quality defects in a target image representing the correct answer for the input image; and an update means for updating the parameters of the image quality enhancement processing based on the first loss. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a technology for efficiently learning high-image-quality processing. [Brief explanation of the drawing]

[0009] [Figure 1A] A diagram showing the shooting scene 101 and the captured image 106. [Figure 1B] A diagram explaining how to generate the correct image 110. [Figure 1C] A diagram illustrating the learning process for demosaicing. [Figure 1D] A diagram showing an example of a captured image. [Figure 1E] A diagram for explaining the learning of noise removal processing. [Figure 2] A block diagram showing an example of the functional configuration of a high-image-quality processing system. [Figure 3] A block diagram showing an example of the hardware configuration of a computer device. [Figure 4] A flowchart of the learning of image quality defect patterns. [Figure 5] A diagram showing a specific example of the processing according to the flowchart of FIG. 4. [Figure 6] A diagram for explaining a method of calculating a loss value. [Figure 7] A diagram for explaining Modification Example 1 of the First Embodiment. [Figure 8] A diagram for explaining Modification Example 2 of the First Embodiment. [Figure 9] A diagram for explaining Modification Example 3 of the First Embodiment. [Figure 10] A diagram for explaining the Second Embodiment. [Figure 11] A diagram for explaining Modification Example 1 of the Second Embodiment. [Figure 12] A diagram for explaining the Third Embodiment. [Figure 13] A block diagram showing an example of the functional configuration of a high-image-quality processing system. [Figure 14] A flowchart according to the Fourth Embodiment. [Figure 15] A diagram for explaining the Fourth Embodiment. [Figure 16] A flowchart of Modification Example 2 of the Fourth Embodiment. [Figure 17] An example of the functional configuration of a high-image-quality processing system [Figure 18] A flowchart of the Fifth Embodiment. [Figure 19] A block diagram showing an example of the functional configuration of a high-image-quality processing system. [Figure 20] A flowchart of Modification Example 1 of the Fourth Embodiment.

Modes for Carrying Out the Invention

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] [First Embodiment] This embodiment describes an image processing device that enhances the image quality of images captured by an imaging device. In this embodiment, to enhance the image quality, a "NN model trained using Deep Learning to perform a process (image quality enhancement process) that obtains a high-quality color image from a Bayer array mosaic image captured by an imaging device" is used.

[0012] To train such a neural network (NN) model, a training dataset is created in advance, which includes the input mosaic image and the ground truth image obtained by appropriately demosaicing the mosaic image. Then, the error (loss value) between the image obtained by inputting the mosaic image from the training dataset into the NN model and performing calculations on the NN model (the resulting image) and the ground truth image from the training dataset is calculated, and the parameters of the NN model (such as weight parameters) are updated based on this loss value. This series of processes is repeated to train the NN model on demosaicing.

[0013] Here, in addition to learning the demosaicing process described above, by performing "learning image quality degradation patterns," which updates the parameters of the NN model based on the loss value derived from "patterns related to image quality degradation (image quality degradation patterns)" occurring in the resulting image, the learning of the NN model can be controlled to reduce image quality degradation. In this embodiment, by performing image quality degradation pattern learning on an NN model that has been trained for demosaicing, an NN model trained for the above-mentioned image quality enhancement process is generated.

[0014] In this embodiment, we describe a case where the "image quality detriment pattern" is a "checkerboard-like false pattern that arises when missing pixels cannot be completely interpolated from the Bayer array," but this is only one example.

[0015] Next, an example of acquiring a mosaic image used for training the image quality enhancement process in the NN model will be described. In this embodiment, the shooting scene 101 shown in Figure 1A will be used as an example. The shooting scene 101 includes a mark 104 that contains hues that are complementary, such as green and magenta, and a building 105 with many edges that are prone to artifacts. The user 102 uses the imaging device 103 to capture such a shooting scene 101. The mosaic image corresponding to the "captured image 106 including the mark 104 and the building 105" obtained by such shooting is used as a training image in the image processing device 201.

[0016] The learning of demosaicing in the NN model will be explained using Figures 1B and 1C. As shown in Figure 1B, a Bayer array mosaic image 108 is obtained by taking an image using the imaging device 103. By performing a simple demosaicing process (simple demosaicing) on ​​the mosaic image 108, an RGB color image 109 is obtained, which consists of a plane of R pixel values ​​(R plane), a plane of G pixel values ​​(G plane), and a plane of B pixel values ​​(B plane). The simple demosaicing process is assumed to be the known technique of bilinear interpolation, but other demosaicing processes may also be used. Then, the RGB color image 109 is reduced to generate a ground truth image 110, which is an image with reduced image quality defects such as moiré. More specifically, the R plane, G plane, and B plane in the RGB color image 109 are reduced to generate the R plane, G plane, and B plane in the ground truth image 110.

[0017] Then, as shown in Figure 1C, pixel values ​​are subsampled from the R plane, G plane, and B plane of the ground truth image 110 to generate a mosaic image 111. The mosaic image 111 is then input to the NN model 112, and the resulting image 113 obtained by performing calculations in the NN model 112 is used to calculate the error between the result image 113 and the ground truth image 110 as a loss value. Based on this calculated loss value, the weight parameters of the NN model 112 are updated to train the demosaicing process in the NN model 112.

[0018] In this embodiment, an NN model 112 that has learned image quality enhancement processing is generated by learning image quality detriment patterns from an NN model 112 that has learned such demosaicing processing.

[0019] Next, an example of the functional configuration of the high-image-quality processing system according to this embodiment will be described using the block diagram in Figure 2. As shown in Figure 2, the high-image-quality processing system 107 according to this embodiment includes an image processing device 201, a storage unit 202, and an output unit 208.

[0020] First, let's describe the memory unit 202. The memory unit 202 stores mosaic images obtained by imaging devices. The memory unit 202 is a device that can be used as storage for hard disk drives, server devices, and the like.

[0021] Next, the output unit 208 will be described. The output unit 208 is a device capable of outputting various types of information output from the image processing device 201. For example, the output unit 208 may be a display device capable of displaying various types of information output from the image processing device 201, or it may be a projection device such as a projector that projects the information. Furthermore, the output unit 208 may be a device that outputs (transmits) the information to an external device via a network such as a LAN or the Internet.

[0022] Next, the image processing device 201 will be described. The image processing device 201 learns demosaicing in the NN model based on the mosaic image stored in the memory unit 202, as described above. Note that the demosaicing learning described using Figures 1B and 1C above may be performed by the image processing device 201 using the mosaic image stored in the memory unit 202, or it may be performed by a separate device from the image processing device 201. The image processing device 201 then learns image quality degradation patterns for the NN model that has already learned demosaicing, thereby generating an NN model that has learned image quality enhancement. The learning of image quality degradation patterns for the NN model that has already learned demosaicing (hereinafter simply referred to as the NN model) will be explained according to the flowchart in Figure 4.

[0023] In step S401, the image enhancement unit 203 acquires the "Bayer array mosaic image captured by camera 103" stored in the memory unit 202 as the input image. The image enhancement unit 203 then inputs the acquired mosaic image into the NN model and performs calculations (image enhancement processing) on ​​the NN model to generate a result image, which is the result of demosaicing the mosaic image.

[0024] In step S402, the image acquisition unit 204 acquires the result image generated in step S401 as the detection target image, which is used to detect image quality degradation patterns.

[0025] In step S403, the detection unit 205 convolves a "filter with a checkerboard pattern" onto the target image acquired in step S402, thereby detecting false patterns that degrade image quality as image quality degradation patterns. More specifically, in this embodiment, the image quality degradation pattern is a checkerboard-like false pattern that occurs when pixel values ​​cannot be properly interpolated during demosaicing. This makes it possible to detect image quality degradation patterns in the target image, that is, image quality degradation patterns in the resulting image.

[0026] In step S404, the loss calculation unit 206 acquires an enlarged image as the target image, which is obtained by enlarging the "correct image obtained from the mosaic image acquired in step S401 by the process described using Figure 1B" to the same size as the image to be detected. Alternatively, the loss calculation unit 206 may acquire an RGB color image obtained by a simplified demosaicing process on the mosaic image in the process shown in Figure 1B as the target image instead of an enlarged image of the correct image generated from the mosaic image. In any case, the target image is an image that represents the correct answer for the mosaic image.

[0027] The loss calculation unit 206 then calculates the difference between the pixel value of each pixel that constitutes the image quality degradation pattern in the target image and the pixel value of the pixel that is positionally corresponding to that pixel in the target image, and calculates the sum of these differences as the error (loss value).

[0028] In step S405, the learning unit 207 updates the weight parameters of the NN model to reduce the loss value calculated in step S404. The process in steps S401 to S405 is repeated until the specified conditions are met. Various conditions can be applied to the specified conditions, for example, the loss value being less than or equal to a threshold, the difference between the previous loss value and the current loss value (change in loss value) being less than or equal to a threshold, or the number of repetitions of the process in steps S401 to S405 being greater than or equal to a threshold.

[0029] The process according to the flowchart in Figure 4 will be explained with the specific example shown in Figure 5. As shown in Figure 5(a), the image enhancement unit 203 inputs the mosaic image 501 into the NN model 112 and performs calculations (image enhancement processing) on ​​the NN model 112 to generate a result image 503 corresponding to the mosaic image 501. Here, it is assumed that a checkerboard-like false pattern occurs in the result image 503 as an image quality detriment pattern.

[0030] Then, as shown in Figure 5(c), the detection unit 205 convolves the result image 503 acquired as the image to be detected with the filter 505 exemplified in Figure 5(b), thereby obtaining a detection result image 506 that represents the image quality degradation pattern 507 in the result image 503.

[0031] In this embodiment, to detect false patterns in the resulting image 503, a checkerboard-like 4x4 false pattern detection filter with filter coefficients correlated to a Bayer array is used as filter 505. Note that the size of the filter and the filter coefficients used for detection are not limited to a specific size and coefficients; for example, a filter with a size of 5x5 can be used, or a filter in which positive and negative values ​​are repeated for each pixel in both the vertical and horizontal directions as coefficients may also be used.

[0032] The loss calculation unit 206 then calculates the difference between the pixel value of each pixel that constitutes the image quality degradation pattern 507 (the region in the result image 503 that is positionally corresponding to the image quality degradation pattern 507 in the detected result image 506) shown within frame 503a in the result image 503 acquired as the image to be detected, and the pixel value of the pixel that is positionally corresponding to that pixel within frame 504a in the target image 504 corresponding to the mosaic image 501, and calculates the sum of these differences as the error (loss value).

[0033] For example, as shown in Figure 6, the loss calculation unit 206 calculates the difference in pixel values ​​between pixels that are positionally corresponding to the pixel group 601A of the image quality degradation pattern 507 in the result image 503 and the pixel group 601B corresponding to the pixel group of the image quality degradation pattern 507 in the target image 504. The loss calculation unit 206 then calculates the sum of these calculated differences as the error (loss value). For example, if the number of pixels in the pixel group of the image quality degradation pattern 507 is N, the loss calculation unit 206 calculates the loss value L according to the following formula.

[0034] L=(601A1-601B1) 2 +(601A2-601B2) 2 +(601A3-601B3) 2+…+(601AN-601BN) 2 Furthermore, the method for calculating the loss value based on the difference between pixel values ​​is not limited to a specific method; various calculation methods can be applied. The learning unit 207 then updates the weight parameters of the NN model 112 to reduce the loss value calculated in this way.

[0035] Thus, according to this embodiment, by learning not only demosaicing processing but also image quality detriment patterns, it becomes possible to learn image quality enhancement processing that efficiently reduces detrimental effects in addition to demosaicing processing.

[0036] The following describes some modifications of this embodiment. In the following modifications, the differences from the first embodiment will be explained, and unless otherwise specified, they will be considered the same as the first embodiment.

[0037] <Example 1> In the first embodiment, the result image was used as the image to be detected, but any image other than the result image may be used as the image to be detected, as long as it is possible to detect image quality degradation patterns in the result image.

[0038] The image quality detriment that should be reduced to improve the accuracy of image enhancement processing is, in other words, a pattern that does not exist in the target image but exists only in the result image. Therefore, it is acceptable to use a difference image, calculated by subtracting the total pixel values ​​of the result image and the target image, as the image to be detected.

[0039] For example, in step S402, the image acquisition unit 204 acquires a difference image 701 between the result image 503 and the target image 504 as the detection target image, as shown in Figure 7(a). The pixel value of the pixel position (x, y) in the difference image 701 is the difference between the pixel value of the pixel position (x, y) in the result image 503 and the pixel value of the pixel position (x, y) in the target image 504.

[0040] Then, in step S403, as shown in Figure 7(b), the detection unit 205 convolves the difference image 701 (the image to be detected acquired in step S402) with a "filter 505 having a checkerboard pattern" to obtain a detection result image 702 that represents the image quality degradation pattern 703 in the difference image 701.

[0041] As a result, the loss calculation unit 206 can identify the region in the result image 503 that corresponds to the image quality degradation pattern 703 in the detected result image 702 as the image quality degradation pattern 703 in the result image 503.

[0042] Similarly, the loss calculation unit 206 can identify the region in the target image 504 that corresponds to the image quality degradation pattern 703 in the detected result image 702 as the image quality degradation pattern 703 in the target image 504.

[0043] <Modification 2> In the first embodiment, image quality degradation patterns in the resulting image were detected by convolution of a filter onto the resulting image. However, other methods may be used to detect image quality degradation patterns in the resulting image. For example, the image quality degradation patterns to be detected may be registered as template images, and the image quality degradation patterns in the resulting image may be detected by template matching using the template images.

[0044] For example, as shown in Figure 8, a pattern of image quality defects is created in advance as a template image 801. Then, in step S403, the detection unit 205 calculates the similarity between each position on the result image 503, which is the image to be detected acquired in step S402, and the image in the region of the result image 503 that overlaps with the template image 801. The similarity between images is, for example, the sum of the differences in pixel values ​​of pixels that are positionally corresponding between the images.

[0045] The detection unit 205 then detects image quality degradation patterns 802 in the resulting image 503 based on the similarity calculated for each position. For example, the detection unit 205 detects images in the region overlapping with the template image 801 at the position where the highest similarity was found in the resulting image 503 as image quality degradation patterns 802. Alternatively, the detection unit 205 also detects images in the region overlapping with the template image 801 at the position where a similarity above a threshold was found in the resulting image 503 as image quality degradation patterns 802.

[0046] <Variation 3> In the first embodiment, we described a case in which the image quality enhancement process is learned to "obtain a high-quality color image from a Bayer array mosaic image captured by an imaging device." In contrast, in the learning of the image quality enhancement process according to this embodiment, the NN model, which has been learned to perform noise reduction processing to remove noise from captured images taken by an imaging device, is further trained to learn image quality detrimental patterns.

[0047] For example, there are cases where noise reduction processing is performed on noisy images captured with a high-sensitivity camera, removing only the noise components, using a neural network model trained with deep learning.

[0048] To train such a neural network (NN) model, a training dataset is created in advance, consisting of a set of input noisy images and noise-free ground truth images obtained by appropriately removing noise components from the noisy images. Then, the error (loss value) between the image obtained by inputting the noisy images from the training dataset into the NN model and performing calculations on the NN model (result image) and the noise-free ground truth image from the training dataset is calculated, and the parameters of the NN model (such as weight parameters) are updated based on this loss value. This series of processes is repeated to train the NN model's noise reduction processing.

[0049] Here, by learning image quality degradation patterns in addition to learning the noise reduction process described above, the learning of the NN model can be controlled to reduce image quality degradation. In this embodiment, an NN model trained on the image quality enhancement process described above is generated by learning image quality degradation patterns on an NN model that has been trained on the noise reduction process.

[0050] In this embodiment, we describe a case where the "image quality detriment pattern" is a "scratch noise type artifact that occurs because it cannot be completely removed due to a random bias in noise components," but this is only one example.

[0051] The training of the denoising process in the NN model will be explained using Figures 1D and 1E. In training the denoising process, as shown in Figure 1D, a set of images is prepared in advance as a training dataset: image 106, which was taken with a relatively low ISO sensitivity to avoid noise, and image 114, which was taken with a relatively high ISO sensitivity to introduce noise. Images 106 and 114 may be Bayer array mosaic images, or RGB color images obtained by demosaicing the mosaic images. However, images 106 and 114 are limited to images taken of the same scene with the same field of view.

[0052] Then, as shown in Figure 1E, the result image 117 obtained by inputting the building 115 in the captured image 114 into the NN model 116 and performing calculations in the NN model 116 is obtained, and the error between this result image and the building 105 in the captured image 106 is calculated as a loss value. Then, the weight parameters of the NN model 116 are updated based on this calculated loss value, thereby training the noise reduction process in the NN model 116.

[0053] The noise reduction process described using Figures 1D and 1E above may be performed by the image processing device 201 using captured images stored in the memory unit 202, or it may be performed in a device separate from the image processing device 201.

[0054] In this modified version, the NN model 116 that has learned such noise reduction processing is used to generate an NN model 116 that has learned image quality degradation patterns, thereby generating an NN model 116 that has learned image quality enhancement processing.

[0055] In other words, in this modified example, in step S401, the image enhancement unit 203 inputs the captured image 114 stored in the memory unit 202 to the NN model 116 and generates the resulting image 117 by performing calculations (image enhancement processing) on ​​the NN model 116.

[0056] In step S402, the image acquisition unit 204 acquires the resulting image 117 as the image to be detected for detecting image quality detriment patterns. In step S403, the detection unit 205 convolves a "filter with a checkerboard pattern" onto the image to be detected acquired in step S402, thereby detecting false patterns that are image quality detriment patterns.

[0057] In step S404, the loss calculation unit 206 calculates the difference between the pixel value of each pixel constituting the image quality degradation pattern in the target image and the pixel value of the pixel that is positionally corresponding to that pixel in the target image, the captured image 106. The loss calculation unit 206 then calculates the sum of these calculated differences as the error (loss value). In step S405, the learning unit 207 updates the weight parameters of the NN model 116 to reduce the loss value calculated in step S404.

[0058] The learning process for the image enhancement process according to this embodiment will be explained with reference to a specific example shown in Figure 9. As shown in Figure 9(a), the image enhancement unit 203 inputs the noisy image 901 to the NN model 116 and performs calculations (image enhancement processing) on ​​the NN model 116 to generate a result image 903 corresponding to the noisy image 901.

[0059] Here, as shown in Figure 9(b), an image quality degradation pattern is created in advance as a template image 905. The detection unit 205 then calculates the similarity between each position on the result image 903 and the image in the region of the result image 903 that overlaps with the template image 905.

[0060] The detection unit 205 then detects the image quality degradation patterns 906 in the resulting image 903 based on the similarity calculated for each position. In this modified example, the method for detecting the image quality degradation patterns 906 in the resulting image 903 is not limited to a specific method, and a method of detecting image quality degradation patterns by filter convolution, as in the first embodiment, may also be adopted.

[0061] The loss calculation unit 206 then calculates the difference between the pixel value of each pixel constituting the image quality degradation pattern 906 in the result image 903 and the pixel value of the pixel that is positionally corresponding to that pixel in the target image 904, as shown in Figure 9(a). The loss calculation unit 206 then calculates the sum of these calculated differences as the error (loss value).

[0062] [Second Embodiment] In the following embodiments, including this embodiment, the differences from the first embodiment will be described, and unless otherwise specified below, they will be the same as the first embodiment. In the first embodiment, the loss value was calculated based on the difference in pixel values ​​calculated for each pixel constituting the image quality degradation pattern. In contrast, in this embodiment, the region surrounding the image quality degradation pattern is identified, and the loss value is calculated based on the difference in pixel values ​​calculated for each pixel within that region.

[0063] In the following, this embodiment will be explained with reference to a specific example shown in Figure 10. As shown in Figure 10(a), the detection unit 205 obtains a detection result image 506 representing the image quality degradation pattern in the result image 503 by convolving the filter 505 onto the result image 503. In the detection result image 506, the region of the image quality degradation pattern has non-zero pixel values, and the region other than the image quality degradation pattern has zero pixel values. The detection unit 205 then identifies the leftmost pixel position LX, the rightmost pixel position RX, the uppermost pixel position UY, and the lowermost pixel position DY among the pixel positions of the pixels with non-zero pixel values ​​in the detection result image 506. The detection unit 205 then identifies a rectangular region with pixel positions LX, RX, UY, and DY as the leftmost, rightmost, uppermost, and lowermost regions, respectively, as the image quality degradation pattern region 1001. Furthermore, the method for identifying the image quality detriment pattern region is not limited to a specific method. For example, the detection result image 506 may be divided into multiple subregions, and the set of subregions consisting only of pixels with non-zero pixel values ​​may be identified as the image quality detriment pattern region.

[0064] The loss calculation unit 206 then calculates the difference in pixel values ​​between pixels in the image quality degradation pattern region 1001 in the result image 503 and the image quality degradation pattern region 1001 in the target image 504, and calculates the sum of these differences as the error (loss value).

[0065] For example, if N is the number of pixels in the pixel group of the image quality detriment pattern region 1001, the loss calculation unit 206 calculates the loss value L using the following MSE as the loss function. MSE is an abbreviation for Mean Squared Error.

[0066] L={(1002A1-1002B1) 2 +(1002A2-1002B2) 2 +(1002A3-1002B3) 2 +…+(1002AN-1002BN) 2} / N Furthermore, other types of functions may be used for the loss function; for example, the loss value may be calculated using Cross Entropy, or it may be calculated in the same manner as in the first embodiment. Thus, according to this embodiment, learning can be performed to reduce not only the image quality detriment pattern but also the error with the target image located in the surrounding area.

[0067] The following describes some modifications of this embodiment. In the following modifications, the differences from the second embodiment will be explained, and unless otherwise specified, the modifications will be considered the same as those of the second embodiment.

[0068] <Example 1> In the first embodiment (including modifications) and the second embodiment, image quality detriment patterns were detected by convolving a filter having a checkerboard pattern. However, in actual filter convolution processing, there are cases where areas other than the image quality detriment patterns to be detected are also falsely detected. Therefore, after performing the image quality detriment pattern detection processing by filter convolution, it is also possible to perform a process to remove falsely detected areas that are unrelated to the image quality detriment patterns to be detected.

[0069] The process of step S403 according to this embodiment will be explained using Figure 11 as an example. As shown in Figure 11(a), in step S403, the detection unit 205 convolves the filter 505 onto the result image 503 to obtain a detection result image 1101 that represents the image quality degradation pattern in the result image 503. The detection result image 1101 is an image that also includes false detection regions of the image quality degradation pattern.

[0070] However, as shown in Figure 11(b), in step S403, the detection unit 205 performs a false detection area removal process 1102 on the detection result image 1101 to remove false detection areas other than the image quality detriment pattern to be detected, and generates a detection result image 1103 that includes only the image quality detriment pattern. One example of a false detection area here is a low-luminance component that was falsely detected during detection by filter convolution. In order to remove this low-luminance component, the detection unit 205 performs a false detection area removal process 1102 on the detection result image 1101.

[0071] For example, the detection unit 205 uses a well-known conversion formula to determine the pixel value of the L component, which is the luminance component, from the pixel values ​​of the RGB components of the detection result image 1101. Then, the detection unit 205 multiplies the pixel value of the L component by the pixel value of the RGB components to obtain the detection result image 1103, which is the result of performing the false detection area removal process 1102 on the detection result image 1101.

[0072] Another example of a false detection area is a colored edge component that is falsely detected during detection by filter convolution. To remove this colored edge component, the detection unit 205 may calculate the average pixel value of the RGB components of the detection result image 1101 and perform a false detection area removal process 1102 to set pixel values ​​lower than the calculated average pixel value to 0, thereby obtaining a detection result image 1103 in which only the image quality detrimental patterns are extracted.

[0073] Alternatively, a false detection area removal process 1102 may be performed, in which pixel values ​​in the detection result image 1101 obtained by simple convolution using a filter that are lower than a pre-set threshold are set to 0. Furthermore, multiple false detection area removal processes 1102 may be combined and performed.

[0074] Thus, according to this embodiment, even if the detected image contains false detection areas, it becomes possible to appropriately detect only the image quality degradation patterns, and therefore, learning can be performed to control the reduction of image quality degradation.

[0075] [Third Embodiment] The method for calculating the loss value according to this embodiment will be explained with reference to a specific example shown in Figure 12. As shown in Figure 12(a), the loss calculation unit 206 obtains a detection result image 1201 by convolving the filter 505 onto the result image 503. Similarly, the loss calculation unit 206 obtains a detection result image 1202 by convolving the filter 505 onto the target image 504.

[0076] Since the result image 503 contains image quality degradation pattern 507, the detected result image 1201 is an image containing image quality degradation pattern 507. On the other hand, since the target image 504 does not contain image quality degradation pattern 507, the detected result image 1202 is an image that does not contain image quality degradation pattern 507.

[0077] Then, as shown in Figure 12(b), the loss calculation unit 206 calculates the difference in pixel values ​​between pixels that are positionally corresponding to the image quality degradation pattern 507 in the detection result image 1201 and the pixel group 1203B that is positionally corresponding to the image quality degradation pattern region 507 in the detection result image 1202, and calculates the sum of these differences as the error (loss value).

[0078] Thus, according to this embodiment, since the image quality detriment pattern is emphasized by the detection filter and the loss value is calculated, it becomes possible to perform learning that emphasizes a predetermined image quality detriment.

[0079] [Fourth Embodiment] In this embodiment, one filter is selected from among several filters applicable to the detection of image quality degradation patterns to be applied to the image to be detected. The learning of image quality degradation patterns for an NN model that has already been trained on demosaicing will be explained according to the flowchart in Figure 14. In Figure 14, processing steps similar to those shown in Figure 4 are given the same step numbers, and the explanation of these processing steps will be omitted.

[0080] In step S1401, the detection unit 205 determines which filter to apply to the detection target image acquired in step S402 from among several filters. The processing in step S1401 will be explained with reference to a specific example shown in Figure 15.

[0081] The following describes the process of determining which filter to apply to the image to be detected from filters 505, 1501, and 1502 shown in Figure 15(a). Filters 505, 1501, and 1502 are all pre-generated filters.

[0082] Filter 505 is a 4x4 false pattern detection filter for detecting image quality detrimental patterns. Filter 1501 is a 3x3 dot pattern detection filter for detecting dot patterns, and filter 1502 is a 5x5 high-frequency repeating pattern detection filter for detecting high-frequency repeating patterns.

[0083] The detection unit 205 generates a detection result image 506 by convolving the result image 503, which is the image to be detected, with the filter 505. The detection unit 205 also generates a detection result image 1503 by convolving the result image 503 with the filter 1501. The detection unit 205 also generates a detection result image 1504 by convolving the result image 503 with the filter 1502.

[0084] The detection unit 205 then calculates the detection intensity for each of the detection result images 506, 1503, and 1504 based on the non-zero pixel values. The method for calculating the detection intensity based on the non-zero pixel values ​​is not limited to a specific method.

[0085] For example, the detection unit 205 may calculate the detection intensity for the detection result image as the sum of the non-zero pixel values ​​in the detection result image, or it may calculate the detection intensity for the detection result image as the average value of the non-zero pixel values ​​in the detection result image. Alternatively, the detection unit 205 may calculate other statistical values ​​of the non-zero pixel values ​​in the detection result image as the detection intensity for the detection result image.

[0086] The detection unit 205 then determines which of the filters 505, 1501, and 1502 was used to generate the detection result image with the highest detection intensity, and selects this filter to be used in step S403.

[0087] Note that the method for determining the filter is not limited to a specific method. For example, the detection unit 205 calculates the smallest rectangular area encompassing all pixels with non-zero pixel values ​​as the detection range for each of the detection result images 506, 1503, and 1504. The detection unit 205 then determines the filter used to generate the detection result image with the largest area (number of pixels) as the filter to be used in step S403.

[0088] When using template matching instead of filter convolution, the template image to be used may be determined in the same way as the filter determination method described above. In that case, the detection intensity for each detected image may be the minimum or maximum similarity obtained by template matching to that detected image, or the detection intensity may be calculated based on that similarity. Alternatively, the detection range may be the number of positions in the template image for which similarity below a threshold has been calculated.

[0089] Thus, according to this embodiment, even if the image quality degradation patterns occurring in the resulting image are not known in advance, it becomes possible to select an appropriate filter from multiple filters. Therefore, by detecting image quality degradation patterns in accordance with the learning progress of the NN model and learning to reduce those image quality degradation patterns, the accuracy of the image quality enhancement process can be improved.

[0090] The following describes some modifications of this embodiment. Note that the following modifications will only describe the differences from the fourth embodiment; unless otherwise specified, they will be considered the same as the fourth embodiment.

[0091] <Example 1> In this modified example, the filter to be used in step S403 is determined according to user operation from among multiple filters. An example of the functional configuration of the high-image-quality processing system according to this modified example is shown in the block diagram of Figure 19. The configuration in Figure 19 is the same as the configuration in Figure 2, but with the addition of the input unit 1901.

[0092] The input unit 1901 is a user interface such as a keyboard, mouse, or touch panel screen, which allows the user to input various instructions and information to the image processing device 201 by operating it. The user operates the input unit 1901 to select one of several filters to be used in step S403.

[0093] The learning of image quality degradation patterns for a NN model that has already been trained on demosaicing will be explained according to the flowchart in Figure 20. In Figure 20, processing steps similar to those shown in Figure 4 are given the same step numbers, and the explanation for these processing steps will be omitted.

[0094] In step S2001, the image enhancement unit 203 displays the resulting image generated by the processing in step S401 on the output unit 208. The user views the displayed resulting image and uses the input unit 1901 to select a filter from among several filters that can identify the image quality degradation pattern occurring in the resulting image.

[0095] In step S2002, the detection unit 205 receives user input from the input unit 1901 and acquires the filter selected in accordance with the user input as the filter to be used in step S403.

[0096] In step S403, the detection unit 205 convolves the filter acquired in step S2002 onto the image to be detected acquired in step S402, thereby detecting false patterns that detract from image quality as image quality detrimental patterns.

[0097] This allows for efficient learning limited to the image quality detriment patterns that the user wishes to reduce. In this modified example, the filter selection was performed according to user input, but the template image used in template matching could also be selected according to user input.

[0098] <Modification 2> In this modified version, the filter to be used in step S403 is determined from the component analysis of the image to be detected. An example of the functional configuration of the high-image-quality processing system according to this modified version is shown in the block diagram of Figure 13. In the configuration shown in Figure 13, the image processing device 201 further includes an image information acquisition unit 1301 in addition to the configuration shown in Figure 2. The image information acquisition unit 1301 acquires at least one of the hue, frequency components, and edge components as image information from the image to be detected acquired by the image acquisition unit 204.

[0099] The learning of image quality degradation patterns for a NN model that has already been trained on demosaicing will be explained according to the flowchart in Figure 16. In Figure 16, processing steps similar to those shown in Figure 4 are given the same step numbers, and explanations for these processing steps are omitted.

[0100] In step S1601, the image information acquisition unit 1301 acquires image information of the result image by performing component analysis of the result image acquired as the detection target image in step S402. The component acquired by component analysis is at least one of the following components: the H component when the RGB color space is converted to the HSV color space, the frequency component when the Fourier transform is performed, and the edge component when Canny edge detection is performed. The conversion to the HSV color space, the Fourier transform, and the Canny edge detection here may be performed using known techniques, and are not limited to these. For example, as a method for acquiring the frequency component, the component may be acquired by filtering to extract only specific frequency components, or as edge detection, the component may be acquired by convolution of a Sobel filter.

[0101] In step S1602, the detection unit 205 determines which filter to be used in step S403 from among a plurality of filters, based on the image information acquired by the image information acquisition unit 1301 in step S1601.

[0102] An example of a method for determining which filter to apply to the image to be detected from filters 505, 1501, and 1502 shown in Figure 15(a) based on the H component is described below. If the image to be detected contains an H component that is a diagonal component, the detection unit 205 determines that filter 505 is the filter to apply to the image to be detected.

[0103] An example of a method for determining which filter to apply to the target image from filters 505, 1501, and 1502 shown in Figure 15(a) based on edge components will be explained. The detection unit 205 calculates the content of edge components in the target image, and if the content of edge components is above a threshold, it determines that filter 1501 will be applied to the target image.

[0104] An example of a method for determining which filter to apply to the image to be detected from filters 505, 1501, and 1502 shown in Figure 15(a) based on frequency components will be explained. If the image to be detected contains high-frequency components above a threshold, the detection unit 205 determines that filter 1502 will be applied to the image to be detected.

[0105] This allows for the determination of filters based on component analysis of the resulting image, enabling more accurate detection of image quality degradation patterns. As a result, learning can be performed with more precise control over the reduction of image quality degradation.

[0106] [Fifth Embodiment] In this embodiment, we describe a case in which both a loss value based on the overall image difference and a loss value based on the difference in image quality degradation patterns are calculated between the detected image and the target image, and the NN model is trained to reduce these loss values. According to this embodiment, it becomes possible to improve the accuracy of image quality enhancement for the entire image while also reducing the image quality degradation patterns that occur.

[0107] A block diagram of Figure 17 shows an example of the functional configuration of the high-image-quality processing system according to this embodiment. The configuration shown in Figure 17 is the same as the configuration shown in Figure 2, but with the loss calculation unit 206 replaced by a first loss value calculation unit 1701, a second loss value calculation unit 1702, and a loss value integration unit 1703.

[0108] The first loss calculation unit 1701 obtains the loss value obtained by performing the same processing as the loss calculation unit 206 described above as the first loss value. In other words, the first loss value calculation unit 1701 obtains the detection target image and the target image. In this embodiment as well, instead of an enlarged image of the mosaic image, an RGB color image obtained by a simplified demosaicing process on the mosaic image in the processing of Figure 1B may be used. The first loss value calculation unit 1701 then calculates the difference between the pixel value of each pixel constituting the image quality degradation pattern in the detection target image and the pixel value of the pixel that is positionally corresponding to that pixel in the target image, and calculates the sum of these differences as the error (first loss value).

[0109] The second loss value calculation unit 1702 calculates the difference in pixel values ​​of pixels that are positionally corresponding between the detection target image and the target image, and calculates the sum of these differences as the error (second loss value). The second loss value may also be calculated based on a loss function; for example, MSE may be used as the loss function, or Cross Entropy may be used.

[0110] The loss value merging unit 1703 obtains the first loss value calculated by the first loss calculation unit 1701 and the second loss value calculated by the second loss value calculation unit 1702, and calculates an integrated loss by integrating the first loss value and the second loss value. For example, the loss value merging unit 1703 calculates the integrated loss value as the sum of the result obtained by multiplying the first loss value by a weight value α and the second loss value.

[0111] The weight value α may be a pre-set parameter, or it may be a parameter that is dynamically determined during processing. In the latter case, for example, the loss value integration unit 1703 may determine the weight value α according to the processing result of step S403.

[0112] For example, the loss value integration unit 1703 calculates the detection intensity as a sum, mean, or other statistical value for non-zero pixel values ​​in the detection result image obtained by convolving a filter onto the image to be detected. Alternatively, the loss value integration unit 1703 calculates the smallest rectangular region encompassing all pixels with non-zero pixel values ​​in the detection result image as the detection range, and calculates the area (number of pixels) of this detection range as the detection intensity. The loss value integration unit 1703 then sets a weight value α that increases as the detection intensity increases. Note that the above processing can also be applied in the case of template matching.

[0113] The learning process of the NN model according to this embodiment will be explained according to the flowchart in Figure 18. In Figure 18, processing steps similar to those shown in Figure 4 are given the same step numbers, and the explanation of these processing steps will be omitted.

[0114] In step S1801, the first loss value calculation unit 1701 calculates the first loss value using the detection target image and the target image. In step S1802, the second loss value calculation unit 1702 calculates the second loss value using the detection target image and the target image.

[0115] In step S1803, the loss value merging unit 1703 calculates an integrated loss value using the first loss value calculated by the first loss calculation unit 1701 and the second loss value calculated by the second loss value calculation unit 1702.

[0116] In step S405, the learning unit 207 updates the weight parameters of the NN model to reduce the integrated loss value calculated in step S1803. The processing in steps S401 to S405 is repeated until the specified conditions are met. In this way, according to this embodiment, it becomes possible to learn how to improve the overall image quality of the resulting image while also reducing the image quality detrimental patterns that occur.

[0117] [Sixth Embodiment] In the first to fifth embodiments (including variations), we described cases where image quality enhancement processing was performed using an NN model. However, the model applicable to image quality enhancement processing is not limited to NN models; other types of models may also be used.

[0118] [Seventh Embodiment] The functional parts of the image processing device 201 shown in Figures 2, 13, 17, and 19 may be implemented in hardware or in software (computer program). In the latter case, a computer device capable of executing such a computer program is applicable to the image processing device 201.

[0119] An example of a computer device hardware configuration applicable to the image processing device 201 will be explained using the block diagram in Figure 3. Note that the hardware configuration example shown in Figure 3 is merely one example of a computer device hardware configuration applicable to the image processing device 201, and can be modified or changed as appropriate.

[0120] The CPU (Central Processing Unit) 301 executes various processes using computer programs and data stored in the RAM (Random Access Memory) 304. In this way, the CPU 301 controls the operation of the entire computer system and also executes or controls the various processes described as being performed by the image processing device 201.

[0121] The ROM (Read Only Memory) 303 stores configuration data for the computer device, computer programs and data related to the startup of the computer device, computer programs and data related to the basic operation of the computer device, and so on.

[0122] RAM304 has an area for storing computer programs and data loaded from ROM303 and external memory305. RAM304 also has an area for storing computer programs and data received from external devices via I / O(Input / Output)308. Furthermore, RAM304 has a work area used by CPU301 when executing various processes. In this way, RAM304 can provide various areas as needed.

[0123] The external memory 305 is a non-volatile memory device such as a hard disk drive. The external memory 305 stores the OS (operating system), computer programs and data for the CPU 301 to execute or control various processes described as being performed by the image processing device 201, and so on. The external memory 305 can be used, for example, as the storage unit 202 described above.

[0124] The input unit 306 is a user interface such as a keyboard, mouse, or touch panel screen, which allows the user to input various instructions and information to the computer device through user operation. The input unit 306 can be used, for example, as the input unit 1901 described above.

[0125] The display unit 307 is a display device having an LCD screen or a touch panel screen, and can display the processing results of the CPU 301 as images, characters, etc. The display unit 307 may also be a projection device such as a projector that projects images or characters. Furthermore, the display unit 307 may be an LED (Light Emitting Diode) indicator. The display unit 307 can, for example, be used as the output unit 208 described above.

[0126] I / O308 functions as an interface for data communication with external devices. For example, an imaging device can be connected to I / O308 via wired or wireless connection. In this case, the computer device can acquire captured images, mosaic images, etc., from the imaging device via I / O308. The CPU301, ROM303, RAM304, external memory305, input unit306, display unit307, and I / O308 are all connected to the system bus302.

[0127] The numerical values, processing timing, processing order, processing entity, data (information) structure / acquisition method / destination / source / storage location, etc., used in the above embodiment are given as examples for the purpose of providing a concrete explanation, and are not intended to limit the scope to such examples.

[0128] Furthermore, some or all of the embodiments described above may be used in appropriate combinations. Alternatively, some or all of the embodiments described above may be used selectively.

[0129] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0130] The inventions described herein include the following image processing apparatus, image processing method, and computer program. (Item 1) A calculation means for calculating a first loss based on pixel values ​​corresponding to patterns related to image quality defects in a result image obtained by image quality enhancement processing of an input image, and pixel values ​​corresponding to patterns related to image quality defects in a target image representing the correct answer for the input image. An update means for updating the parameters of the image enhancement process based on the first loss, An image processing apparatus characterized by comprising: (Item 2) The image processing apparatus according to item 1, characterized in that the calculation means detects a pattern related to image quality defects in the resulting image by performing a filter convolution on the resulting image. (Item 3) The image processing apparatus according to item 1, characterized in that the calculation means performs filter convolution on the result image and processes the result of said convolution to remove areas of false detection of patterns related to image quality defects, thereby detecting patterns related to image quality defects in the result image. (Item 4) The image processing apparatus according to item 1, characterized in that the calculation means detects patterns related to image quality defects in the result image and the target image by performing filter convolution on the difference image between the result image and the target image. (Item 5) The image processing apparatus according to item 1, characterized in that the calculation means detects the pattern related to the image quality defects in the resulting image by performing template matching on the resulting image using an image of the pattern related to the image quality defects as a template image. (Item 6) The image processing apparatus according to item 1, characterized in that the calculation means determines which template image to apply to the result image from among a plurality of template images based on the image obtained by applying each of the plurality of template images to the result image, and detects a pattern related to image quality defects in the result image by performing template matching on the result image using the determined template image. (Item 7) The image processing apparatus according to item 1, characterized in that the calculation means detects patterns related to image quality defects in the resulting image by performing template matching on the resulting image using a template image selected from a plurality of template images according to user operation. (Item 8) The image processing apparatus according to any one of items 1 to 5, characterized in that the calculation means determines which filter to convolve with the result image from among a plurality of filters based on the image obtained by each of the plurality of filters convolving with the result image. (Item 9) The image processing apparatus according to any one of items 1 to 5, characterized in that the calculation means determines a filter selected from among a plurality of filters according to user operation as a filter to be convolved with the resulting image. (Item 10) The image processing apparatus according to any one of items 1 to 5, characterized in that the calculation means determines which filter to convolve with the result image from among a plurality of filters based on the results of component analysis of the result image. (Item 11) The image processing apparatus according to any one of items 1 to 10, characterized in that the calculation means calculates a first loss based on the pixel values ​​of the pixel group of patterns related to image quality defects in the result image and the pixel values ​​of the pixel group of patterns related to image quality defects in the target image. (Item 12) The image processing apparatus according to any one of items 1 to 10, characterized in that the calculation means calculates a first loss based on the pixel values ​​of the pixel group in the region containing the pattern related to the image quality degradation in the result image and the pixel values ​​of the pixel group in the region containing the pattern related to the image quality degradation in the target image. (Item 13) The image processing apparatus according to any one of items 1 to 10, characterized in that the calculation means calculates a first loss based on a pixel value corresponding to a pattern related to image quality defects detected from the result image and the pixel value of a pixel in the target image corresponding to the detected pattern. (Item 14) The calculation means further calculates a second loss based on the result image and the target image, The update means updates the parameters based on the first loss and the second loss. An image processing apparatus according to any one of items 1 to 13, characterized by the features described herein. (Item 15) The image processing apparatus according to item 14, characterized in that the update means determines a weight value for the first loss based on an image obtained by convolving the result image with a filter, and updates the parameter based on the sum of the result of multiplying the first loss by the determined weight value and the second loss. (Item 16) The aforementioned input image is a mosaic image of the Bayer array, The aforementioned target image is an image generated based on the aforementioned input image. An image processing apparatus according to any one of items 1 to 15, characterized by the features described herein. (Item 17) The aforementioned input image is an image taken at a relatively high ISO sensitivity, which causes noise to occur. The aforementioned image was captured at a relatively low ISO sensitivity to avoid noise generation. An image processing apparatus according to any one of items 1 to 15, characterized by the features described herein. (Item 18) The aforementioned image enhancement process is performed using a neural network. The aforementioned parameters are the weight parameters of the neural network. An image processing apparatus according to any one of items 1 to 17, characterized by the features described herein. (Item 19) An image processing method performed by an image processing device, The calculation means of the image processing device includes a calculation step of calculating a first loss based on pixel values ​​corresponding to patterns related to image quality defects in a result image obtained by image quality enhancement processing of an input image, and pixel values ​​corresponding to patterns related to image quality defects in a target image representing the correct answer for the input image. The update means for the image processing apparatus includes an update step of updating the parameters of the image enhancement process based on the first loss, and An image processing method characterized by comprising: (Item 20) A computer program for causing a computer to function as one of the means of an image processing apparatus described in any one of items 1 through 18.

[0131] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0132] 107: High-resolution processing system 201: Image processing device 202: Memory unit 203: High-resolution processing unit 204: Image acquisition unit 205: Detection unit 206: Loss calculation unit 207: Learning unit 208: Output unit

Claims

1. A calculation means for calculating a first loss based on pixel values ​​corresponding to patterns related to image quality defects in a result image obtained by image quality enhancement processing of an input image, and pixel values ​​corresponding to patterns related to image quality defects in a target image representing the correct answer for the input image. An update means for updating the parameters of the image enhancement process based on the first loss, An image processing apparatus characterized by comprising:

2. The image processing apparatus according to claim 1, characterized in that the calculation means detects a pattern related to image quality defects in the resulting image by performing a filter convolution on the resulting image.

3. The image processing apparatus according to claim 1, characterized in that the calculation means performs filter convolution on the result image and processes the result of said convolution to remove areas of false detection of patterns related to image quality defects, thereby detecting patterns related to image quality defects in the result image.

4. The image processing apparatus according to claim 1, characterized in that the calculation means detects patterns related to image quality defects in the result image and the target image by performing filter convolution on the difference image between the result image and the target image.

5. The image processing apparatus according to claim 1, characterized in that the calculation means detects the pattern related to the image quality defects in the resulting image by performing template matching on the resulting image using an image of the pattern related to the image quality defects as a template image.

6. The image processing apparatus according to claim 1, wherein the calculation means determines which template image to apply to the result image from among a plurality of template images based on the image obtained by applying each of the plurality of template images to the result image, and detects a pattern related to the image quality defects in the result image by performing template matching on the result image using the determined template image.

7. The image processing apparatus according to claim 1, characterized in that the calculation means detects patterns related to image quality defects in the resulting image by performing template matching on the resulting image using a template image selected from a plurality of template images according to user operation.

8. The image processing apparatus according to claim 1, characterized in that the calculation means determines which filter to convolve with the result image from among a plurality of filters based on the image obtained by each of the plurality of filters convolving with the result image.

9. The image processing apparatus according to claim 1, characterized in that the calculation means determines a filter selected from among a plurality of filters according to user operation as a filter to be convolved onto the result image.

10. The image processing apparatus according to claim 1, characterized in that the calculation means determines which filter to convolve with the result image from among a plurality of filters based on the results of component analysis of the result image.

11. The image processing apparatus according to claim 1, characterized in that the calculation means calculates a first loss based on the pixel values ​​of the pixel group of the pattern related to the image quality detriment in the result image and the pixel values ​​of the pixel group of the pattern related to the image quality detriment in the target image.

12. The image processing apparatus according to claim 1, characterized in that the calculation means calculates a first loss based on the pixel values ​​of the pixel group in the region containing the pattern related to the image quality degradation in the result image and the pixel values ​​of the pixel group in the region containing the pattern related to the image quality degradation in the target image.

13. The image processing apparatus according to claim 1, characterized in that the calculation means calculates a first loss based on a pixel value corresponding to a pattern related to image quality defects detected from the result image and the pixel value of a pixel in the target image corresponding to the detected pattern.

14. The calculation means further calculates a second loss based on the result image and the target image, The update means updates the parameters based on the first loss and the second loss. The image processing apparatus according to feature 1.

15. The image processing apparatus according to claim 14, characterized in that the updating means determines a weight value for the first loss based on an image obtained by convolving the filter onto the result image, and updates the parameter based on the sum of the result of multiplying the first loss by the determined weight value and the second loss.

16. The aforementioned input image is a mosaic image of the Bayer array, The aforementioned target image is an image generated based on the aforementioned input image. The image processing apparatus according to feature 1.

17. The aforementioned input image is a photograph taken at a relatively high ISO sensitivity, which causes noise to occur. The aforementioned image was taken at a relatively low ISO sensitivity to avoid noise generation. The image processing apparatus according to feature 1.

18. The aforementioned image enhancement process is performed using a neural network. The aforementioned parameters are the weight parameters of the neural network. The image processing apparatus according to feature 1.

19. An image processing method performed by an image processing device, The calculation means of the image processing device includes a calculation step of calculating a first loss based on pixel values ​​corresponding to patterns related to image quality defects in a result image obtained by image quality enhancement processing of an input image, and pixel values ​​corresponding to patterns related to image quality defects in a target image representing the correct answer for the input image. The update means for the image processing apparatus includes an update step of updating the parameters of the image enhancement process based on the first loss, and An image processing method characterized by comprising:

20. A computer program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 18.

Images