Image processing device, image processing method, and program

Abstract

It provides the display of high-resolution processed images as requested by the user. [Solution] A second image is generated by enhancing the image quality of the first image through a first image enhancement process. A third image is generated by enhancing the image quality of the first image through a second image enhancement process that is different from the first image enhancement process. Display control is performed to display at least one of the second image and the third image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Conventionally, in a camera or image editing software for a PC, etc., for raw image information (RAW image) captured by an imaging sensor, development processing such as Bayer processing or high-image-quality processing by noise removal, optical distortion correction, or image optimization is performed. As a result of performing development processing including Bayer processing or noise removal, it has been found that image quality defects such as artifacts or moire zipper noise occur in specific regions of the image. Until now, when wanting to correct these image quality defects, the user has had to change the parameter settings of the development processing and develop again from the beginning. However, depending on the rule-based method such as Bayer processing or noise removal used in the development process, or the model of the neural network, there are cases where it is difficult to remove or reduce the image quality defects that the user wants to improve.

[0003] For example, in Patent Document 1, by generating a plurality of images with different image processing effects using a single neural network model, a high-image-quality image is provided to the user.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the method disclosed in Patent Document 1, due to various factors such as the size and characteristics of the neural network model, or its compatibility with the input image, changing the image processing effect or internal parameters may have little effect on certain image quality defects. In this case, it is difficult to generate an image with the image quality defects removed or reduced.

[0006] The present invention aims to provide a display of high-resolution processed images desired by the user. [Means for solving the problem]

[0007] To achieve the object of the present invention, for example, an image processing apparatus according to one embodiment has the following configuration: a first generation means for generating a second image obtained by a first image enhancement process to enhance the image quality of a first image; a second generation means for generating a third image obtained by a second image enhancement process different from the first image enhancement process to enhance the image quality of the first image; and a display control means for performing display control to display at least one of the second image and the third image. [Effects of the Invention]

[0008] It provides the display of high-resolution processed images as requested by the user. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram showing an example of the hardware configuration of an image processing device. [Figure 2] A block diagram showing an example of the functional configuration of an image processing device. [Figure 3] A flowchart showing an example of processing by the image processing device according to Embodiment 1. [Figure 4] A diagram showing an example of a display controlled by the display control according to Embodiment 1. [Figure 5] A flowchart showing an example of processing by the image processing device according to Embodiment 2. [Figure 6] A flowchart showing an example of processing by the image processing device according to Embodiment 3. [Figure 7] A diagram showing an example of a display controlled by the display control according to Embodiment 3. [Figure 8] A flowchart showing an example of the notification process according to Embodiment 4. [Figure 9] A diagram showing an example of the display of a notification according to Embodiment 4. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] <Embodiment 1> <Hardware Configuration> Figure 1 is a block diagram showing an example of the hardware configuration of the image processing device according to this embodiment. The image processing device 100 according to this embodiment includes an imaging unit 101, RAM 102, ROM 103, display unit 104, input unit 105, and control unit 106. The hardware components of the image processing device 100 are configured to communicate with each other and are connected by a bus or the like. The image processing device 100 according to this embodiment may also have a configuration that does not include the imaging unit 101, but the following description will assume a configuration that includes the imaging unit 101. The image processing device 100 may be a device built into an imaging device such as a digital camera or digital video camera, or it may be a device built into a personal computer, mobile phone, drive recorder, robot, or drone equipped with a camera function. Furthermore, the image processing device 100 may be a device built into any electronic device that performs image quality enhancement processing, or it may be a personal computer or server device that connects to such electronic device and performs each process.

[0012] The imaging unit 101 acquires an imaging image. The imaging unit 101 according to this embodiment includes an imaging lens, an image sensor, an A / D converter, an aperture control device, and a focus control device. The imaging lens includes a fixed lens, a zoom lens, a focus lens, an aperture, and an aperture motor. The image sensor includes a CCD or CMOS that converts an optical image of a subject into an electrical signal. The A / D converter converts an analog signal into a digital signal.

[0013] The imaging unit 101 converts the subject image formed on the imaging surface of the image sensor by the imaging lens into an electrical signal, applies signal processing of A / D conversion processing to the electrical signal by the A / D converter, and supplies it to the RAM 102 as image data. The aperture control device controls the operation of the aperture motor and changes the aperture diameter of the aperture to control the aperture of the imaging lens. The focus control device controls the operation of the focus motor based on the phase difference of a pair of focus detection signals obtained from the image sensor, and drives the focus lens to control the focus state of the imaging lens. Note that the generation process and generation mechanism of the imaging image by the imaging unit 101 are not limited to such, and those in general imaging processing can be arbitrarily adopted.

[0014] The RAM 102 stores the image data obtained by the imaging unit 101 and the image data for display on the display unit 104. The RAM 102 according to this embodiment is assumed to have a storage capacity sufficient to store a predetermined number of still images and a moving image for a predetermined time. Here, the RAM 102 also serves as a memory for image display (video memory) and supplies the image data for display to the display unit 104.

[0015] The ROM 103 is a storage device such as a magnetic storage device or a semiconductor memory, and stores a program read based on the operation of the control unit 106, data that must be stored for a long time, and the like.

[0016] The display unit 104 is composed of a liquid crystal display or the like and can display various data or processing results to the user.

[0017] The input unit 105 consists of input devices such as switches, buttons, keys, or keyboards, and accepts input from the user. The input unit 105 may also function as a display unit 104, for example, a touch panel. The control unit 106 detects the input via the bus and controls each unit to perform actions corresponding to the input.

[0018] The control unit 106 is composed of a CPU (Central Processing Unit). The control unit 106 executes programs stored in the ROM 103 and realizes the functions of the image processing device 100. The control unit 106 also controls the imaging unit 101 and performs aperture control, focus control, and exposure control. For example, the control unit 106 performs AE (automatic exposure) processing, which automatically determines exposure conditions (shutter speed, storage time, aperture value, or sensitivity) based on the subject brightness information of the image data obtained by the imaging unit 101. The control unit 106 can also automatically set the focus detection area using the subject area detection result, and can realize a tracking AF processing function for any subject area. Furthermore, the control unit 106 can perform AE processing based on the brightness information of the focus detection area and perform image processing (for example, gamma correction processing or AWB (auto white balance) adjustment processing) based on the pixel values ​​of the focus detection area. The control unit 106 also performs display control by controlling the input unit 105.

[0019] <Embodiment 1> The image processing apparatus 100 according to this embodiment performs image quality enhancement processing. In particular, the image processing apparatus 100 according to this embodiment can process images that include image quality degradation (hereinafter referred to as "degraded images") and perform image quality enhancement processing. In the following, the term "image quality detriment" may be used to refer to degradation in image quality and other factors that are removed by the image quality enhancement processing.

[0020] In the following description, the image processing apparatus 100 according to this embodiment includes an imaging unit 101, and performs image enhancement processing on images acquired by the imaging unit 101. However, the images processed by the image processing apparatus 100 are not limited to those acquired by the imaging unit 101, but may be images stored in a storage device, or images acquired via wireless communication, etc. Furthermore, the imaging unit 101 does not need to be provided in the image processing apparatus 100, and images acquired by an external imaging device may be used as the processing target.

[0021] The image processing apparatus 100 according to this embodiment generates a first image and a second image, which are high-resolution versions of the image to be processed, by using a first image enhancement process and a second image enhancement process that is different from the first image enhancement process. Figure 2 is a block diagram showing an example of the functional configuration of the image processing apparatus 100 according to this embodiment. The image processing apparatus 100 shown in Figure 2 includes a first image enhancement processing unit (first processing unit) 201, a second image enhancement processing unit (second processing unit) 202, an image display unit 203, an evaluation value calculation unit 204, and a data storage unit 205.

[0022] The first processing unit 201 performs a first image enhancement process on the image to be processed. The first image enhancement process in this embodiment is a process that can generate a compressed image (JPG, PNG, etc.) which is a first image enhancement processed image from a RAW image which is the input image. For example, the first image enhancement process can be noise reduction, debayering, or aberration correction. In this embodiment, the first image enhancement process is described below as a process that corrects such an input image.

[0023] The second processing unit 202 performs a second image enhancement process on the image to be processed. The second image enhancement process in this embodiment is a process that can generate a new compressed image (JPG, PNG, etc.) which is an enhanced image from a compressed input image (JPG, PNG, etc.) using a generative AI such as a GAN or a diffusion model. In the following, an image enhancement process that does not use a generative AI may be referred to as a "normal image enhancement process" in contrast to an image enhancement process that uses a generative AI. In this embodiment, the second image enhancement process is described below as a process that generates an image using such a generative AI without correcting the input image.

[0024] The image display unit 203 performs processing to display the input image or the image that has undergone high-resolution processing on the display unit 104, or performs processing to switch the image to be displayed.

[0025] The evaluation value calculation unit 204 calculates an evaluation value (such as SSIM or PSNR), which will be described later, using the input image and the image that has been processed for high image quality, or using only the image that has been processed for high image quality.

[0026] The data storage unit 205 stores images. The data storage unit 205 can store, for example, images before image enhancement processing and images after image enhancement processing.

[0027] First, we will explain an example of image enhancement processing using a generation AI, such as a diffusion model, by the second processing unit 202 according to this embodiment. In the image generation process using the diffusion model, a diffusion process is performed to add noise to the input image, and a dediffusion process is performed to remove the noise. In the dediffusion process, an image is generated based on training data and generation conditions such as the input image, text, or color. In the diffusion process, information about image quality defects that were present in the image before processing may be lost by adding noise once, but since the training data and the surrounding areas of the image quality defects that are referenced in the dediffusion process basically do not contain image quality defects, the possibility of actively reproducing image quality defects during generation is low.

[0028] The generation conditions in this embodiment may be information input as text, information input as an image, or any other information that is input when generating an image in the generation AI.

[0029] More specifically, the second processing unit 202 in this embodiment can perform image quality enhancement processing using a technique called Inpaint. Inpaint is a technique that generates an image only of the areas not masked by inputting an input image and a mask image that indicates the areas of the input image from which image quality defects are to be removed into a generation AI. When generating the areas not masked, it is possible to perform image quality enhancement processing that reproduces continuous patterns or colors by using the color and edge information of the masked area, thereby reducing the sense of incongruity between the inside and outside of the masked area in the generated image. In addition, since the diffusion model can be given generation conditions, by adding edge information of the areas not masked, it is possible to generate an image while maintaining structural information such as patterns and shapes that exist only in the areas not masked. The second processing unit 202 can perform such image quality enhancement processing using the diffusion model.

[0030] Furthermore, the difference between the first image enhancement process performed by the first processing unit 201 and the second image enhancement process performed by the second processing unit 202 (in this case, image enhancement process using generation AI) will also be explained.

[0031] Here, when generating high-resolution processed images, no generation conditions are applied in the first high-resolution processing, while generation conditions are applied in the second high-resolution processing. In the high-resolution processing using the generation AI according to this embodiment, by applying generation conditions, it is possible to generate high-resolution processed images while taking into account information such as text, color, or edges that are applied separately from the input image and parameter settings.

[0032] In addition, I will explain three differences between the first and second image enhancement processes.

[0033] The first difference is that conventional image enhancement processing is less likely to affect the structural information of the input image compared to image enhancement processing using a generation AI. In this embodiment, structural information refers to, for example, the shape, color, or pattern of an object, or the outline or hairstyle of a person's face. In the second image enhancement processing according to this embodiment, the image is generated from scratch by the generation AI, so the structural information of the object may differ only in the area of ​​the image generated by the generation AI.

[0034] The second difference lies in the input image size. In typical image enhancement processes, the input image size is the patch size. The patch size is the size of one image when the image is divided into manageable units (for example, because a large image cannot be processed at once). On the other hand, in the image enhancement process using the generation AI according to this embodiment, a region larger than the patch size, including the area surrounding the patch, is used. This is because, when referencing surrounding information during the process of reshaping areas with image quality defects, some information from such surrounding areas is necessary.

[0035] The third difference is that the standard image enhancement process is less prone to hallucination than the image enhancement process using the generation AI. In this embodiment, hallucination refers to the phenomenon of generating false information that is not based on facts, such as generating a person or object that did not exist in the original image. Thus, the first image enhancement process in this embodiment can be considered a process that has a lower frequency of hallucination after processing than the second image enhancement process.

[0036] Figure 3 is a flowchart showing an example of image processing performed by the image processing device 100 according to this embodiment. The processing shown in Figure 3 is started, for example, when an input image to be processed is input (for example, when imaging is started by the user, the captured image is used as the input image), and each process is performed by the control unit 106.

[0037] In S301, the control unit 106 sets a partial region of the input image in which it wants to correct or reduce image quality defects such as noise, blurred lines, artifacts, moiré patterns, or zipper noise. Here, the control unit 106 may set the partial region based on user input, for example, or based on the input image. For example, the control unit 106 may extract one or more candidate regions from the input image using a predetermined detection algorithm, and set a partial region to be processed from among the extracted candidate regions according to the magnitude of the value or the size of the region. Hereinafter, the explanation will be given assuming that a partial region is the target of processing, but the entire input image may be the target of the image quality enhancement processing.In the following, when simply referred to as "partial region," it refers to the partial region that is set to be processed in this way, and when referred to as "partial image," it refers to the image within such a partial region.

[0038] This section describes one of the predetermined detection algorithms used by the control unit 106 to detect candidate regions, specifically one that detects zipper noise or artifacts. For example, the control unit 106 can use a Sobel filter or the Canny method to detect edges in the image and determine that regions with a high density of pixels exceeding a threshold are regions where zipper noise or artifacts occur. Furthermore, when a sub-region is defined based on user input, the user may input the range to be designated as a sub-region on the image, or the user may select a sub-region from among the candidate regions extracted by the control unit 106.

[0039] In S302, the control unit 106 generates a first image (first high-resolution processed image) by performing a first high-resolution processing on a partial region of the image, and a second image (second high-resolution processed image) by performing a second high-resolution processing. Here, a high-resolution processed image is defined as an image obtained by combining a high-resolution processed partial image, generated by performing a predetermined high-resolution processing on a partial image, with the input image. Furthermore, it is sufficient for one or more first high-resolution processed images and second high-resolution processed images to be generated, but the number is not particularly limited. Here, the number of first high-resolution processed images and second high-resolution processed images to be generated can be set in advance by the user, and it is assumed that multiple images of each are generated.

[0040] In S303, the evaluation value calculation unit 204 calculates an evaluation value for the image enhancement process using the input image and at least one of the first and second image enhancement processed images generated in S302. The evaluation value here is an evaluation value that assesses the smallness of the image change in the image enhancement process of the image enhancement processed image, and an evaluation value of the image enhancement processed image relative to the input image is used, for example, SSIM (Structural SIMilarity) (similarity). SSIM is one of the evaluation indices that assess the degree to which the structural information between the input image and the image enhancement processed image is similar. The evaluation value calculation unit 204 can calculate SSIM using, for example, the following formula (1). This SSIM is a value between 0 and 1, and the closer it is to 0, the less similar the structure is between the input image and the image enhancement processed image, and the closer it is to 1, the more similar the structure is.

number

[0041] Here, the input image is shown as image x, and the image after high-resolution processing is shown as image y. Also, μ x μ is the average pixel value of image x. y σ is the average pixel value of image y. xσ is the standard deviation of the pixel values ​​in image x. y σ is the standard deviation of the pixel values ​​in image y. xy is the covariance between x and y, and C1 and C2 are constants (particularly for stabilizing the output value when the denominator value becomes small).

[0042] Other evaluation values ​​besides SSIM include PSNR (Peak Signal to Noise Ratio) for evaluating image quality, or SIFT for evaluating structural information similarity, similar to SSIM. The evaluation value calculation unit 204 can calculate PSNR using, for example, the following equation (2). In this embodiment, the PSNR is calculated as the PSNR of the input image relative to the high-resolution processed image. Here, a larger PSNR indicates less image degradation, and a smaller PSNR indicates more image degradation.

number

[0043] Here, N is the number of pixels, Y i Y^ is the number of pixels in the high-resolution processed image. i is the pixel value of the input image, and MAX is the maximum brightness value.

[0044] Furthermore, the evaluation value may be calculated using the input image and the high-resolution processed image as described above, or it may be calculated using only the high-resolution processed image. Examples of evaluation values ​​calculated using only the high-resolution processed image include contrast, noise level, or the number of detected objects; all evaluation metrics commonly used in image processing and image recognition are applicable. If the entire image is processed in S301, the evaluation value is calculated from the entire image.

[0045] In S304, the image display unit 203 performs display control to display at least one of the first high-resolution processed image and the second high-resolution processed image, and terminates the process shown in Figure 3. Here, the image display unit 203 performs display control of the first high-resolution processed image and the second high-resolution processed image based on the evaluation value calculated in S303. For example, the image display unit 203 can display the first high-resolution processed image and the second high-resolution processed image in a display order based on the evaluation value.

[0046] Figure 4(a) shows an example of displaying the first and second high-resolution processed images together in descending order of similarity, without distinguishing between them, using similarity as the evaluation value. This display method is suitable for selecting the image most similar to the input image from all high-resolution processed images. Here, a sorting method that displays from left to right in descending order of evaluation value is exemplified, but the display order may also be ascending order of evaluation value, generation order, image file size order, etc., and the display may be based on a sorting method arbitrarily added or set by the user. Furthermore, it is not necessary to sort based on a single evaluation value, and sorting may be performed using the result of combining multiple evaluation values. For example, the image display unit 203 can sort images in order of structural similarity and low noise level by combining SSIM and noise level. In addition, in the display example shown in Figure 4(a), the first and second high-resolution processed images are displayed together, but it is also possible to display only the first (normal) high-resolution processed image, or only the second (generated using generation AI) high-resolution processed image. This type of display processing allows users to select an enhanced image that does not differ significantly from the input image.

[0047] Figure 4(b) shows an example of distinguishing between a first high-resolution processed image and a second high-resolution processed image using similarity as an evaluation value, and displaying them separately in descending order of similarity. Here, the first display, which shows the first high-resolution processed images in descending order, and the second display, which shows the second high-resolution processed images in descending order, are displayed side by side. This display method is suitable for comparing the first (normal) high-resolution processed image and the second (high-resolution processed image generated using a generation AI) high-resolution processed image. Furthermore, such a display allows for verification based on the trade-off relationship that normal high-resolution processing has limitations in improving image quality defects but does not significantly change the input image, while high-resolution processing using a generation AI may significantly change the input image but can generate an image without image quality defects. With this display processing, users can select the desired high-resolution processed image based on the trade-off between the effects of the first high-resolution processing and the second high-resolution processing.

[0048] With this configuration, it is possible to generate a first high-resolution processed image and a second high-resolution processed image, and to perform display control to display at least one of them. In particular, it becomes possible to control the display of a normal high-resolution processed image and a high-resolution processed image using generation AI based on their evaluation values. Therefore, it becomes possible to provide a display that allows the user to select an image with an evaluation value similar to the input image.

[0049] <Embodiment 2> In Embodiment 1, a first image enhancement process and a second image enhancement process were performed on the input image, and then the display control of the resulting image group was performed. On the other hand, the image processing device 100 according to Embodiment 2 performs the display of the first image enhancement processed image and the display of the second image enhancement processed image in stages. The image processing device 100 according to this embodiment has the same configuration as the configuration in Embodiment 1 and can perform the same processing, so redundant explanations will be omitted.

[0050] The image processing device 100 according to this embodiment performs display control to display a second high-resolution processed image if the evaluation value of the first high-resolution processed image does not meet predetermined conditions. For example, the image processing device 100 may generate a first high-resolution processed image and display the first high-resolution processed image if the evaluation value of the first high-resolution processed image meets predetermined conditions. Here, it is assumed that the second high-resolution processed image is generated and displayed if the evaluation value of the first high-resolution processed image does not meet predetermined conditions. However, similar to Embodiment 1, a first high-resolution processed image and a second high-resolution processed image may be generated first, and display control may be performed so that the first high-resolution processed image is displayed if the evaluation value of the first high-resolution processed image meets predetermined conditions, and the second high-resolution processed image is displayed otherwise. Here, if the evaluation value of the first high-resolution processed image does not meet the predetermined conditions, the display of the second high-resolution processed image may consist of displaying only the second high-resolution processed image, or it may be displayed alongside the first high-resolution processed image.

[0051] The following describes an example of image processing performed by such an image processing device 100. Figure 5 is a flowchart showing an example of image processing performed by the image processing device 100 according to this embodiment. The processing shown in Figure 5 is started, for example, when an input image to be processed is input (for example, when imaging is started by the user, the captured image is used as the input image), and each process is performed by the control unit 106. S501 is the same process as S301 in Embodiment 1, so its explanation is omitted here.

[0052] In S502, which follows S501, the first processing unit 201 generates a first high-resolution processed image by performing a first high-resolution processing on the image of a partial region.

[0053] In steps S503 to S507, the image processing device 100 controls the display of the first high-resolution processed image and the second high-resolution processed image. The display control according to this embodiment will be described below.

[0054] In S503, the image display unit 203 displays the first high-resolution processed image generated in S502 on the display unit 104. The display here is the same as the process described with reference to Figure 4(b) of Embodiment 1, excluding the display of the second high-resolution processed image.

[0055] In S504, the evaluation value calculation unit 204 determines whether the first high-resolution processed image generated in S502 (or displayed in S503) satisfies predetermined conditions. Here, the evaluation value calculation unit 204 can determine that the first high-resolution processed image satisfies predetermined conditions if, for example, SSIM or PSNR is used as an evaluation value and such evaluation value is above (or below) a desired threshold. Alternatively, in addition to displaying the first high-resolution processed image in S503, the evaluation value calculation unit 204 may also display a UI (for example, a button) that accepts user input on whether the displayed image has the desired image quality, and determine whether the predetermined conditions are met based on the user input to the UI. If it is determined that the first high-resolution processed image satisfies predetermined conditions, the process in Figure 5 ends; otherwise, the process proceeds to S505.

[0056] In S505, the second processing unit 202 generates a second high-resolution image by performing a second high-resolution processing on the image of a partial region.

[0057] In S506, the image display unit 203 displays the second high-resolution processed image generated in S505 on the display unit 104, and the process shown in Figure 5 is completed. The display here may be the same as the process described with reference to Figure 4(b) of Embodiment 1, excluding the display of the first high-resolution processed image, or the first high-resolution processed image and the second high-resolution processed image may be displayed side by side.

[0058] This processing allows for display control to be performed so that a second high-resolution image is displayed if the first high-resolution image does not meet predetermined conditions. Therefore, it becomes possible to present the image desired by the user without necessarily having the user review all the high-resolution images. Furthermore, it becomes possible to reduce the user's waiting time.

[0059] <Embodiment 3> In the image processing apparatus 100 according to Embodiment 1, as an example of calculating an evaluation value, a process was described in which an evaluation value of the high-resolution processed image is calculated by comparing the input image with the high-resolution processed image. In the image processing apparatus 100 according to Embodiment 3, when the pixel values ​​of the input image (here, pixel values ​​related to image quality defects) satisfy predetermined image quality conditions, the first high-resolution processed image is used instead of the input image in calculating the evaluation value. The image processing apparatus 100 according to this embodiment has the same configuration as the configuration in Embodiment 1 and can perform the same processing, so redundant explanations are omitted.

[0060] The following describes an example of image processing performed by such an image processing device 100. Figure 6 is a flowchart showing an example of image processing performed by the image processing device 100 according to this embodiment. The processing shown in Figure 6 is started, for example, when an input image to be processed is input (for example, when imaging is started by the user, the captured image is used as the input image), and each process is performed by the control unit 106. Since S601 to S602 and S606 are the same processes as S301 to S302 and S304 in Embodiment 1, their explanation is omitted here.

[0061] In S603, following S602, the evaluation value calculation unit 204 determines whether the pixel values ​​of the input image meet the predetermined image quality conditions. If the pixel values ​​of the input image meet the predetermined image quality conditions, the process proceeds to S604; otherwise, the process proceeds to S605. Here, the evaluation value calculation unit 204 can determine, based on the pixel values ​​of the input image, whether the image quality defects occurring in the input image are greater than or equal to the amount (threshold) that satisfies the predetermined image quality conditions. For example, the evaluation value calculation unit 204 may use the noise amount as the image quality defect, and use the difference between the original input image and the filtered image obtained by simply removing noise using the standard deviation of the luminance value of the input image or a Gaussian filter as the noise amount, and if the noise amount is greater than or equal to a predetermined value, the predetermined image quality conditions are considered to be met. Alternatively, for example, blur may be used as the image quality defect, and the determination of whether the predetermined image quality conditions are met may be made based on the edge intensity generated by edge detection using a Sobel filter or the like from the input image, or the determination of whether the predetermined image quality conditions are met may be made based on the proportion of high-frequency components obtained by frequency analysis of the input image. For example, the evaluation value calculation unit 204 may use a machine learning model that takes an image as input and outputs an evaluation value of image defects (e.g., noise level) in the input image to determine whether the image quality defects are greater than or equal to a predetermined quality condition based on the output evaluation value of image defects. Furthermore, for example, the evaluation value calculation unit 204 may use a machine learning model that has been trained to output whether or not the image quality defects in the input image are greater than or equal to a predetermined quality condition to perform this determination. In this way, the image quality defects evaluated based on pixel values ​​can be set as desired, and the threshold value used as the predetermined quality condition can be arbitrarily set by the user. In this way, as the predetermined quality condition, for example, it is possible to use a value obtained by evaluating the noise level of the image that is greater than or equal to a predetermined threshold.

[0062] As described above, the image processing apparatus 100 according to this embodiment can use a first high-resolution processed image instead of the input image in Embodiment 1, and calculate an evaluation value of other high-resolution processed images based on the first high-resolution processed image and other high-resolution processed images. Hereinafter, the first high-resolution processed image used instead of the input image will be referred to as the "reference image".

[0063] In S604, the evaluation value calculation unit 204 selects a reference image from the first high-resolution processed images. Here, the evaluation value calculation unit 204 may select, for example, the first high-resolution processed image that was judged to have the lowest evaluation value of image quality defects (e.g., noise level) used in S603, or it may select a first high-resolution processed image selected based on user input. By selecting a reference image from the first high-resolution processed images, as described above, it becomes possible to select a reference image from among images in which structural information has changed little with respect to the input image and the possibility of hallucination occurring is low.

[0064] In S605, an evaluation value of the high-resolution processed image is calculated based on the reference image and the high-resolution processed image (excluding the reference image). This process can be performed in the same way as in S303 of Embodiment 1, except that the reference image is used instead of the input image, so redundant explanations are omitted. In this case, if it is determined in S603 that the predetermined image quality conditions are not met, the input image is used as the reference image.

[0065] Figure 7 shows an example of the display of the first high-resolution processed image and the second high-resolution processed image when the evaluation value (similarity) is calculated using a reference image instead of the input image. In Figure 7, the display is the same as in Figure 4(a) of Embodiment 1, except that the reference image is additionally displayed. However, the first high-resolution processed image and the second high-resolution processed image may be displayed separately, as shown in Figure 4(b).

[0066] This process allows for the calculation of evaluation values ​​for other enhanced images using a first enhanced image as a reference image, provided that the input image meets the specified image quality requirements. Therefore, if an input image has a significant amount of image quality degradation, it is possible to enhance the image quality of such input image, set a reference image that is close to the original image (where hallucination and other issues are less likely to have occurred), and then display enhanced images that are close to that reference image. Consequently, it becomes possible to present users with images that have had image quality degradation removed or reduced and whose evaluation values ​​are similar to those images.

[0067] <Embodiment 4> The image processing apparatus 100 according to Embodiment 4 has the same configuration as the one in Embodiment 1 and can perform the same processing, so redundant explanations will be omitted. Furthermore, the image processing apparatus 100 according to this embodiment determines whether the pixel values ​​of an image to be processed (for example, an image stored in the data storage unit 205) meet predetermined image quality conditions. If the predetermined image quality conditions are met, the image processing apparatus 100 then notifies the user that it recommends performing image quality enhancement processing on the image.

[0068] The following describes an example of image processing performed by such an image processing device 100. Figure 8 is a flowchart showing an example of a determination process performed by the image processing device 100 according to this embodiment to determine whether or not to notify the user that it is recommended to perform image quality enhancement processing on the image. The processes shown in Figure 8 are started, for example, at a predetermined timing when imaging is not being performed, when sleep operation is performed, when charging is performed, or at a timing based on input from the user for a start operation, and each process is performed by the control unit 106.

[0069] First, a loop process from S801 to S802 is initiated, selecting one of the candidate images to be processed (here, assumed to be a group of images stored in the data storage unit 205) as the image to be processed. In S801, the evaluation value calculation unit 204 determines whether the pixel values ​​of the image to be processed meet predetermined image quality conditions. The determination process performed in S801 is basically the same as that performed in S603, but the threshold used may be different from that in S603. If the predetermined image quality conditions are met, the process proceeds to S802. If the predetermined image quality conditions are not met, the process proceeds to S803 if all of the candidate images to be processed have already been selected for processing; otherwise, an image that has not yet been selected for processing is chosen, and the process returns to S801.

[0070] In S802, the evaluation value calculation unit 204 sets the image to be processed as an image for which high-quality processing is recommended. Next, if all of the candidate images to be processed have been selected for processing, the evaluation value calculation unit 204 proceeds to S803; otherwise, it selects an image that has not yet been selected for processing and returns to S801.

[0071] In S803, the image display unit 203 issues a notification regarding the image that was set in S802 as an image for which high-quality processing is recommended, and terminates the process shown in Figure 8. This notification may be a notification indicating that there is an image for which high-quality processing is recommended, a notification indicating the number of images for which high-quality processing is recommended, or a notification displaying an image for which high-quality processing is recommended.

[0072] Figure 9 shows an example of a notification displayed by the image display unit 203. In Figure 9, the notification displays the number of images for which high-resolution processing is recommended, and a UI that allows the user to choose whether or not to review those images.

[0073] This process allows for notifications recommending image enhancement for images whose pixel values ​​meet predetermined image quality requirements. Therefore, it reduces the user's need for visual inspection.

[0074] The disclosures herein include the following image processing apparatus, image processing methods, and programs. (Item 1) A first generation means that generates a second image obtained by improving the image quality of a first image through a first image enhancement process, A second generation means generates a third image in which the first image has been enhanced in quality by a second image enhancement process different from the first image enhancement process, A display control means that performs display control to display at least one of the second image and the third image, An image processing device equipped with the following features. (Item 2) The system further comprises calculation means for calculating a first evaluation value for evaluating the smallness of the image change in the second image during the first image enhancement process, and a second evaluation value for evaluating the smallness of the image change in the third image during the second image enhancement process. The image processing apparatus according to item 1, characterized in that the display control means performs display control to display at least one of the second image and the third image based on the first evaluation value and the second evaluation value. (Item 3) The image processing apparatus according to item 2, characterized in that the first evaluation value is the similarity between the first image and the second image, and the second evaluation value is the similarity between the first image and the third image. (Item 4) The image processing apparatus according to item 2, characterized in that the display control means performs display control to display the second image and the third image in an order based on the first evaluation value and the second evaluation value. (Item 5) The image processing apparatus according to item 4, characterized in that the display control means performs display control to display the second image and the third image in descending order based on the first evaluation value and the second evaluation value. (Item 6) The image processing apparatus according to any one of items 2 to 5, characterized in that the display control means performs display control so as to display the second image and the third image without distinguishing between them. (Item 7) The image processing apparatus according to any one of items 2 to 5, characterized in that the display control means performs display control to distinguish between the second image and the third image when displaying them. (Item 8) The image processing apparatus according to any one of items 2 to 7, characterized in that the display control means displays the second image when the first evaluation value satisfies a predetermined condition, and displays the third image when the first evaluation value does not satisfy the predetermined condition. (Item 9) The image processing apparatus according to item 8, wherein the second generation means is characterized in that it generates the third image when the second evaluation value does not satisfy the predetermined conditions. (Item 10) The image processing apparatus according to item 8, characterized in that the first evaluation value is the similarity between the first image and the second image, and the predetermined condition is that the similarity is greater than or equal to a predetermined threshold. (Item 11) A determination means for determining whether the first image satisfies predetermined image quality conditions, The system further includes a selection means for selecting one of the image groups generated by the first image enhancement process as a reference image if the first image satisfies predetermined image quality conditions. Furthermore, The calculation means is If the first image satisfies the predetermined image quality conditions, the first evaluation value is calculated as the similarity between the reference image and the second image, and the second evaluation value is calculated as the similarity between the reference image and the third image. An image processing apparatus according to any one of items 2 to 10, characterized in that, if the first image does not meet predetermined image quality conditions, the first evaluation value is calculated as the similarity between the first image and the second image, and the second evaluation value is calculated as the similarity between the first image and the third image. (Item 12) The image processing apparatus according to item 11, characterized in that the predetermined image quality condition is that the evaluation value obtained by evaluating the noise amount of the first image is greater than or equal to a predetermined threshold. (Item 13) The image processing apparatus according to item 11, wherein the selection means selects the image with the smallest noise level evaluation value from among the group of images generated by the first image enhancement process as the reference image. (Item 14) The image processing apparatus according to any one of items 1 to 13, characterized in that the first image enhancement process is a process of correcting the first image to generate a second image, and the second image enhancement process is a process of generating a third image without correcting the first image. (Item 15) The image processing apparatus according to item 14, characterized in that the first image enhancement process is a process that causes a lower frequency of hallucination after processing than the second image enhancement process. (Item 16) The image processing apparatus according to any one of items 1 to 14, characterized in that the first image enhancement process is a process that enhances the image quality of an image of patch size as input, and the second image enhancement process is a process that enhances the image quality of an image larger than the patch size image, including the image of patch size, as input. (Item 17) A determination means for determining whether the fourth image satisfies predetermined image quality conditions, An image processing apparatus according to any one of items 1 to 16, further comprising: a notification means for providing a notification recommending the execution of image quality enhancement processing for the fourth image when the fourth image satisfies predetermined image quality conditions. (Item 18) The process involves generating a second image by performing a first image enhancement process, which enhances the image quality of the first image. A step of generating a third image in which the first image has been enhanced in quality by a second image enhancement process different from the first image enhancement process, A step of performing display control to display at least one of the second image and the third image, An image processing method comprising: (Item 19) A program for causing a computer to function as one of the means of an image processing apparatus described in any one of items 1 through 17.

[0075] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0076] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0077] 100: Image processing device, 101: Imaging unit, 102: RAM, 103: ROM, 104: Display unit, 105: Input unit, 106: Control unit

Claims

1. A first generation means that generates a second image obtained by improving the image quality of a first image through a first image enhancement process, A second generation means generates a third image obtained by improving the image quality of the first image through a second image quality improvement process different from the first image quality improvement process, A display control means that performs display control to display at least one of the second image and the third image, An image processing device equipped with the following features.

2. The system further comprises a calculation means for calculating a first evaluation value for evaluating the smallness of the image change in the second image during the first image enhancement process, and a second evaluation value for evaluating the smallness of the image change in the third image during the second image enhancement process. The image processing apparatus according to claim 1, characterized in that the display control means performs display control to display at least one of the second image and the third image based on the first evaluation value and the second evaluation value.

3. The image processing apparatus according to claim 2, characterized in that the first evaluation value is the similarity between the first image and the second image, and the second evaluation value is the similarity between the first image and the third image.

4. The image processing apparatus according to claim 2, characterized in that the display control means performs display control to display the second image and the third image in an order based on the first evaluation value and the second evaluation value.

5. The image processing apparatus according to claim 4, characterized in that the display control means performs display control to display the second image and the third image in descending order based on the first evaluation value and the second evaluation value.

6. The image processing apparatus according to claim 2, characterized in that the display control means performs display control so as to display the second image and the third image without distinguishing between them.

7. The image processing apparatus according to claim 2, characterized in that the display control means performs display control to distinguish between the second image and the third image when displaying them.

8. The image processing apparatus according to claim 2, characterized in that the display control means displays the second image when the first evaluation value satisfies a predetermined condition, and displays the third image when the first evaluation value does not satisfy the predetermined condition.

9. The image processing apparatus according to claim 8, characterized in that the second generation means generates the third image when the second evaluation value does not satisfy the predetermined conditions.

10. The image processing apparatus according to claim 8, characterized in that the first evaluation value is the similarity between the first image and the second image, and the predetermined condition is that the similarity is greater than or equal to a predetermined threshold.

11. A determination means for determining whether the first image satisfies predetermined image quality conditions, The system further includes a selection means for selecting one of the image groups generated by the first image enhancement process as a reference image if the first image satisfies predetermined image quality conditions. Furthermore, The calculation means is If the first image satisfies the predetermined image quality conditions, the first evaluation value is calculated as the similarity between the reference image and the second image, and the second evaluation value is calculated as the similarity between the reference image and the third image. The image processing apparatus according to claim 2, characterized in that, if the first image does not meet the predetermined image quality conditions, the first evaluation value is calculated as the similarity between the first image and the second image, and the second evaluation value is calculated as the similarity between the first image and the third image.

12. The image processing apparatus according to claim 11, characterized in that the predetermined image quality condition is that the evaluation value obtained by evaluating the noise amount of the first image is greater than or equal to a predetermined threshold.

13. The image processing apparatus according to claim 11, characterized in that the selection means selects the image with the smallest noise level evaluation value from among the group of images generated by the first image enhancement process as a reference image.

14. The image processing apparatus according to claim 1, characterized in that the first image enhancement process is a process of correcting the first image to generate a second image, and the second image enhancement process is a process of generating a third image without correcting the first image.

15. The image processing apparatus according to claim 14, characterized in that the first image enhancement process is a process that causes a lower frequency of hallucination after processing than the second image enhancement process.

16. The image processing apparatus according to claim 1, characterized in that the first image enhancement process is a process that enhances the image quality of an image of patch size as input, and the second image enhancement process is a process that enhances the image quality of an image larger than the patch size image, including the image of patch size, as input.

17. A determination means for determining whether a fourth image satisfies predetermined image quality conditions, The image processing apparatus according to claim 1, further comprising: notification means for providing notification that it is recommended to perform image quality enhancement processing on the fourth image when the fourth image satisfies predetermined image quality conditions.

18. A first image enhancement process generates a second image by enhancing the image quality of the first image, A step of generating a third image by performing a second image enhancement process, which is different from the first image enhancement process, on the first image enhancement process, A step of performing display control to display at least one of the second image and the third image, An image processing method comprising:

19. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 17.

Images