Image processing device, imaging apparatus, control method, and program

Abstract

To provide a technology that enables a user to confirm the noise reduction effect at an early stage.SOLUTION: An image processing device includes selection means for selecting a local area from a first image displayed on a display unit, processing means for generating a second image indicating the local area from the first image and performing high-quality enhancement processing on the second image, and control means for displaying on the display unit a third image on which the high-quality enhancement processing has been performed by the processing means.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, a control method, and a program.

Background Art

[0002] Conventionally, in an imaging apparatus such as a camera, raw image information (RAW image) captured by an imaging sensor is subjected to debayering processing (demosaicing processing) and converted into a signal composed of luminance and color difference. Then, so-called development processing such as high-image-quality processing, optical distortion correction, and image optimization is performed on each signal.

[0003] When shooting is performed in an environment containing many low-luminance areas, the sensitivity of the imaging sensor of the imaging apparatus may be set high for shooting. However, when shooting is performed with such a high-sensitivity setting, noise is likely to occur in the image. Therefore, it is required to improve the processing performance for high image quality, particularly noise reduction.

[0004] In a technique for generating an image with high image quality by performing noise reduction, a high-image-quality processing method using noise reduction (hereinafter referred to as NR) is known. However, to perform high-precision NR processing, a huge amount of calculation is required, so the processing takes time.

[0005] On the other hand, Patent Document 1 discloses shortening the processing time by separating the image processing method before and after shooting. Specifically, during the shooting operation, since it is necessary to perform image processing at high speed to check the image, high-image-quality processing that emphasizes speed is performed by performing simple image processing. And when the image is displayed and checked after shooting, a method is adopted in which more load-intensive image processing is performed to emphasize high image quality. Thereby, during the shooting operation, it is possible to confirm to what extent the noise in the image captured by the imaging apparatus can be reduced.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-179851 [Summary of the Invention] [Problems to be Solved by the Invention]

[0007] However, in the technology described in Patent Document 1, since the high-image-quality processing before and after shooting is different, it is difficult to confirm whether a desired image can be captured during the shooting operation. That is, there is a problem that it is difficult for the user to confirm the noise reduction effect at an early stage.

[0008] The present invention has been made in view of the above problems, and an object thereof is to provide a technology for enabling a user to confirm a noise reduction effect at an early stage. [Means for Solving the Problems]

[0009] An image processing apparatus according to the present invention for achieving the above object includes: selection means for selecting a local area from a first image being displayed on a display unit; processing means for generating a second image indicating the local area from the first image and performing high-image-quality processing on the second image; control means for causing the display unit to display a third image on which the high-image-quality processing has been performed by the processing means; and is characterized by comprising. [Effects of the Invention]

[0010] According to the present invention, it becomes possible for a user to confirm a noise reduction effect at an early stage. [Brief Description of the Drawings]

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and duplicate explanations are omitted.

[0013] (Embodiment 1) Hereinafter, embodiments of applying the present invention to a camera will be described. However, the present invention can be implemented in any electronic device that performs high-quality processing in a moving image. Such electronic devices include, but are not limited to, imaging devices such as digital cameras and digital video cameras, as well as personal computers, mobile phones, drive recorders, robots, drones, etc. equipped with a camera function.

[0014] <Hardware Configuration> FIG. 1 shows an example of the hardware configuration of the camera according to this embodiment. The camera 100 according to this embodiment includes an imaging device 101, a RAM 102, a ROM 103, an image processing device 104, an input / output device 105, and a control device 106. Each device is configured to be communicable with each other and is connected by a bus or the like. In addition, in this embodiment, a configuration in which the camera 100 includes the imaging device 1010 is shown, but the entire camera 100 may be referred to as an imaging device.

[0015] The imaging device 101 is composed of an imaging lens, an imaging element, an A / D converter, a diaphragm control device, a focus control device, etc. The imaging lens includes a fixed lens, a zoom lens, a focus lens, a diaphragm, and a diaphragm motor. The imaging element includes a CCD, a CMOS, etc. that convert the optical image of the subject into an electrical signal.

[0016] The A / D converter converts an analog signal into a digital signal. The imaging device 101 converts the subject image formed on the imaging surface of the imaging element by the imaging lens into an electrical signal, applies A / D conversion processing to the electrical signal by the A / D converter, and supplies it to the RAM 102 as image data. A live view display can be performed by sequentially transferring the image data to the input / output device 105 and displaying it. The input / output device 105 may be the rear monitor provided in the camera 100. The live view can be displayed in a still image shooting standby state, a moving image shooting standby state, during moving image recording, etc., and the captured subject image is displayed almost in real time.

[0017] The aperture control device controls the operation of the aperture motor and changes the aperture diameter of the aperture to control the aperture of the imaging lens. The focus control device controls the operation of the focus motor based on the phase difference between a pair of focus detection signals obtained from the imaging device, and controls the focus state of the imaging lens by driving the focus lens.

[0018] RAM 102 stores the image data obtained by the imaging device 101 and the image data for display on the input / output device 105. RAM 102 has a storage capacity sufficient to store a predetermined number of still images or a moving image for a predetermined time. Further, RAM 102 also serves as a memory for image display (video memory) and supplies the image data for display to the input / output device 105.

[0019] ROM 103 is a storage device such as a magnetic storage device or a semiconductor memory, and stores various programs and data that need to be stored for a long time. The image processing device 104 performs high-image-quality processing for reducing noise and improving the image quality on an image in which noise has occurred. Details of the configuration and operation of the image processing device 104 will be described later.

[0020] The input / output device 105 is composed of an input device group such as switches, buttons, keys, and touch panels for the user to input instructions to the camera 100, and a display such as an LCD or an organic EL display. Inputs through the input device group are detected by the control device 106 through the bus, and the control device 106 controls each part to realize an operation corresponding to the input. Further, in the input / output device 105, the touch detection surface of the touch panel is the display surface of the display. The touch panel can use various types of touch panels such as a low-hard film method, a capacitance method, and an optical sensor method. The input / output device 105 displays a live view image by sequentially transferring and displaying the image data.

[0021] The control device 106 includes one or more CPUs (Central Processing Units). The control device 106 executes the programs stored in the ROM 103 to realize each function of the camera 100. Further, the control device 106 controls the imaging device 101 to perform aperture control, focus control, exposure control, etc. For example, it executes AE (Automatic Exposure) processing to automatically determine exposure conditions (shutter speed or integration time, aperture value, sensitivity) based on the subject luminance information of the image data obtained by the imaging device 101.

[0022] In addition, the control device 106 can reduce the noise generated in shooting with a high ISO sensitivity setting by using the noise reduction processing result during high sensitivity setting by the image processing device 104. Also, by using the detection result of the subject area, automatic setting of the focus detection area can be performed, and a tracking AF processing function for an arbitrary subject area can be realized. Furthermore, AE processing can be performed based on the luminance information of the focus detection area, or image processing (such as gamma correction processing or AWB (Auto White Balance) adjustment processing, etc.) can be performed based on the pixel values of the focus detection area.

[0023] Also, the control device 106 performs display control by controlling the input / output device 105. For example, based on the result detected by the image processing device 104, an indicator (for example, a rectangular frame surrounding the area) representing the position of the current subject area is superimposed on the display image.

[0024] The image processing device 104 described in this embodiment performs high-quality processing to reduce the noise generated in the live view display image when shooting a subject in a low luminance area with a high ISO sensitivity setting by the imaging device 101 in real time. Then, by displaying the high-quality processing effect so that it can be seen on a display such as the rear monitor of the camera 100, the high-quality processing effect is presented to the user.

[0025] In this embodiment, in order to confirm the high-image-quality processing effect in real time, high-image-quality processing is performed only on some local regions in the image displayed in the live view. Also, in order to make it easier to visually recognize the high-image-quality processing effect even on a display with a low resolution such as the rear monitor of the camera 100, the local regions in the image displayed in the live view after high-image-quality processing are enlarged and displayed on the screen at an equal magnification or a magnification close to the equal magnification.

[0026] <Functional Configuration> FIG. 2 is an explanatory diagram of the configurations of the camera 100 and the image processing apparatus 104 in this embodiment. The camera 100 includes a data storage unit 201, a user input unit 202, a display unit 203, and an image processing apparatus 104. The image processing apparatus 104 includes an image acquisition unit 211, a condition determination processing unit 212, a local region selection unit 213, a local region acquisition unit 214, a high-image-quality processing unit 215, and a screen output processing unit 216.

[0027] The data storage unit 201 is an area for holding the image captured by the imaging device 101. The image for live view display is also temporarily held in the data storage unit 201. The image acquisition unit 211 acquires the image held in the data storage unit 201. The image to be acquired is assumed to be, for example, a noisy image captured with a high-sensitivity setting of ISO 51200 for live view display.

[0028] The condition determination processing unit 212 determines whether to perform high-image-quality processing on the image acquired by the image acquisition unit 211. The user input unit 202 is an input device of the camera 100, and acquires, for example, information on the user's operation on the image acquired by the image acquisition unit 211 using a touch panel type liquid crystal display. The input device may be external, and for example, user operation information may be acquired using a mouse or a keyboard.

[0029] The local area selection unit 213 selects a local area of the image acquired by the image acquisition unit 211 based on the user operation information acquired by the user input unit 202. The local area acquisition unit 214 acquires the local area selected by the local area selection unit 213 from the image acquired by the image acquisition unit 211. The high-quality processing unit 215 performs high-quality processing to reduce noise on the local area image acquired by the local area acquisition unit 214.

[0030] The screen output processing unit 216 performs processing for outputting the local area image that has undergone high-quality processing by the high-quality processing unit 215. For example, on the live view display image acquired by the image acquisition unit 211, image processing is performed on the local area image processed by the high-quality processing unit 215 to an equal magnification or close to an equal magnification to a resolution at which the high-quality processing can be understood, and they are superimposed or synthesized.

[0031] The display unit 203 displays the result processed by the screen output processing unit 216 on the output device of the camera 100. As the output device, for example, a liquid crystal display or an organic EL display is used.

[0032] <Process> FIG. 3 is a flowchart showing the flow of the process according to the present embodiment. In step S301, the user input unit 202 receives from the user a switch to a dedicated mode setting for performing high-quality processing. The selection of a predetermined mode (a dedicated mode for performing high-quality processing) among a plurality of modes is received from the user, and when the predetermined mode (a dedicated mode for performing high-quality processing) is selected, it is determined that high-quality processing is to be performed. Alternatively, by adding a setting parameter for automatically performing high-quality processing to the imaging parameters that can be set by the imaging device 101, the user can arbitrarily set so that high-quality processing is performed on the subsequently live-view displayed image.

[0033] Also, when shooting in an environment where noise is likely to occur, it may be switched to a dedicated mode setting that automatically performs high-quality processing. The environment where noise is likely to occur is assumed to be when shooting in a low-luminance area or when shooting with an increased ISO sensitivity. In such an environment, based on the user-set threshold value, it is switched to a dedicated mode for performing high-quality processing. The switch to the dedicated mode for performing high-quality processing may be performed automatically, or a method may be used to ask the user whether to perform the switch via a pop-up or the like so that the user can select the switch.

[0034] In step S302, the condition determination processing unit 212 determines whether to perform high-quality processing based on the dedicated mode setting for performing high-sensitivity processing performed in step S301. If it is a dedicated mode for performing high-quality processing and the conditions for performing high-quality processing are met, the process proceeds to step S303. On the other hand, if it is not a dedicated mode for performing high-quality processing or the conditions for performing high-quality processing are not met, the process ends.

[0035] Here, FIGS. 4(a), 4(b), and 4(c) are an example of the details of the process for determining whether to perform high-quality processing in step S302 with respect to the set value set in step S301.

[0036] Figure 4(a) shows the details of the determination process in step S302 when the user has previously set a threshold for the ISO sensitivity in step S301. In step S401, the condition determination processing unit 212 acquires the ISO sensitivity information set in the imaging device 101. In step S402, the condition determination processing unit 212 compares the user-set threshold set in step S301 with the ISO sensitivity information acquired in step S401, and determines whether the ISO sensitivity is equal to or higher than the threshold. If the ISO sensitivity is equal to or higher than the threshold, the process proceeds to step S403. On the other hand, if the ISO sensitivity is lower than the threshold, it is determined that the high-image-quality processing is not to be performed, and the process ends. In step S403, the condition determination processing unit 212 switches to a dedicated mode for performing the high-image-quality processing. Thus, the processing of Figure 4(a) ends. In this way, it is also possible to acquire the information on the ISO sensitivity and determine that the high-image-quality processing is to be performed when the ISO sensitivity is equal to or higher than the threshold.

[0037] Subsequently, Figures 4(b) and 4(c) show an example of the details of the determination process in step S302 when the user has previously set a threshold for the low-luminance region in step S301.

[0038] In Figure 4(b), in step S411, the condition determination processing unit 212 acquires the average luminance of the first image being live-view displayed, which is acquired by the image acquisition unit 211. Note that the average luminance acquired in this step may be performed on the local region image acquired by the local region acquisition unit 214. In step S412, the condition determination processing unit 212 compares the user-set threshold previously set in step S301 with the average luminance information acquired in step S411, and determines whether the average luminance is equal to or lower than the threshold. If the average luminance is equal to or lower than the threshold, the process proceeds to step S413. On the other hand, if the average luminance is greater than the threshold, it is determined that the high-image-quality processing is not to be performed, and the process ends. In step S413, the condition determination processing unit 212 switches to a dedicated mode for performing the high-image-quality processing. Thus, the processing of Figure 4(b) ends. In this way, it is also possible to acquire the information on the average luminance of the first image and determine that the high-image-quality processing is to be performed when the average luminance is equal to or lower than the threshold.

[0039] Next, in FIG. 4(c), in step S421, the condition determination processing unit 212 divides the first image being live-view displayed and acquired by the image acquisition unit 211 into fixed sizes. Each process from step S422 to step S424 is performed on all the divided images. In step S422, the condition determination processing unit 212 calculates the average luminance for each divided image. In step S423, the condition determination processing unit 212 determines whether the average luminance calculated in step S422 is equal to or less than the user-set threshold value preset in step S301. If the average luminance is equal to or less than the user-set threshold value, the process proceeds to step S424. On the other hand, if the average luminance is greater than the user-set threshold value, the process returns to step S422. When the processing has been performed on all the divided images, the loop ends.

[0040] In step S424, the condition determination processing unit 212 counts it as the number of low-luminance areas. When the processing for all the divided images is completed, the loop ends and the process proceeds to step S425. In step S425, the condition determination processing unit 212 compares the number of low-luminance areas counted in step S424 with the user-set threshold value set in step S301, and determines whether the ratio of the count number of low-luminance areas to the number of divisions is equal to or greater than the threshold value. If the ratio of the low-luminance areas is equal to or greater than the threshold value, the process proceeds to step S426. On the other hand, if the ratio of the low-luminance areas is less than the threshold value, it is determined that high-image-quality processing is not performed, and the process ends. In step S426, the condition determination processing unit 212 switches to a dedicated mode for performing high-image-quality processing. Thus, the processing of FIG. 4(c) ends. In this way, the first image is divided into a plurality of areas, and information on the average luminance of each of the plurality of areas is acquired. And it may be determined that high-image-quality processing is performed when the ratio of the areas where the average luminance is equal to or less than the threshold value is equal to or greater than the threshold value.

[0041] Next, return to the description of FIG. 3. In step S303, based on the user operation information acquired by the user input unit 202, the local area selection unit 213 selects a local area of the first image that is being live-viewed and acquired by the image acquisition unit 211. The user input unit 202 functions as a reception unit that receives user operations, and the user selects a local area in the image being live-viewed where the user wants to confirm the high-image-quality effect. The selection of the local area from the first image is performed based on the user operation information acquired by the user input unit 202.

[0042] Here, it is assumed that the user performs an operation of pinching out an arbitrary range within the first image acquired by the image acquisition unit 211 and displaying it at the same magnification, and the range displayed at the same magnification is selected as the local area. However, the selection method by user operations is not limited to this. For example, when the user touches an arbitrary position in the first image acquired by the image acquisition unit 211 where the user wants to focus, the area around the touched focus position may be selected as the local area. For example, a rectangular area of a predetermined size centered on the touch position (focus position) may be used as the local area.

[0043] Also, when the user touches an object candidate area within the first image detected by the imaging device 101, the object candidate area or a peripheral area of a predetermined size including the object candidate area may be selected as the local area. The object candidates mentioned here refer to objects in various categories such as people, animals, vehicles, and local parts such as the whole body, head, and pupils of a person or an animal.

[0044] In step S304, the local area acquisition unit 214 displays at the same magnification an arbitrary range within the first image that is being live-viewed according to the user operation. Then, the locally magnified range is cut out from the image being live-viewed as a second image (local area image) for high-image-quality processing.

[0045] In step S305, the condition determination processing unit 212 determines whether high-quality processing can be performed on the second image acquired in step S304 in real time. The determination of whether it is a size that allows real-time processing can be made, for example, by holding the number of pixels that can be subjected to high-quality processing within 10 ms and determining whether the number of pixels of the local region image acquired in step S304 falls within that range of the number of pixels.

[0046] Here, if the number of pixels of the second image acquired in step S304 is a number of pixels that cannot be subjected to high-quality processing within 10 ms, the second image is re-acquired to a number of pixels that can be subjected to high-quality processing within 10 ms and automatically adjusted. That is, the number of pixels of the second image is adjusted so that high-quality processing can be performed on the second image indicating the local region in real time.

[0047] Alternatively, the user may be notified that the number of pixels of the second image cannot be subjected to high-quality processing in real time, and the processing from step S303 to step S305 may be repeated.

[0048] FIGS. 5(a), 5(b), and 5(c) are examples of a method of notifying the user that the number of pixels of the second image cannot be subjected to high-quality processing in real time. FIG. 5(a) is an example of causing a message ("Unremoved noise") indicating that high-quality processing cannot be performed in real time to be pop-up displayed 501 on the display unit 203. FIG. 5(b) is an example of superimposing a message ("Unremoved noise") 502 indicating that high-quality processing cannot be performed in real time on the second image acquired in step S304 and displaying it on the display unit 203.

[0049] FIG. 5(c) is an example of color-coding and displaying the color of the outer frame 503 of the second image obtained in step S304 depending on whether high-quality processing can be performed in real time or not. For example, when high-quality processing cannot be performed in real time, the color of the outer frame 503 may be displayed in red, and when high-quality processing can be performed, the color of the outer frame 503 may be displayed in green. Alternatively, without being limited to these methods, discrimination may be made using a display form capable of determining whether high-quality processing can be performed in real time, and the display may be differentiated accordingly. By presenting information indicating whether high-quality processing can be performed in real time on the second image indicating a local area in this way, the user can easily recognize this fact.

[0050] In step S306, the high-quality processing unit 215 performs high-quality processing on the second image obtained in step S304. The local image obtained by performing this high-quality processing on the second image is defined as the third image. This high-quality processing may be NR processing (noise reduction processing) for the purpose of noise reduction. When performing more accurate NR processing, the amount of calculation increases, and the range of local areas that can be processed in real time becomes narrower. As a solution, by switching to a simple NR processing with a small amount of calculation, high-quality processing may be performed in real time on a local area image with a larger number of pixels. That is, the second noise reduction processing with a smaller processing load than the normal noise reduction processing may be configured to be executable, and the second noise reduction processing may be performed on a second image having a larger number of pixels than the second image and indicating a wider local area.

[0051] For example, a parameter that can be switched to a setting that enables NR processing to be performed on a second image with a larger number of pixels may be provided in the menu of the imaging device 101 so that the user can arbitrarily adjust the amount of calculation of the NR processing. Further, the high-quality processing is not limited to NR processing, and a neural network model (hereinafter referred to as an NN model) learned for the purpose of noise reduction may be used. That is, high-quality processing may be performed using a neural network model learned to reduce noise.

[0052] Here, FIGS. 6(a) and 6(b) are an example of a parameter menu for making the number of pixels of the second image determined in step S305 variable. FIG. 6(a) shows an example in which the number of pixels of the second image determined in step S305 is set as the range of the local area, and the parameter 601 set for that range is presented in a pull-down format for the user to select from "wide / medium / narrow". For example, when the user selects "wide", NR processing can be performed on a wider second image with a larger number of pixels. However, since it is necessary to perform a simpler NR processing compared to when "narrow" is selected, the degree of image quality improvement is lower.

[0053] Also, FIG. 6(b) shows an example in which the setting parameter (NR processing intensity parameter) 602 for adjusting the amount of calculation of the NR processing performed in step S306 is presented in a pull-down format for the user to select from "strong / medium / weak". For example, when the user selects "strong", a high-quality processing with a larger amount of calculation and higher precision can be performed, but the number of pixels of the second image that can be processed in real time is small, and the range of the local area is narrow. The additional parameter menu is not limited to the pull-down format. For example, a text field may be added to directly specify the number of pixels of the second image.

[0054] In step S307, the screen output processing unit 216 performs processing for screen output on the high-quality processing result of step S306, and displays the third image together with the first image on the display unit 203. In the processing for screen output, in order to make it easy for the user to confirm the high-quality processing effect, an equal magnification or an enlargement processing close to equal magnification is performed on the third image that has undergone the high-quality processing. The equal magnification processing is performed using a processing for complementing between pixels, such as linear interpolation or nearest neighbor interpolation. The display unit 203 may be an output device attached to the back of the camera 100, for example, a liquid crystal display.

[0055] Here, FIG. 7 shows an example in which the screen output processing unit 216 superimposes or synthesizes the third image that has been subjected to high-quality processing and is scaled equally on the first image (predetermined area) that is being live-viewed and then displays the result 701 on the display unit 203. According to the display example in FIG. 7, even when the user adjusts the position of the camera 100 according to the movement of the subject, it is possible to clearly grasp which area in the image has been subjected to high-quality processing.

[0056] As an example, the superimposition position of the third image in the present embodiment is set to the upper right of the image being live-viewed as shown in FIG. 7, but it is not limited to this. For example, the user may be able to arbitrarily change the superimposition position, or the superimposition position may be automatically adjustable so that the position selected in step S303 can be confirmed.

[0057] As described above, in the present embodiment, a local area is selected from the first image being displayed on the display unit 203, a second image indicating the local area is generated from the first image, and high-quality processing is performed on the second image. Then, the third image on which the high-quality processing has been performed is displayed on the display unit 203. More specifically, the high-quality processed local area image is scaled equally or processed to a resolution close to the equally scaled display for the image of the local area selected from the image live-viewed on the rear monitor of an imaging device such as a camera. Then, it is superimposed or synthesized on the image live-viewed on the rear monitor and displayed.

[0058] According to the present embodiment, it becomes possible for the user to confirm the noise reduction effect at an early stage and take a desired image. In addition, it becomes possible to perform high-quality processing in real time on the image being live-viewed and present the effect of the high-quality processing after shooting to the user in an easy-to-understand manner. That is, it becomes possible to confirm the effect of the high-quality processing after shooting in real time during the shooting operation.

[0059] In addition, in this embodiment, although an example of NR processing related to noise reduction processing is given, this embodiment can also be applied to other degradation corrections such as super-resolution processing, and style conversion (for example, processing of converting a color image into a monochrome image).

[0060] Note that in this embodiment, an example has been described in which it is determined to perform high-quality processing when switching to a dedicated mode for performing high-quality processing in S301 of FIG. 3 and satisfying the conditions of FIG. 4(a), FIG. 4(b), or FIG. 4(c). However, the present invention is not limited to this example. Regardless of the conditions of FIG. 4(a), FIG. 4(b), or FIG. 4(c), it may be determined to perform high-quality processing in response to the switching to the dedicated mode for performing high-quality processing in S301 of FIG. 3. Alternatively, regardless of the switching to the dedicated mode for performing high-quality processing, it may be determined to perform high-quality processing when any of the conditions of FIG. 4(a), FIG. 4(b), or FIG. 4(c) is satisfied.

[0061] (Embodiment 2) In this embodiment, another example of clearly presenting the high-quality processing effect after shooting to the user will be described. In Embodiment 1, an example was shown in which high-quality processing was performed in real time on the image being live-viewed, and an image of a local area where high-quality processing was performed, such as equal-magnification display, was displayed so that the high-quality processing effect could be easily confirmed. In contrast, in this embodiment, an example of clearly presenting the high-quality processing effect for the image (the saved image that has been shot) reproduced and displayed after shooting will be shown.

[0062] Since the device configuration in this embodiment is the same as that in Embodiment 1, detailed description thereof will be omitted. Regarding the processing flow as well, since it is the same as the processing flow described with reference to FIG. 3 in Embodiment 1, the description thereof will be omitted. The differences from Embodiment 1 are the processing of the screen output unit 216 in FIG. 2 and the processing of displaying the result of high-quality processing in step S307 in FIG. 3 on the screen.

[0063] FIG. 8(a) and FIG. 8(b) show an example of the processing content in the screen output unit 216 in the present embodiment being displayed on the display unit 203. In the example of FIG. 8(a), in steps S301 to S306 in FIG. 3, high-quality processing is performed on the second image selected by the user. In the present embodiment, since it is assumed that high-quality processing is performed on the image reproduced and displayed after shooting, real-time performance is not necessarily required. Therefore, high-quality processing can be performed on any local area image selected by the user. At this time, since it is not necessary to consider the number of pixels of the local area image, the processing in step S305 can also be omitted.

[0064] Also, since it is not necessary to adjust the position of the camera 100 in accordance with the movement of the subject, the process of equally magnifying and displaying the third image 801 subjected to high-quality processing in step S306 over the entire display area of the display unit 203 is performed in step S307.

[0065] Next, in the example of FIG. 8(b), in step S307, the process of simultaneously displaying the second image 802 selected by the user in step S303 and the third image 803 subjected to high-quality processing in step S306 on the display unit 203 is performed. In the present embodiment, in order to make the second image 802 and the third image 803 comparable, they are displayed side by side in parallel. However, it is not limited to this, and they may be arranged vertically and displayed.

[0066] Also, parallel display may be performed to compare the change content of the setting parameter regarding the calculation amount of the NR process described in FIG. 6(b). In that case, the third image subjected to high-quality processing by the NR process before the change and the third image subjected to high-quality processing by the NR process after the change may be simultaneously displayed on the display unit 203 in step S307.

[0067] (Embodiment 3) In this embodiment, a high-quality processing is performed in real time on the image being displayed as a live view image, and another example is described in which an image of a local area subjected to high-quality processing is displayed in an equal magnification display or the like so that the high-quality processing effect can be easily confirmed. In Embodiment 1, an example was shown in which the rear monitor of the camera was used as the display unit that outputs the result of the high-quality processing. In contrast, in this embodiment, an example of output display on the camera finder is shown.

[0068] Since the device configuration in this embodiment is the same as that in Embodiment 1, detailed description is omitted. Also, regarding the processing flow, since it is the same as the processing flow described with reference to FIG. 3 in Embodiment 1, the description is omitted. The difference from Embodiment 1 lies in the display unit 203 in FIG. 2 and the output device that displays the result of the high-quality processing in step S307 in FIG. 3 on the screen.

[0069] FIG. 9 is an example showing the display unit 203 in this embodiment. In order for the user to confirm as a live view image, by displaying the same output 902 as in step S307 in Embodiment 1 in the EVF (Electronic View Finder) 901, the high-quality processing effect can be confirmed. The EVF functions as a display unit for confirming the subject to be imaged. This embodiment is also applicable to the display example of Embodiment 2, and by displaying the output 902 in the EVF 901, the high-quality processing effect can be confirmed.

[0070] The disclosure of this specification includes the following image processing apparatus, imaging apparatus, control method, and program.

[0071] (Item 1) selection means for selecting a local area from a first image being displayed on a display unit; processing means for generating a second image indicating the local area from the first image and performing high-quality processing on the second image; control means for causing the display unit to display a third image on which the high-quality processing has been performed by the processing means; An image processing apparatus comprising the above.

[0072] (Item 2) The image processing apparatus according to item 1, wherein the control means causes the display unit to display the third image together with the first image.

[0073] (Item 3) The image processing apparatus according to item 1 or 2, wherein the selection means selects the local area from the first image being live-view displayed on the display unit.

[0074] (Item 4) An acquisition means for acquiring information on ISO sensitivity; A determination means for determining to perform the high-image-quality processing by the processing means when the ISO sensitivity is equal to or higher than a threshold value, further comprising: The image processing apparatus according to any one of items 1 to 3, wherein the selection means selects the local area when it is determined by the determination means that the high-image-quality processing is to be performed.

[0075] (Item 5) An acquisition means for acquiring information on the average luminance of the first image; A determination means for determining to perform the high-image-quality processing by the processing means when the average luminance is equal to or lower than a threshold value, further comprising: The image processing apparatus according to any one of items 1 to 3, wherein the selection means selects the local area when it is determined by the determination means that the high-image-quality processing is to be performed.

[0076] (Item 6) A dividing means for dividing the first image into a plurality of areas; An acquisition means for acquiring information on the average luminance of each of the plurality of areas; A determination means for determining to perform the high-image-quality processing by the processing means when the ratio of the areas where the average luminance is equal to or lower than the threshold value is equal to or higher than the threshold value, further comprising: The image processing apparatus according to any one of Items 1 to 3, wherein the selection means selects the local area when it is determined by the determination means that the high-quality processing is to be performed.

[0077] (Item 7) The image processing apparatus further includes reception means for receiving a user operation, The image processing apparatus according to any one of Items 1 to 6, wherein the selection means selects a local area that is displayed at the same magnification by the user operation with respect to the first image that is being live-viewed on the display unit.

[0078] (Item 8) The image processing apparatus further includes reception means for receiving a user operation, The image processing apparatus according to any one of Items 1 to 6, wherein the selection means selects the local area based on the focus position specified by the user operation with respect to the first image that is being live-viewed on the display unit.

[0079] (Item 9) The image processing apparatus according to any one of Items 1 to 6, wherein the selection means selects the local area based on an object candidate area detected from the first image that is being live-viewed on the display unit.

[0080] (Item 10) The image processing apparatus according to any one of Items 1 to 9, further comprising presentation means for presenting information indicating whether it is possible to perform the high-quality processing on the second image indicating the local area in real time.

[0081] (Item 11) The image processing apparatus according to any one of Items 1 to 10, further comprising adjustment means for adjusting the number of pixels of the second image so that it is possible to perform the high-quality processing on the second image indicating the local area in real time.

[0082] (Item 12) The image processing apparatus according to any one of Items 1 to 11, wherein the high-quality processing is noise reduction processing.

[0083] (Item 13) The processing means is capable of performing a second noise reduction process with a smaller processing load than the noise reduction process, The image processing apparatus according to Item 12, wherein the processing means is capable of performing the second noise reduction process on a second image having a larger number of pixels and indicating a wider local area than the second image.

[0084] (Item 14) The image processing apparatus according to any one of Items 1 to 11, wherein the high-quality processing is performed by using a neural network model trained to reduce noise.

[0085] (Item 15) The image processing apparatus according to any one of Items 1 to 14, wherein the control means magnifies the third image by the same magnification and superimposes or synthesizes it on a predetermined area on the first image being live-view displayed on the display unit.

[0086] (Item 16) The image processing apparatus according to Item 1, wherein the first image is a captured and saved image.

[0087] (Item 17) The image processing apparatus according to Item 16, wherein the control means causes the display unit to display the second image together with the third image.

[0088] (Item 18) Selection reception means for receiving selection of a predetermined mode among a plurality of modes, Determination means for determining to perform the high-quality processing by the processing means when the predetermined mode is selected, The image processing apparatus according to any one of Items 1 to 17, further comprising

[0089] (Item 19) The image processing apparatus according to any one of Items 1 to 18, and the display unit, An imaging apparatus comprising

[0090] (Item 20) The imaging apparatus according to Item 19, wherein the display unit is a rear monitor of the imaging apparatus or an EVF (Electronic View Finder) for confirming a subject to be imaged.

[0091] (Item 21) A control method for an image processing apparatus, comprising: a selection step of selecting a local area from a first image being displayed on a display unit; a processing step of generating a second image indicating the local area from the first image and performing high-quality processing on the second image; a control step of causing the display unit to display a third image on which the high-quality processing has been performed in the processing step; A control method for an image processing apparatus, characterized by comprising

[0092] (Item 22) A program for causing a computer to execute the control method for an image processing apparatus according to Item 21.

[0093] (Other Embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0094] The invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the claims are appended to disclose the scope of the invention.

Explanation of Reference Numerals

[0095] 104: Image processing apparatus, 211: Image acquisition unit, 212: Condition determination processing unit, 213: Local area selection unit, 214: Local area acquisition unit, 215: High image quality processing unit, 216: Screen output processing unit

Claims

1. selection means for selecting a local area from a first image being displayed on a display unit; processing means for generating a second image indicating the local area from the first image and performing a high-quality processing on the second image; control means for causing the display unit to display a third image on which the high-quality processing has been performed by the processing means; An image processing apparatus, comprising:

2. The image processing apparatus according to claim 1, wherein the control means causes the display unit to display the third image together with the first image.

3. The image processing apparatus according to claim 1, wherein the selection means selects the local area from the first image being live-view displayed on the display unit.

4. acquisition means for acquiring information on ISO sensitivity; determination means for determining that the high-quality processing is to be performed by the processing means when the ISO sensitivity is equal to or higher than a threshold value; The image processing apparatus according to claim 1, further comprising: wherein the selection means selects the local area when the determination means determines that the high-quality processing is to be performed.

5. acquisition means for acquiring information on the average luminance of the first image; determination means for determining that the high-quality processing is to be performed by the processing means when the average luminance is equal to or lower than a threshold value; The image processing apparatus according to claim 1, further comprising: wherein the selection means selects the local area when the determination means determines that the high-quality processing is to be performed.

6. division means for dividing the first image into a plurality of areas; acquisition means for acquiring information on the average luminance of each of the plurality of areas; determination means for determining that the high-quality processing is to be performed by the processing means when the ratio of the areas where the average luminance is equal to or lower than a threshold value is equal to or higher than a threshold value; The image processing apparatus according to claim 1, further comprising: wherein the selection means selects the local area when the determination means determines that the high-quality processing is to be performed.

7. The image processing apparatus according to claim 1, further comprising reception means for receiving a user operation, wherein the selection means selects a local area that is displayed at the same magnification by the user operation with respect to the first image being live-view displayed on the display unit.

8. The image processing apparatus according to claim 1, further comprising reception means for receiving a user operation, The image processing apparatus according to claim 1, wherein the selection means selects the local area based on a focus position designated by the user operation with respect to the first image live-view displayed on the display unit.

9. The image processing apparatus according to claim 1, wherein the selection means selects the local area based on an object candidate area detected from the first image live-view displayed on the display unit.

10. The image processing apparatus according to claim 1, further comprising presentation means for presenting information indicating whether it is possible to perform the high-image-quality processing on the second image indicating the local area in real time.

11. The image processing apparatus according to claim 1, further comprising adjustment means for adjusting the number of pixels of the second image so that it is possible to perform the high-image-quality processing on the second image indicating the local area in real time.

12. The image processing apparatus according to claim 1, wherein the high-image-quality processing is noise reduction processing.

13. The processing means is capable of performing a second noise reduction processing having a lower processing load than the noise reduction processing, The image processing apparatus according to claim 12, wherein the processing means is capable of performing the second noise reduction processing on the second image having a larger number of pixels than the second image and indicating a wider local area.

14. The image processing apparatus according to claim 1, wherein the processing means performs the high-image-quality processing using a neural network model learned to reduce noise.

15. The image processing apparatus according to claim 1, wherein the control means superimposes or synthesizes the third image at the same magnification on a predetermined area on the first image live-view displayed on the display unit.

16. The image processing apparatus according to claim 1, wherein the first image is a captured and saved image.

17. The image processing apparatus according to claim 16, wherein the control means causes the display unit to display the second image together with the third image.

18. Selection reception means for receiving selection of a predetermined mode among a plurality of modes, Determination means for determining that when the predetermined mode is selected, the high-image-quality processing is performed by the processing means; The image processing apparatus according to claim 1, further comprising the same.

19. The image processing apparatus according to claim 1; The display unit; An imaging apparatus comprising the same.

20. The imaging apparatus according to claim 19, wherein the display unit is a rear monitor of the imaging apparatus or an EVF (Electronic View Finder) for checking a subject to be imaged.

21. A control method for an image processing apparatus, comprising: A selection step of selecting a local area from a first image being displayed on a display unit; A processing step of generating a second image indicating the local area from the first image and performing high-image-quality processing on the second image; A control step of causing the display unit to display a third image on which the high-image-quality processing has been performed in the processing step; A control method for an image processing apparatus, characterized by including the same.

22. A program for causing a computer to execute the control method for an image processing apparatus according to claim 21.

Images