Image processing device, image processing method, and program

Abstract

The present invention provides an image processing device that enables imaging while performing NR (Noise Reduction) processing with settings desired by the user, while reducing the delay time of noise reduction processing. [Solution] The image captured by the imaging device is acquired. It is determined whether or not the imaging conditions of the imaging device are being adjusted. The noise in the image is reduced. The process for reducing the noise in the image is switched to a first reduction process if the imaging conditions of the imaging device are not being adjusted, and to a second reduction process that has a shorter delay time between image input and output in the noise reduction process than the first reduction process if the imaging conditions of the imaging device are being adjusted.

Description

【Technical Field】 【0001】 The present invention relates to an image processing apparatus, an image processing method, and a program. 【Background Art】 【0002】 Conventionally, a technique for amplifying the brightness of an image by applying gain to the image under low illumination is known. However, when the brightness is amplified by applying gain, noise may be added at the same time. When noise is added to the image, the visibility of the subject decreases. Therefore, a technique for restoring an image using an image restoration function such as a Noise Reduction (hereinafter NR) function is known. 【0003】 Generally, the higher the NR processing with a high image restoration effect, the higher the processing load and the longer the processing time. Therefore, in Patent Document 1, in order to perform development processing in real time, a high-quality development process including NR is performed on the attention area, and a high-speed development process not including NR is performed on the non-attention area, and the images obtained from both processes are synthesized and output. By performing high-quality development processing only on the attention area, it is possible to obtain a high-quality image while shortening the processing time, and it is possible to confirm the image quality in real time. Further, Patent Document 2 discloses a technique for easily confirming the change of an image by image quality improvement processing by switching, arranging, or overlapping and displaying a pre-correction image and a post-correction image. If it is a pre-correction image, it can be displayed in real time, and if it is a post-correction image, a high-quality image can be displayed. 【Prior Art Documents】 【Patent Documents】 【0004】 【Patent Document 1】 Japanese Patent Application Laid-Open No. 2018-182550 【Patent Document 2】 Japanese Patent Application Laid-Open No. 2020-39851 【Summary of the Invention】 【Problems to be Solved by the Invention】 【0005】 However, the technology disclosed in Patent Document 1 always applies NR processing to the area of ​​interest and not to the area of ​​interest. In other words, it does not disclose how to improve the visibility of the area of ​​interest or the entire image including the area of ​​interest through NR processing. Now, consider a system that applies NR processing in real time while shooting video, where the photographer manually adjusts the focus position using an operating device. The focus adjustment method involves manually operating the focus lens using an operating device and adjusting the focus position while viewing the image. For example, if the area of ​​interest is set to be wide in order to improve the visibility of the entire image, the processing load of NR increases and the processing time increases. Therefore, the temporal discrepancy between the feel of operating the operating device and the image becomes larger, making it difficult to adjust the focus position to the desired position. In particular, when using processing that has a high effect on NR, such as NR using Deep Learning (hereinafter DL), the processing load of NR becomes even higher. Furthermore, the technology disclosed in Patent Document 2 displays the pre-correction image and the corrected image by switching, arranging, or overlaying them, so it does not consider a method to improve visibility while shortening the processing time. 【0006】 The present invention aims to enable imaging while performing NR processing with settings desired by the user, while reducing the delay time of noise reduction processing. [Means for solving the problem] 【0007】 To achieve the object of the present invention, for example, an image processing apparatus according to one embodiment comprises the following configuration: an acquisition means for acquiring an image captured by an imaging device; a determination means for determining whether or not the imaging conditions of the imaging device are being adjusted; a reduction means for reducing noise in the image; and a switching means for switching the noise reduction process performed by the reduction means to a first reduction process if the imaging conditions of the imaging device are not being adjusted, and to a second reduction process that has a shorter delay time for image input / output in the noise reduction process than the first reduction process if the imaging conditions of the imaging device are being adjusted. [Effects of the Invention] 【0008】 This reduces the delay time of noise reduction processing while enabling imaging with NR processing at the user's desired settings. [Brief explanation of the drawing] 【0009】 [Figure 1] A block diagram showing an example of the functional configuration of the image processing apparatus according to Embodiment 1. [Figure 2] A flowchart illustrating an example of NR processing performed by an image processing device. [Figure 3] A diagram illustrating NR processing by an image processing device. [Figure 4] A block diagram showing an example of the functional configuration of an image processing apparatus according to Embodiment 2. [Figure 5] A flowchart illustrating an example of the process for defining a sub-region. [Figure 6] A diagram illustrating the process of setting up a sub-region. [Figure 7] A block diagram showing an example of a hardware configuration. [Modes for carrying out the invention] 【0010】 The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted. 【0011】 [Embodiment 1] The image processing device 100 according to Embodiment 1 performs noise reduction (NR processing) on ​​an input image. Here, consider, for example, a case where a user adjusts the focus as an imaging condition while checking the captured image that has undergone NR processing. The user operates while checking the image in order to focus on the main subject, but if the processing load of the NR processing increases and the processing time becomes long, a time difference will occur between the image (video) output as a result of the NR processing and the real-time situation, which may make it difficult to adjust the focus appropriately. From this perspective, the image processing device 100 according to this embodiment acquires an image captured by an imaging device and determines whether or not the imaging conditions of the imaging device are being adjusted. Next, the image processing device 100 switches the NR processing on the acquired image to a first reduction process if the imaging conditions of the imaging device are not being adjusted, and to a second reduction process which has a shorter delay time between image input and output in the noise reduction process than the first reduction process if the imaging conditions of the imaging device are being adjusted. The details of the processing performed by such an image processing device 100 will be described below. 【0012】 The functional configuration of the image processing apparatus according to this embodiment will be explained with reference to Figure 1. Figure 1 is a block diagram showing an example of the configuration of the image processing apparatus 100 according to this embodiment. The image processing apparatus 100 includes an input unit 101, a control unit 102, and an output unit 106. The input unit 101 acquires an image signal (image) output from a camera (not shown). The input unit 101 may acquire the image via a wired connection or via wireless communication. Hereinafter, when simply referred to as "camera," it refers to an imaging device that transmits an image to be processed to the image processing apparatus 100 according to this embodiment. In this embodiment, the camera and the image processing apparatus 100 are described as separate devices, but the image processing apparatus 100 may be a device built into the camera. If the image processing apparatus 100 is a device built into the camera, the input unit 101 can acquire an image captured by the camera's imaging mechanism. 【0013】 The control unit 102 includes an adjustment determination unit 103, a mode switching unit 104, and an NR processing unit 105. The adjustment determination unit 103 determines whether or not the camera's imaging conditions are being adjusted. Here, the following explanation assumes that the camera's focal position is used as the imaging condition. The adjustment determination unit 103 may, for example, acquire position information of the focus lens from the camera and determine that the focal position is being adjusted if the position of the focus lens has changed. Alternatively, the adjustment determination unit 103 may calculate an image evaluation value based on the pixel values ​​of the image, such as the contrast value of the image, based on the image signal acquired by the input unit 101, or acquire it from the camera, and determine that the focal position is being adjusted if the image evaluation value has changed. Alternatively, the adjustment determination unit 103 may acquire a signal from the camera indicating that a user is performing an operation on the camera, and determine that the focal position is being adjusted if such a signal has been acquired. Thus, the determination by the adjustment determination unit 103 can arbitrarily adopt a determination method based on information that allows it to determine that the camera's focal position is being adjusted. 【0014】 The mode switching unit 104 switches the mode of image NR processing, which is performed by the NR processing unit described later, depending on whether or not the focal position is being adjusted. In this embodiment, the mode switching unit 104 sets the NR processing mode to the first mode when the focal position is not being adjusted, and sets the NR processing mode to the second mode when the focal position is being adjusted. Here, the second mode is a mode in which the input / output delay time in NR processing is shorter compared to the first mode. 【0015】 Here, the input / output delay time in NR processing is defined as the time from when the input unit 101 acquires an image signal until the output unit 106, described later, outputs the NR-processed image to an external device. In the following, when simply referred to as "delay time," it refers to the time from when the input unit 101 acquires an image signal until the output unit 106 outputs the NR-processed image to an external device. However, the delay time here can be defined in any way as long as it is a time that allows evaluation of the delay time in the output of the image due to NR processing. For example, in an image processing device 100 equipped with an image display function on a display not shown, the delay time may be the time from when the control unit 102 starts NR processing on the image until the image with completed NR processing is displayed on the display of the image processing device 100. By performing NR processing in a second mode with a shorter delay time, the temporal discrepancy between the user's feel of operating the operating member and the image can be reduced, making it easier to adjust the focus position to the desired position. The mode switching unit 104 may set the second mode to a mode that performs NR processing that results in a delay time of, for example, several hundred milliseconds (e.g., 300 milliseconds). 【0016】 Furthermore, the first mode in this embodiment is a mode in which the image processing device 100 performs (normal) NR processing as desired by the user. The NR processing settings in the first mode may be determined based on user input, based on the image signal being captured, or an initial setting may be applied. 【0017】 The second mode is a mode with a shorter delay time than the first mode as described above. The second mode may be any mode that performs NR processing as long as the delay time is shorter than that of the first mode. For example, the second mode may be a mode that performs NR processing with a lighter processing load than the first mode, or a mode that performs NR processing on a partial region smaller than the region to be processed in the NR processing in the first mode in the image. Also, the second mode may be a mode that performs NR processing using a machine learning model different from the machine learning model used by the first mode for NR processing, or a mode in which the number of image signals used as input when performing NR processing once is smaller than that in the first mode. For example, the input in one NR processing may be an image of the first number of frames (e.g., 5 frames) that are temporally continuous in the first mode, and an image of the second number of frames (e.g., 3 frames) smaller than the first number of frames in the second mode. 【0018】 In the following, it is assumed that the second mode is a mode that performs NR processing on a partial region smaller than the region to be processed in the NR processing in the first mode in the image. Here, the partial region where NR processing is performed in the second mode is not particularly limited as long as it is a region set in the image and smaller than the region where NR processing is performed in the first mode. For example, the processing range of NR processing in the first mode may be the entire image, and the processing range of NR processing in the second mode may be a partial region in such an entire image. Here, it is assumed that the partial region where NR processing is performed in the second mode is a partial region including the subject in the image. 【0019】 For example, the partial area where NR processing is executed in the second mode may be a predetermined area centered on the center point of the image. This predetermined area can be assumed to be a rectangular area centered on the center point of the image and having half the width and height of the entire image. Also, such a partial area may be set in any manner as long as it is a predetermined area, such as the lower half area of the entire image. Further, such a partial area may be set based on an input from a user who operates the camera that captures the image. For example, as the partial area including the subject, a bounding box surrounding a predetermined subject detected by a known image recognition process can be used. An example of subject detection will be described later with reference to Embodiment 2. 【0020】 The NR processing unit 105 performs NR processing to reduce noise in the image. Since general image noise reduction processing can be used for the NR processing, detailed description of the processing itself here is omitted. The NR processing unit 105 according to the present embodiment reduces noise in the image by the NR processing in the mode switched by the mode switching unit 104. The output unit 106 outputs the image processed by the NR processing unit 105 to the outside of the image processing apparatus 100. 【0021】 Next, referring to FIG. 2, the NR processing executed by the image processing apparatus 100 according to the present embodiment will be described. FIG. 2 is a flowchart showing an example of the NR processing executed by the image processing apparatus 100. The processing shown in FIG. 2 is started when, for example, an operation to enable the NR processing is input by the user, and is then executed periodically. 【0022】 In S200, the input unit 101 acquires an image from the camera. 【0023】 In S201, the control unit 102 acquires focus position adjustment information. The focus position adjustment information is the position information of the focus lens or the image evaluation value of the image signal described above. 【0024】 In S202, the adjustment determination unit 103 determines whether the focal position is being adjusted based on the focal position adjustment information acquired in S201. If the focal position is being adjusted, the process proceeds to S203; otherwise, the process proceeds to S204. 【0025】 In S203, the mode switching unit 104 sets the mode of the NR processing performed by the NR processing unit 105 to the second mode and proceeds to S205. In S204, the mode switching unit 104 sets the mode of the NR processing performed by the NR processing unit 105 to the first mode and proceeds to S205. 【0026】 In S205, the NR processing unit 105 performs NR processing based on the mode set in S203 or S204. In S206, the output unit 106 outputs the image processed with NR in S205 to the outside of the image processing device 100, and the process shown in Figure 2 is completed. 【0027】 Next, with reference to Figure 3, an example of an image that has undergone NR processing by the image processing device 100 according to this embodiment will be described. Figure 3(a) shows an example of an image before NR processing. In the image shown in Figure 3(a), countless noises cover the main subject, resulting in poor visibility. Therefore, in the image shown in Figure 3(a), although the main subject is located near the center of the image, the focus is not on the subject, resulting in blurring. 【0028】 Figure 3(b) shows an example of an image in the state shown in Figure 3(a) after applying NR processing in the first mode. In the image shown in Figure 3(b), NR processing in the first mode has been applied to the entire image, and the noise has been reduced. However, in the state shown in Figure 3(b), the main subject is not in focus. 【0029】 Figure 3(c) is a diagram illustrating the input image and NR processing applied to the input image when the user begins operating the control element in the situation where the image shown in Figure 3(b) is output. In Figure 3(c), the camera's focal position is being adjusted in response to the user's operation, and the adjustment determination unit 103 also determines that the focal position is being adjusted. Therefore, the NR processing mode is set to the second mode, and in the image shown in Figure 3(c), NR processing is performed only on the partial region including the main subject in the center of the image. By performing NR processing only on a partial region rather than the entire image, the processing load is reduced and the delay time is shortened, thereby reducing the temporal discrepancy between the user's feel of operating the control element and the image, making it easier to adjust the focal position to the position desired by the user. 【0030】 Figure 3(d) shows an example of an image generated by NR processing performed after the user has finished their operation (the focus position is no longer being adjusted) in the situation described with reference to Figure 3(c). In the image shown in Figure 3(d), the light NR processing performed only on a partial area, as described with reference to Figure 3(c), has set a suitable focus position for the area near the main subject, and the focus is on the main subject. In the situation shown in Figure 3(d), since the focus position is not being adjusted, NR processing in the first mode is performed on the entire image in which the focus is on the main subject, improving the overall visibility of the image. 【0031】 With this configuration, it is possible to determine whether or not the focus position is being adjusted, and depending on the result of that determination, the noise reduction (NR) processing on the image can be switched between a first mode and a second mode with a shorter delay time than the first mode. Therefore, it is possible to reduce the delay time while improving the visibility of the main subject. Furthermore, after the focus position has been adjusted, the NR processing mode can be returned to the first mode, making it possible to perform imaging while performing normal NR processing (especially with settings desired by the user). Thus, it has become possible to manually adjust the focus position to the desired subject while using the image recovery function. 【0032】 In this embodiment, the determination of whether or not the focal position is being adjusted has been described as being performed based on the position information of the focus lens or a change in the image evaluation value. However, the determination process by the adjustment determination unit 103 is not limited to this method, as long as it can determine whether or not the focal position is being adjusted. For example, the determination that the focal position is being adjusted may be made when the camera is performing autofocus (hereinafter referred to as AF) operation (for example, when an AF start instruction has been given), or when a drive command for the focus lens has been sent to the camera (from an external device such as a PC). In such cases, the AF start instruction or the drive command for the focus lens will be acquired (received) at S201 in Figure 2. 【0033】 In this embodiment, the first mode was described as performing NR processing on the entire image, but the processing range in the first mode may also be a partial region of the entire image. In such cases, the range in which NR processing is performed in the second mode shall be set to be smaller than the range in which NR processing is performed in the first mode. 【0034】 Furthermore, in this embodiment, the first and second modes are described as being switched depending on whether the focal position is being adjusted or not. However, as mentioned above, using conditions based on the focal position as imaging conditions is just one example; for example, conditions based on the angle of view, zoom, or aperture may be used as imaging conditions, or whether or not control commands for the angle of view, zoom, or aperture are being executed may be used. In addition, it may be possible to set whether or not to switch the NR processing mode according to such imaging conditions (for example, based on user input). 【0035】 For example, the image processing device 100 may also notify the user whether the current NR processing mode is the first mode or the second mode. For example, the image processing device 100 may display information indicating whether NR processing is being performed in the first mode or the second mode (information indicating the current mode) separately from the image (for example, on a display unit not shown), or it may display the information indicating the current mode superimposed on the image. 【0036】 [Embodiment 2] In Embodiment 1, an example was described in which, in the second mode, NR processing is performed on a sub-region of the image that is smaller than the processing range in the first mode. In this case, if the region where the subject exists is assumed to be (for example, the center of the image), it is possible to set the sub-region to include such a region, but the subject does not always exist in such a region. From this viewpoint, the image processing apparatus 100 according to this embodiment detects a subject in the image and sets a sub-region to include such a subject detection region as the sub-region of the image to be processed for NR in the second mode. 【0037】 Figure 4 is a block diagram showing an example of the functional configuration of the image processing apparatus 100 according to this embodiment. The image processing apparatus 100 according to this embodiment has the same configuration as that of Embodiment 1, except that it includes a partial area setting unit 400, and is capable of performing the same processing, so redundant explanations will be omitted. 【0038】 The partial area setting unit 400 sets a partial area in the image to be subjected to NR processing in the second mode. In this embodiment, the partial area setting unit 400 detects a predetermined subject in the image and sets the area containing the subject as the partial area. Here, the image processing device 100 is assumed to perform the subject detection, but the configuration is not limited to this as long as information about the area where the subject has been detected can be obtained. For example, a camera separate from the image processing device 100 may detect the subject from the captured image and transmit the detection information along with the image to the image processing device 100. Alternatively, the partial area setting unit 400 may set the partial area based on the AF frame, which is the detection area for AF in the camera, or the AE frame, which is the detection area for automatic exposure compensation (hereinafter referred to as AE). If there are multiple AF frames, the area in which the subject with the highest AF evaluation value is located may be set as the partial area. Furthermore, if the user adjusts the focus position and the focus position passes through a focus position that is in focus on a subject in a partial area of ​​an AF frame selected at least one step earlier, the AF frame in which the subject with the next highest AF evaluation value is located may be set as the partial area. Furthermore, if multiple AE frames exist, the area within the frame containing the subject with the exposure value closest to the correct exposure may be set as a partial area. In this case, the image processing device 100 can set the correct exposure value (for example, based on user input). 【0039】 The partial region setting unit 400 may also set a partial region based on user input. The method of setting a region by user input is not particularly limited; for example, the partial region may be specified using coordinate parameters with the upper left corner of the image as the origin, or the partial region may be specified by receiving user input on an image displayed on a touch panel. 【0040】 In this embodiment, the NR processing unit 105 performs NR processing in the second mode on the partial region set by the partial region setting unit 400. 【0041】 Next, with reference to Figure 5, the process by which the partial region setting unit 400 sets a partial region will be explained. Figure 5 is a flowchart showing an example of the process by which the partial region setting unit 400 sets a partial region. The process shown in Figure 5 may be performed immediately after S200 in Figure 2, or it may be performed independently of the process shown in Figure 2. 【0042】 In S500, the control unit 102 acquires subject information. Subject information according to this embodiment is information including a region (detection region) that indicates a subject detected in the image acquired by the input unit 101. As described above, the detection region may be detected by the image processing device 100 from the image, or the input unit 101 may acquire a detection region generated by the camera. The detection region is, for example, a region that surrounds the entire subject. If there are multiple detection regions, the region closest to the center of the image shall be selected as the detection region to be processed. Alternatively, for example, if there are multiple detection regions, the detection region located within the processing range of the NR processing in the first mode may be selected as the detection region to be processed. 【0043】 In S501, the input unit 101 acquires the AF frame or AE frame set in the camera. Here, the input unit 101 can acquire the AF frame or AE frame, for example, by communicating with the camera. If both the AF frame and AE frame are acquired, one of them (in this case, the AF frame) is selected. If neither the AF frame nor the AE frame is set in the camera, or if communication with the camera is not possible and the AF frame and AE frame cannot be acquired, the fact that the information could not be acquired is retained as information (for example, as flag information). 【0044】 In S502, the control unit 102 acquires a region based on user input (arbitrarily set by the user) as a candidate for a partial region. If no region based on user input is set in S502, the control unit 102 stores information (for example, as flag information) indicating that it could not be acquired. 【0045】 In S503, the control unit 102 sets the processing range in the image for NR processing in the first mode. Here, the control unit 102 accepts user input and sets a user-specified sub-region as the processing range for NR processing in the first mode, but for example, the entire image may be set as the processing range. 【0046】 In S504-S508, the partial area setting unit 400 sets the partial area that will be the processing range for NR processing in the second mode. In S504, the partial area setting unit 400 determines whether the area was acquired in S502. If it was acquired, the process proceeds to S505; otherwise, the process proceeds to S506. In S505, the partial area setting unit 400 sets the partial area that will be the processing range for NR processing in the second mode based on the area based on user input acquired in S502 and the processing range for NR processing in the first mode acquired in S503, and terminates the process shown in Figure 5. The detailed setting method in S504 will be described later. 【0047】 In S506, the partial area setting unit 400 determines whether an AF frame or AE frame was acquired in S501. If an AF frame or AE frame was acquired, the process proceeds to S507; otherwise, the process proceeds to S508. In S507, the partial area setting unit 400 sets a partial area that will be the processing range for NR processing in the second mode, based on the AF frame or AE frame acquired in S501 and the processing range for NR processing in the first mode acquired in S503, and terminates the process shown in Figure 5. A detailed setting method in S507 will be described later. 【0048】 In S508, the partial area setting unit 400 sets a partial area that will be the processing range for NR processing in the second mode, based on the subject information acquired in S500 and the processing range for NR processing in the first mode acquired in S503, and then terminates the processing shown in Figure 5. The detailed setting method in S508 will be described later. 【0049】 Here, we will explain the process assuming that a sub-region is set based on one of the regions acquired in S500 to S502. However, it is also possible that only one of S500 to S502 is executed, and only the corresponding process from S504 to S508 is executed. In this case, each region is acquired in S500 to S502, and the sub-region based on user input is used preferentially. This is because a region arbitrarily set by the user is likely to be a region where the user intends to focus on a subject. In the example in Figure 5, if there is no region arbitrarily set by the user, a sub-region is then set based on the AF frame or AE frame. This is because the AF frame or AE frame set by the camera is likely to be a region where the user wants to focus on a subject. 【0050】 Next, referring to Figure 6, the partial region setting process performed in S505, S507, and S508 will be explained. Here, the detection regions 602a, 602b, and 602c shown in Figure 6 will be explained as detection regions acquired in S500. However, the same processing can be performed even if these are AF frames or AE frames acquired in S501, or regions based on user input acquired in S502, so explanations for such examples will be omitted. 【0051】 Figure 6(a) shows an example of a partial region 601a in image 600a that is the processing range for NR processing in the first mode acquired in S503, and a detection region 602a included in the subject information acquired in S500. This is either an AF frame or AE frame acquired in S501, or a region arbitrarily set by the user acquired in S502. The selection method is as described above, according to the flowchart in Figure 5. Here, the detection region 602a is a smaller area than the partial region 601a, and it is assumed that the processing load when performing NR processing on the detection region 602a is smaller than the processing load when performing NR processing on the partial region 601a. ​​Therefore, in this case, the partial region setting unit 400 sets the detection region 602a as the partial region that will be the processing range for NR processing in the second mode. 【0052】 Next, Figure 6(b) will be explained. Figure 6(b) shows an example of a partial region 601b in image 600b that is the processing range for NR processing in the first mode acquired in S503, and a detection region 602b included in the subject information acquired in S500. In the example shown in Figure 6(b), the detection region 602b is equal to or larger than the partial region 601b, and they have an overlapping partial region 603b. In this case, the partial region setting unit 400 can, for example, set the partial region 603b as the partial region that will be the processing range for NR processing in the second mode. Alternatively, for example, the partial region setting unit 400 may set a region smaller than the partial region 601b, including the partial region 603b, as the partial region that will be the processing range for NR processing in the second mode. Since the region to be processed for NR in the first mode may be a region of interest to the user, including such a region in the partial region that will be the processing range for NR processing in the second mode makes it possible to provide a more user-friendly user experience. 【0053】 Next, Figure 6(c) will be explained. Figure 6(c) shows an example of a partial region 601c in image 600c that is the processing range for NR processing in the first mode acquired in S503, and a detection region 602c included in the subject information acquired in S500. In the example shown in Figure 6(c), the detection region 602c is equal to or larger than the partial region 601c, and they do not have overlapping regions. In such a case, the partial region setting unit 400 sets a partial region 603c, which is smaller than the partial region 601c and is included in the detection region 602c, as the partial region that will be the processing range for NR processing in the second mode. 【0054】 This processing method allows the system to set a sub-region to be processed in the second mode, determine whether the focus position is being adjusted, and then switch between the first mode and the second mode (which has a shorter delay than the first mode) for image noise reduction (NR) processing based on the result of that determination. This makes it possible to reduce the delay while improving the visibility of the main subject. Furthermore, after the focus position has been adjusted, the NR processing mode can be returned to the first mode, allowing for imaging while performing normal NR processing (especially with settings desired by the user). Thus, it has become possible to manually adjust the focus position to the desired subject while using the image recovery function. 【0055】 Furthermore, when the user sets an arbitrary area as the processing range for NR processing in the second mode, it may be possible to prevent the user from setting an area larger than the processing range for NR processing in the first mode. Also, when the processing range for NR processing in the second mode is set as the subject detection area, such a subject detection area may be set to be the smallest area containing the subject (in particular, a rectangular area). 【0056】 [Embodiment 3] In the embodiments described above, each processing unit shown in Figure 1, for example, may be implemented by dedicated hardware. Furthermore, some or all of the processing units of the image processing device 100 may be implemented by a computer. In this embodiment, at least a portion of the processing according to each embodiment described above is performed by a computer. 【0057】 Figure 7 shows the basic configuration of a computer. In Figure 7, the processor 701 is, for example, a CPU, which controls the operation of the entire computer. The memory 702 is, for example, RAM, which temporarily stores programs and data. The computer-readable storage medium 703 is, for example, a hard disk or CD-ROM, which stores programs and data long-term. In this embodiment, the programs that realize the functions of each part, which are stored in the storage medium 703, are read into the memory 702. Then, the processor 701 operates according to the programs in the memory 702, thereby realizing the functions of each part. 【0058】 In Figure 7, the input interface 704 is an interface for acquiring information from an external device. The output interface 705 is an interface for outputting information to an external device. The bus 706 connects the above-mentioned parts and enables data exchange. 【0059】 The disclosures herein include the following image processing apparatus, image processing methods, and programs. (Item 1) An acquisition means for acquiring an image captured by an imaging device, A determination means for determining whether or not the imaging conditions of the imaging device are being adjusted, A noise reduction means for reducing noise in the aforementioned image, A switching means that switches the process of reducing the noise of the image by the reduction means to a first reduction process when the imaging conditions of the imaging device are not being adjusted, and to a second reduction process which has a shorter delay time for image input / output in the noise reduction process than the first reduction process when the imaging conditions of the imaging device are being adjusted, A feature comprising: Image processing device. (Item 2) The image processing apparatus according to item 1, characterized in that the imaging condition is the focal position of the imaging device. (Item 3) The image processing apparatus according to item 2, characterized in that the determination means determines whether or not the focal position of the imaging device is being adjusted based on the position of the focus lens of the imaging device or an image evaluation value based on the pixel value of the image. (Item 4) The image processing apparatus according to item 2, characterized in that the determination means determines that the focal position of the imaging device is being adjusted when the imaging device receives a drive command for the focus lens. (Item 5) The image processing apparatus according to item 1, characterized in that the imaging conditions are conditions based on the zoom of the imaging device. (Item 6) The image processing apparatus according to item 1, characterized in that the imaging conditions are conditions based on the aperture of the imaging device. (Item 7) The image processing apparatus according to any one of items 1 to 6, characterized in that the second reduction process is a process of performing noise reduction on a region of the image that is smaller than the region that is processed in the first reduction process. (Item 8) The image processing apparatus according to item 7, characterized in that the processing range in the first reduction process is the entire image, and the processing range in the second reduction process is a partial region of the image. (Item 9) The image processing apparatus according to item 8, further comprising a first setting means for setting the aforementioned partial region. (Item 10) The image processing apparatus according to item 9, characterized in that the first setting means sets the partial region based on user input. (Item 11) The image processing apparatus according to item 9, characterized in that the first setting means sets the partial region based on the AF frame that performs autofocus in the image of the imaging device. (Item 12) The image processing apparatus according to item 11, characterized in that, when there are multiple AF frames, the AF frame in which the subject with the highest AF evaluation value is located is set as the partial region. (Item 13) The image processing apparatus according to item 9, characterized in that the first setting means sets the partial region based on the AE frame, which is the detection area for automatic exposure correction in the image of the imaging device. (Item 14) It further includes a second setting means for setting the correct exposure, The image processing apparatus according to item 13, wherein, when there are multiple AE frames, the first setting means sets the AE frame whose exposure value of the subject within the frame is closest to the appropriate exposure as the partial region. (Item 15) The image processing apparatus according to item 1, characterized in that the second reduction process is a process of reducing the noise of the image using a machine learning model different from the machine learning model used in the first reduction process. (Item 16) The image processing apparatus according to any one of items 1 to 15, characterized in that the determination means determines whether or not the imaging conditions are being adjusted based on whether or not a user is operating the imaging device. (Item 17) The image processing apparatus according to any one of items 1 to 16, further comprising a display means for displaying information indicating whether the reduction means is performing the first reduction process or the second reduction process as a process for reducing noise in the image. (Item 18) The image processing apparatus according to any one of items 1 to 17, further comprising a third setting means for setting whether or not to perform a process to switch the process of reducing noise in the image by the switching means. (Item 19) The acquisition process involves acquiring the image captured by the imaging device, A determination step of determining whether or not the imaging conditions of the imaging device are being adjusted, A reduction step to reduce noise in the aforementioned image, A switching step is provided to switch the process for reducing the noise of the image by the reduction step to a first reduction process if the imaging conditions of the imaging device are not being adjusted, and to a second reduction process that has a shorter delay time for image input / output in the noise reduction process than the first reduction process if the imaging conditions of the imaging device are being adjusted. A feature comprising: Image processing methods. (Item 20) A program to cause a computer to function as one of the means of an image processing device described in any one of items 1 through 18. 【0060】 (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. 【0061】 The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols] 【0062】 100: Image processing device, 101: Input unit, 102: Control unit, 103: Adjustment determination unit, 104: Mode switching unit, 105: NR processing unit, 106: Output unit

Claims

[Claim 1] An acquisition means for acquiring an image captured by an imaging device, A determination means for determining whether or not the imaging conditions of the imaging device are being adjusted, A noise reduction means for reducing noise in the aforementioned image, A switching means that switches the process of reducing the noise of the image by the reduction means to a first reduction process when the imaging conditions of the imaging device are not being adjusted, and to a second reduction process which has a shorter delay time for image input / output in the noise reduction process than the first reduction process when the imaging conditions of the imaging device are being adjusted, An image processing apparatus characterized by comprising: [Claim 2] The image processing apparatus according to claim 1, characterized in that the imaging condition is the focal position of the imaging device. [Claim 3] The image processing apparatus according to claim 2, characterized in that the determination means determines whether or not the focal position of the imaging device is being adjusted based on the position of the focus lens of the imaging device or an image evaluation value based on the pixel value of the image. [Claim 4] The image processing apparatus according to claim 2, characterized in that the determination means determines that the focal position of the imaging device is being adjusted when the imaging device receives a drive command for the focus lens. [Claim 5] The image processing apparatus according to claim 1, characterized in that the imaging conditions are conditions based on the zoom of the imaging device. [Claim 6] The image processing apparatus according to claim 1, characterized in that the imaging conditions are conditions based on the aperture of the imaging device. [Claim 7] The image processing apparatus according to claim 1, characterized in that the second reduction process is a process of performing noise reduction on an area of ​​the image that is smaller than the processing area of ​​the first reduction process. [Claim 8] The image processing apparatus according to claim 7, characterized in that the processing range in the first reduction process is the entire image, and the processing range in the second reduction process is a partial region of the image. [Claim 9] The image processing apparatus according to claim 8, further comprising a first setting means for setting the aforementioned partial region. [Claim 10] The image processing apparatus according to claim 9, characterized in that the first setting means sets the partial region based on user input. [Claim 11] The image processing apparatus according to claim 9, characterized in that the first setting means sets the partial region based on the AF frame that performs autofocus in the image of the imaging device. [Claim 12] The image processing apparatus according to claim 11, characterized in that, when there are multiple AF frames, the AF frame in which the subject with the highest AF evaluation value is located is set as the partial region. [Claim 13] The image processing apparatus according to claim 9, characterized in that the first setting means sets the partial region based on the AE frame, which is the detection region for automatic exposure correction in the image of the imaging device. [Claim 14] It further includes a second setting means for setting the correct exposure, The image processing apparatus according to claim 13, characterized in that, when there are multiple AE frames, the first setting means sets the AE frame whose exposure value of the subject within the frame is closest to the appropriate exposure as the partial region. [Claim 15] The image processing apparatus according to claim 1, characterized in that the second reduction process is a process of reducing the noise of the image using a machine learning model different from the machine learning model used in the first reduction process. [Claim 16] The image processing apparatus according to claim 1, characterized in that the determination means determines whether or not the imaging conditions are being adjusted based on whether or not a user is operating the imaging device. [Claim 17] The image processing apparatus according to claim 1, further comprising a display means for displaying information indicating whether the reduction means is performing the first reduction process or the second reduction process as a process for reducing noise in the image. [Claim 18] The image processing apparatus according to claim 1, further comprising a third setting means for setting whether or not to perform a process to switch the process for reducing noise in the image by the switching means. [Claim 19] The acquisition process involves acquiring the image captured by the imaging device, A determination step of determining whether or not the imaging conditions of the imaging device are being adjusted, A reduction step to reduce noise in the aforementioned image, A switching step is provided to switch the process for reducing the noise of the image by the reduction step to a first reduction process if the imaging conditions of the imaging device are not being adjusted, and to a second reduction process that has a shorter delay time for image input / output in the noise reduction process than the first reduction process if the imaging conditions of the imaging device are being adjusted. An image processing method characterized by comprising: [Claim 20] A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 18.

Images