Image processing apparatus and control method thereof

Abstract

By utilizing noise reduction (NR) processing, status information regarding the imaging state is provided to the user with less delay. [Solution] The image processing device includes an input means for sequentially inputting frame images obtained by imaging by an imaging unit; an NR processing means for applying noise reduction (NR) processing to the frame images to generate NR frame images; a first derivation means for deriving first state information relating to the imaging state in the frame images from the frame images; a superimposing means for generating a superimposed image by superimposing state information based on the first state information obtained from a frame image captured by the imaging unit at a second time following the first time onto the NR frame image generated from the frame image captured by the imaging unit at a first time; and an output means for sequentially outputting the superimposed image corresponding to the frame image input by the input means to the display unit.

Description

Technical Field

[0001] The present invention relates to a technique for assisting focus confirmation.

Background Art

[0002] [[ID=X]]Conventionally, a technique for amplifying the brightness of an image by increasing the gain of the image under low illumination conditions is known. However, noise may be added by increasing the gain. In order to compensate for the decrease in the visibility of the subject due to the addition of noise, a technique for reducing noise in an image using image processing such as noise reduction (NR) is known.

[0003] Patent Document 1 discloses a technique for performing AF processing based on an image signal subjected to NR in order to improve the accuracy of autofocus (AF). Further, Patent Document 2 discloses a technique for switching, arranging, or overlapping and displaying a pre-correction image and a post-correction image by image quality improvement processing.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] Incidentally, processing delays exist in image processing that has a high processing load, such as noise reduction (NR) processing. In particular, NR using deep learning (DL) has the problem of high NR effectiveness but also high processing delay. Therefore, in use cases where the user visually views an NR-processed image and adjusts the focus position, it becomes difficult to confirm the accurate focus position due to the time delay in image display. However, the aforementioned Patent Document 1 does not take into consideration the effect of delays caused by NR processing. Furthermore, as described in Patent Document 2, switching between displaying the image signal before and after NR processing reduces the convenience for the photographer.

[0006] This invention has been made in view of these problems, and aims to provide users with state information regarding the imaging state with less delay while utilizing noise reduction (NR) processing. [Means for solving the problem]

[0007] To solve the above-mentioned problems, the image processing apparatus according to the present invention has the following configuration. That is, the image processing apparatus is An input means for sequentially inputting frame images obtained by imaging by the imaging unit, An NR processing means that applies noise reduction (NR) processing to the aforementioned frame image to generate an NR frame image, A first derivation means for deriving first state information relating to the imaging state in the frame image from the frame image, A superimposing means for generating a superimposed image by superimposing state information based on first state information obtained from a frame image captured by the imaging unit at a second time following the first time onto an NR frame image generated from a frame image captured by the imaging unit at a first time; Output means that sequentially outputs superimposed images corresponding to frame images input by the input means to the display unit, It has. Alternatively, the image processing device is An input means for sequentially inputting frame images obtained by imaging by the imaging unit, An NR processing means that applies NR processing to the aforementioned frame image to generate an NR frame image, A first acquisition means for acquiring field-of-view information that defines the shooting range of the frame image, A superimposing means for generating a superimposed image by superimposing field-of-view information for a frame image captured by the imaging unit at a second time following the first time onto an NR frame image generated from a frame image captured by the imaging unit at a first time; Output means that sequentially outputs superimposed images corresponding to frame images input by the input means to the display unit, It has. [Effects of the Invention]

[0008] According to the present invention, state information regarding the imaging state can be provided to the user with less delay while utilizing noise reduction (NR) processing. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing the configuration of the image processing apparatus in the first embodiment. [Figure 2] This is a flowchart of the process in the first embodiment. [Figure 3] This diagram illustrates the relationship between focus information and the output timing of the noise reduction processed image. [Figure 4] This is a diagram showing the output image. [Figure 5] This figure shows the functional configuration of the image processing apparatus in the second embodiment. [Figure 6] This is a flowchart of the process in the second embodiment. [Figure 7] This diagram explains the calculation of the third focus information. [Figure 8] This figure shows the functional configuration of the image processing apparatus in the third embodiment. [Figure 9] This is a flowchart of the process in the third embodiment. [Figure 10] This diagram explains the selection process for candidate focus information. [Figure 11] This is a diagram showing the overall configuration of the imaging system.

Embodiments for Carrying out the Invention

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] (First Embodiment) As a first embodiment of the image processing apparatus according to the present invention, a shooting system that performs focus control using noise reduction (NR) processing will be described below as an example.

[0012] <System Configuration and Apparatus Configuration> FIG. 11 is a diagram showing the overall configuration of the shooting system. The shooting system includes a camera 1100 which is an imaging unit, an image processing apparatus 1101, an operation member 1102, a lens 1103, and a monitor 11(which is a display unit). In FIG. 11, the camera 1100, the image processing apparatus 1101, and the monitor 1104 are described as a plurality of separate apparatuses, but they may be configured as one apparatus.

[0013] The image processing device 1101 is configured to apply noise reduction (NR) processing to images captured by the camera 1100. The images captured by the camera 1100 are output as video at, for example, 60 frames per second (fps). The photographer can check the current focus position by looking at the NR-processed image displayed on the monitor 1104. When the photographer manually adjusts the focus position to a desired subject, they adjust the focus position by operating the control member 1102 while looking at the monitor 1104 to drive the focus lens of the lens 1103. With this configuration, for example, even if the camera 1100 is shooting in a dark place and the brightness gain is high, the photographer can adjust the focus position by looking at the image with improved visibility of the subject due to NR processing.

[0014] Figure 1(a) is a diagram showing the functional configuration of the image processing apparatus in the first embodiment. Image processing apparatus 100 corresponds to image processing apparatus 1101 in Figure 11.

[0015] The input unit 101 is a functional unit that receives image signals (for example, signals corresponding to frame images) sequentially output from the camera 1100 and transfers them to the control unit 102. The control unit 102 performs various processing on the image signals received from the input unit 101. As will be described in detail later, these various processes include calculating focus information, noise reduction processing, and image superposition processing.

[0016] The first focus information calculation unit 103 derives first focus information from the image signal as imaging state information for that image signal. The first focus information is, for example, the contrast value of the image indicated by the image signal. A higher contrast value indicates that the focus position is in sync with the subject.

[0017] The NR processing unit 104 performs NR processing on the image signal. The method of NR processing is not limited, but it is assumed to be NR processing that involves processing delays equivalent to several frames, such as NR processing using deep learning (DL) or NR processing using multiple input frames.

[0018] The superposition unit 105 superimposes the first focus information calculated by the first focus information calculation unit 103 onto the image signal (NR frame image) that has undergone NR processing by the NR processing unit 104. For example, the numerical value (string image) of the focus information is superimposed on a predetermined position (such as the lower or upper part) of the image signal (frame image). The numerical value may be displayed as a percentage. Alternatively, it may be displayed using a graphical user interface (GUI) component such as a bar / gauge.

[0019] The output unit 106 sequentially outputs the image signal (superimposed image) on which the first focus information has been superimposed by the superimposition unit 105 to an external device (for example, a monitor 1104) of the image processing device 100.

[0020] Figure 1(b) shows the hardware configuration of the image processing device. The image processing device can be configured using a general-purpose computer (PC, etc.). Here, it is assumed that the various processes of the control unit 102 shown in Figure 1(a) are implemented in software (i.e., the CPU executes the program), but some or all of them may be implemented in hardware (for example, using an application-specific integrated circuit (ASIC)).

[0021] The CPU 111 performs various processes by executing programs. Random access memory (RAM) 112 is used as the CPU 111's main memory, work area, and other temporary storage areas. The control program is stored in read-only memory (ROM) 113.

[0022] The neural network (NN) 114 is an computation unit that performs inference (e.g., NR processing) using a machine learning model pre-trained by deep learning. The storage 115 is a hard disk drive (HDD) or solid-state drive (SSD) for storing various data and programs according to this embodiment. An external storage device may be used to perform a similar role. Here, the external storage device can be realized, for example, by a media (recording medium) and an external storage drive for accessing the media. Alternatively, the external storage device may be a server device connected via a network.

[0023] <Device Operation> Figure 2 is a flowchart of the process in the first embodiment. This process starts when the power to the image processing device 100 is turned ON, for example, when the CPU 111 executes a control program stored in the ROM 113. The following processes are performed for each input frame of image signal.

[0024] In S200, the input unit 101 acquires an image signal from the camera 1101. In S201, the first focus information calculation unit 103 calculates first focus information from the image signal acquired in S200. The first focus information is, for example, a contrast value. Alternatively, the first focus information may not be calculated from the image signal, but may be acquired from a camera (not shown) by providing a communication unit that communicates with the camera. When communicating with a camera, the defocus amount obtained from the camera's phase difference sensor may also be used as the first focus information.

[0025] In S202, the NR processing unit 104 performs NR processing on the image signal acquired in S200. In S203, the superposition unit 105 superimposes the first focus information calculated in S201 onto the image signal that has undergone NR processing in S202. In S204, the output unit 106 outputs the image signal with the first focus information superimposed in S203 to the outside of the image processing device 100.

[0026] <Details of the superposition process (S203)> In the superposition process (S203), the first focus information is superimposed on the image signal that has undergone NR processing. However, the frame corresponding to the image signal and the frame from which the first focus information was calculated are different. The reason for this is explained below.

[0027] Figure 3 illustrates the relationship between focus information and the output timing of NR-processed images. The horizontal axis represents elapsed time, assuming that frame images are input sequentially every 1 / 60th of a second. Each rectangle represents a frame, and the same number indicates the same input frame. In other words, Figure 3(a) exemplifies the timing at which frames 1 to 9 are input to the NR processing unit 104. Figure 3(b) exemplifies the timing at which frames 1' to 4', which have undergone NR processing by the NR processing unit 104 from frames 1 to 4 mentioned above, are output.

[0028] In other words, this shows the timing relationship between the image signals before NR processing (frames 1 to 9) and the image signals after NR processing (frames 1' to 4'). As can be seen from the figure, NR processing causes a delay of 5 frames. Note that the processing delay due to NR processing varies depending on the processing power of the CPU or GPU, as well as the NR processing method and algorithm. For example, it varies depending on the size of the machine learning model and the number of frames used.

[0029] The time t306a in Figure 3(a) (input timing of frame 6) and the time t306b in Figure 3(b) (output timing of frame 1') are the same. The first focus information calculation unit 103 calculates the first focus information based on frame 6 acquired by the input unit 101 at time t306a. Meanwhile, the NR processing unit 104 generates frame 1' (a frame obtained by applying NR processing to frame 1) at time t306b. Then, the superposition unit 105 superimposes the first focus information calculated at time t306a (=time t306b) (i.e., focus information calculated based on frame 6) onto frame 1'.

[0030] Generally, the calculation of focus information by the first focus information calculation unit 103 is processed almost in real time (for example, in less than one frame). Therefore, the superposition unit 105 can superimpose the first focus information calculated based on the (highly real-time) frame 6 onto frame 1'. The same superposition process is performed for subsequent frames.

[0031] In other words, when the superimposing unit 105 superimposes the first focus information onto the NR-processed image signal, it superimposes the first focus information calculated from the latest input image signal obtained at the time of superimposing (the image signal before NR processing). As a result, the image signal output from the output unit 106 (the image displayed on the monitor 1104) becomes an image signal that has undergone NR processing (which has a processing delay) with the first focus information (which is highly real-time) superimposed on it.

[0032] Figure 4 shows the output image. Specifically, it shows the image signal output by the output unit 106. The output image 400 is the image shown by the image signal output from the output unit 106. The output image 400 includes the main subject 401 that the photographer wants to focus on. The main subject is specified by the photographer arbitrarily specifying a partial region within the output image 400. In this case, the coordinates of the partial region within the output image 400 are specified, for example, via the operating member 1102. The camera 1100 or the image processing device 1101 may be configured to specify a partial region within the output image 400 based, for example, on object detection.

[0033] As described above, the output image 400 has undergone noise reduction processing by the NR processing unit 104, resulting in a noise-reduced image. Figure 4 illustrates a state where the focus is not on the main subject 401. In addition, GUI components 402-405 are superimposed on the output image 400 as first focus information.

[0034] GUI component 402 displays a numerical value (e.g., contrast value) indicating the degree of focus on the main subject 401. A higher value indicates a better focus position. GUI components 403-405 graphically represent the first focus information. GUI component 405 shows a distance range (e.g., the left side is close distance and the right side is far distance), with the dotted line 403 representing the current focus position and the solid line 404 representing the focus position of the main subject. In other words, the dotted line 403 moves left or right within the distance range 405 according to the degree of focus on the main subject. By displaying the first focus information as shown in GUI components 403-405, the photographer can understand whether the current focus position is in front of or behind the focus position of the main subject. This is effective when the first focus information is information that shows the front or back focus position, such as the amount of defocus.

[0035] The output image 400 is an image in which the visibility of the subject has been improved as a result of NR processing. However, as mentioned above with reference to Figure 3, the video has a delay of 5 frames relative to the current time. Therefore, when the photographer operates the operating member 1102 to drive the focus lens of the lens 1103, they will feel a sense of discomfort because the response of the displayed image to the operation is poor (delayed).

[0036] On the other hand, the first focus information superimposed as GUI components 402-405 in the output image 400 is calculated from the image signal before NR processing is applied. In other words, it is information with little delay relative to the current time (high real-time performance). This means that the first focus information superimposed as GUI components 402-405 responds well to the operation of the operating member 1102 (low delay). Therefore, the photographer can control the focus without feeling any discomfort by operating the operating member 1102 while referring to the first focus information superimposed as GUI components 402-405.

[0037] As described above, according to the first embodiment, highly real-time first focus information is superimposed on the image signal of the image displayed to the user (photographer). As a result, even if there is a delay in image display due to NR processing or the like, the photographer can control the focus without feeling any discomfort by referring to the first focus information.

[0038] Furthermore, the first focus information may be calculated from a portion of the image signal. For example, the photographer may specify an arbitrary region of the image signal using an input component, and the first focus information may be calculated for the specified region. Alternatively, the AF frame, which is the autofocus (AF) detection region set in the camera, may be obtained, and the first focus information may be calculated for the region corresponding to the AF frame.

[0039] Alternatively, the superimposing unit 105 may superimpose the first focus information onto the image signal only when NR processing is enabled. Or, the system may be configured so that the photographer can set whether or not to superimpose the first focus information onto the image signal via the input unit.

[0040] (Second Embodiment) In the second embodiment, a configuration is described in which focus information calculated from an image signal that has undergone NR processing is also used. In the following, the parts that differ from the first embodiment will be described, and the same configurations and processes as in the first embodiment will not be described.

[0041] <Device configuration> Figure 5 shows the functional configuration of the image processing apparatus in the second embodiment. The differences from the first embodiment (Figure 1) will be described below.

[0042] The second focus information calculation unit 500 calculates second focus information, which is focus information, from the image signal that has been subjected to NR processing by the NR processing unit 104. Here, both the first focus information and the second focus information are assumed to be contrast values. The third focus information calculation unit 501 calculates third focus information based on the first focus information and the second focus information. The superposition unit 105 superimposes the third focus information onto the image signal that has been subjected to NR processing by the NR processing unit 104.

[0043] <Device Operation> Figure 6 is a flowchart of the process in the second embodiment. Similar to the first embodiment (Figure 2), this process starts when the power of the image processing device 100 is turned ON, for example, by the CPU 111 executing a control program stored in the ROM 113.

[0044] In S600, the first focus information calculation unit 103 stores the first focus information derived in S201 in memory (such as RAM 112). In S601, the second focus information calculation unit 500 derives the second focus information based on the image signal that has undergone NR processing in S202.

[0045] In S602, the third focus information calculation unit 501 determines whether or not it is time to update the offset value. As will be described in detail later, the offset value is calculated as the difference between the first focus information and the second focus information, and corresponds to the degree of influence on the contrast value of an image caused by NR processing being applied to that image. Here, it is determined that it is time to update when the photographer has finished adjusting the focus position (when the photographer has not operated the operating member 1102). For example, the latest first focus information (stored in the most recent S600) is compared with the first focus information calculated one or more frames prior, and it is determined that the focus position adjustment is complete if the value has not changed or the amount of change is less than a certain amount. The certain amount is, for example, a value corresponding to the depth of field. Alternatively, the position of the focus lens may be obtained from the camera, and it may be determined that the focus position adjustment is complete if the position of the focus lens has not changed. If it is time to update, proceed to S603. On the other hand, if it is not time to update, proceed to S604.

[0046] In S603, the third focus information calculation unit 501 determines (updates) the offset value. The offset value is derived based on the first focus information calculated at least one frame earlier than the latest (stored in the most recent S600) first focus information, and the second focus information calculated in the most recent S601. Specifically, the difference between the first focus information at least one frame earlier and the second focus information is used as the offset value. The initial value of the offset value is 0 or the value calculated during the factory adjustment process. When updating the offset value, it may be updated with the calculated difference itself, or it may be updated with a value that takes into account the offset value of one or more previous offset values ​​(average or weighted).

[0047] In S604, the third focus information calculation unit 501 calculates the third focus information based on the offset value and the first focus information calculated in S201. Specifically, the offset value is added to the first focus information. Weighting may be applied when adding the values.

[0048] In S203, the superimposing section 105 superimposes the third focus information calculated in S604 onto the image signal that has undergone NR processing in S202.

[0049] Figure 7 illustrates the calculation of the third focus information. The horizontal axis represents elapsed time, assuming that frame images are input sequentially every 1 / 60th of a second. Each rectangle represents a frame, and the same number indicates the same input frame. It is also assumed that the photographer did not adjust the focus position at time t706a.

[0050] At time t706a, the third focus information calculation unit 501 updates the offset value. Here, the offset value is updated based on the first focus information calculated at time t701a based on frame 1 and the second focus information calculated at time t706b (time t706a) based on frame 1' which has undergone NR processing. Next, the offset value is added to the first focus information calculated based on frame 6 to calculate the third focus information.

[0051] In other words, the third focus information is focus information that reflects the effect of applying NR processing to frame 6 at time 706a. Subsequently, the superposition unit 105 superimposes the third focus information onto frame 1'. In the same manner, the third focus information is calculated for subsequent frames and superimposed onto the NR-processed image signal.

[0052] Furthermore, if it is not the update timing, the third focus information is calculated based on the offset value calculated at a point one or more frames prior. In other words, the offset value is calculated based on the first focus information calculated before NR processing is applied to a given image signal, and the second focus information calculated after NR processing is applied.

[0053] As described above, according to the second embodiment, the superimposed focus information utilizes information based on the second focus information calculated from the NR-processed image signal. Specifically, a third focus information calculated based on the first and second focus information is superimposed. This allows the photographer to perform focus control with less unnaturalness by referring to the third focus information.

[0054] In the explanation above, the timing for updating the offset value was determined by whether or not the focus position adjustment was complete. However, the timing for updating the offset value could also be determined when the composition changes. Changes in composition can be determined by recognizing the subject in the image signal and observing the movement of the subject. Alternatively, information on the shooting direction (pan, tilt) can be obtained from the camera and used to determine the change.

[0055] (Third embodiment) In the third embodiment, a configuration in which the defocus amount is used as the focus information will be described. In addition, second focus information calculated from an image signal that has undergone NR processing, similar to the second embodiment, will be used. The following describes the parts that differ from the first and second embodiments.

[0056] <Device configuration> Figure 8 shows the functional configuration of the image processing apparatus in the third embodiment. The differences from the block diagrams of the first and second embodiments (Figures 2 and 5) will be explained below.

[0057] Sensor 800 is a sensor capable of capturing images of a subject and detecting the amount of defocus used in phase-detection autofocus (AF). In phase-detection AF, light beams from a subject that have passed through different exit pupil regions of the photographic lens are typically imaged onto a pair of line sensors to calculate the correlation amount and the amount of defocus. Note that the image sensor and the phase-detection sensor may be provided separately, as long as the configuration allows for the acquisition of the amount of defocus.

[0058] The first focus information calculation unit 103 calculates first focus information as imaging state information for the AB image from a pair of phase images (AB images) formed on the imaging surface of the sensor 800. The AB image NR processing unit 801 performs NR processing on the AB image. The AB image NR processing unit 801 may perform NR processing similar to that of the NR processing unit 104 or different NR processing.

[0059] The second focus information calculation unit 500 calculates the second focus information from the AB image that has undergone NR processing by the AB image NR processing unit 801. Here, the first focus information and the second focus information are defocus amount or correlation amount.

[0060] The third focus information calculation unit 501 calculates the third focus information based on the first focus information and the second focus information. The image processing unit 802 performs image processing on the image signal captured by the sensor 800. Image processing includes gamma correction and demosaicing (debayering). The NR processing unit 104 may be integrated with the image processing unit 802. The superposition unit 105 superimposes the third focus information onto the image signal that has been NR processed by the NR processing unit 104.

[0061] <Device Operation> Figure 9 is a flowchart of the process in the third embodiment. Similar to the first and second embodiments (Figures 2 and 6), this process starts when the power to the image processing device 100 is turned ON, for example, by the CPU 111 executing a control program stored in the ROM 113.

[0062] In S200, the control unit 102 acquires an image signal from the sensor 800. In S900, the control unit 102 acquires an A-B image from the sensor 800. In S201, the first focus information calculation unit 103 calculates first focus information from the A-B image acquired in S900. One or more defocus amounts that are candidate focal positions are calculated as first focus information. In other words, at least one defocus amount is calculated as first focus information. In S600, the first focus information calculation unit 103 stores at least one first focus information calculated in S201 in memory.

[0063] In S901, the AB image NR processing unit 801 performs NR processing on the AB image acquired in S900. In S902, the image processing unit 802 performs image processing on the image signal acquired in S200. In S202, the NR processing unit 104 performs NR processing on the image signal that has been processed in S902.

[0064] In S601, the second focus information calculation unit 500 calculates second focus information from the AB image that has undergone NR processing in S901. Similar to the first focus information, at least one second focus information is calculated.

[0065] In S903, the third focus information calculation unit 501 selects the focus information to be superimposed in the superimposition unit 105 from the focus information stored in memory (a set of at least one first focus information and one or more second focus information from at least one frame prior). The selection method will be described later.

[0066] From S602 onward, third focus information is calculated based on the focus information selected in S903, in the same manner as in the second embodiment, superimposed on the NR-processed image signal in the superposition unit 105, and output by the output unit 106.

[0067] Figure 10 illustrates the selection of candidate focus information. In Figure 10, the horizontal axis shows the amount of defocus, and the current focal position of 1000 corresponds to a defocus amount of "0".

[0068] In S201, the first focus information calculation unit 103 calculates four first focus information pieces 1001 to 1004, which are candidate focal positions for the target. In S601, the second focus information calculation unit 500 calculates three second focus information pieces 1005 to 1007, which are candidate focal positions for the target.

[0069] As described above, the first focus information is calculated based on the AB image before NR processing, and the second focus information is calculated based on the AB image after NR processing. Therefore, the second focus information is calculated later than the first focus information by the amount of time required for NR processing. As a result, the first focus information is calculated before the second focus information and is stored in memory.

[0070] In S903, the third focus information calculation unit 501 selects the focus information to be superimposed in the superimposition unit 105. Here, the selection is made from a set of four first focus information pieces 1001 to 1004 and three second focus information pieces 1005 to 1007.

[0071] Specifically, from the four first focus information points, we select candidate focal positions where a second focus information point exists in the vicinity of each. In the case of Figure 10, the second focus information point 1005 exists in the vicinity (within a predetermined distance) of the first focus information point 1002, the second focus information point 1006 exists in the vicinity of the first focus information point 1003, and the second focus information point 1007 exists in the vicinity of the first focus information point 1004. Therefore, these are retained as three candidate focal positions. On the other hand, since there is no second focus information point in the vicinity of the first focus information point 1001, it is not selected as a candidate focal position. The vicinity can be an arbitrarily defined range, but it is preferable that the difference in the amount of defocus of each point is within the depth of field.

[0072] Here, the first focus information is calculated from the AB image, which contains a lot of noise. On the other hand, the second focus information is calculated from the AB image, which has undergone noise reduction (NR) processing. Therefore, the second focus information is more accurate as focus information compared to the first focus information. As a result, the selection process described above makes it possible to exclude the first focus information 1001, which is likely to be less accurate, from the focal position candidates. This makes it possible to improve the accuracy of the focus information.

[0073] Next, select one or more of the three first focus information points that were left as candidate focus positions, and calculate one or more third focus information points. There may be multiple third focus information points. However, if you select only one, it is best to select, for example, the first focus information point 1002 that is closest to the current focus position 1000 (i.e., the one with the smallest absolute value of defocus).

[0074] As explained above, according to the third embodiment, the amount of defocus calculated from the AB image is used as the focus information. Furthermore, by using the second focus information to select the first focus information, it is possible to further improve the accuracy of the focus information.

[0075] In the above explanation, the second focus information is calculated later than the first focus information by the amount of time required for NR processing. Therefore, the first focus information from one or more frames prior, stored in memory, was used to select candidate focus positions. However, the current focus position 1000 may also change during the time required for NR processing. Therefore, the third focus information may be calculated taking into account the change in the current focus position 1000. For example, the system may be configured to add the amount of change in the current focus position 1000 during the time required for NR processing to the third focus information.

[0076] (modified version) The above-described embodiment explained an example of superimposing focus information onto an image. However, it is also effective to superimpose field-of-view information onto the image signal as imaging state information in the image. Field-of-view information refers to information that defines the shooting range, and includes pan information, tilt information, zoom information (PTZ value), etc.

[0077] Even when controlling the field of view manually via an operating device, similar to focus control, a poor response (delay) in the displayed image to the operation will cause discomfort to the photographer. Therefore, by superimposing field of view information onto the image, the photographer can refer to highly real-time field of view information. As a result, even if there is a delay in image display due to noise reduction processing, the photographer can control the field of view without feeling any discomfort by referring to the field of view information. Alternatively, the system may be configured to superimpose both focus information and field of view information onto the image.

[0078] The disclosures herein include the following image processing apparatus, control methods, and programs. (Item 1) An input means for sequentially inputting frame images obtained by imaging by the imaging unit, An NR processing means that applies noise reduction (NR) processing to the aforementioned frame image to generate an NR frame image, A first derivation means for deriving first state information relating to the imaging state in the frame image from the frame image, A superimposing means for generating a superimposed image by superimposing state information based on first state information obtained from a frame image captured by the imaging unit at a second time following the first time onto an NR frame image generated from a frame image captured by the imaging unit at a first time; Output means that sequentially outputs superimposed images corresponding to frame images input by the input means to the display unit, An image processing apparatus characterized by having (Item 2) The first state information is information relating to the contrast in the frame image. The image processing apparatus according to item 1, characterized in that it is a picture processing apparatus. (Item 3) A second derivation means for deriving a second state information relating to the imaging state in the NR frame image from the NR frame image, A third derivation means for deriving a third state information based on the first state information and the second state information, It further possesses, The second state information is information regarding the contrast in the NR frame image, The third derivation means derives the third state information by adding an offset value caused by the NR processing to the first state information obtained from the frame image captured by the imaging unit at the second time, The superposition means generates a superimposed image by superimposing the state information based on the third state information onto the NR frame image generated from the frame image captured by the imaging unit at a first time. The image processing apparatus according to item 2, characterized in that (Item 4) The third derivation means determines the offset value based on the difference between the first state information and the second state information corresponding to the frame image captured by the imaging unit at the first time. The image processing apparatus according to item 3, characterized in that (Item 5) The input means inputs the frame image as a pair of phase images, The first state information is information regarding the amount of defocus in the frame image. The image processing apparatus according to item 1, characterized in that it is a picture processing apparatus. (Item 6) A second derivation means for deriving a second state information relating to the imaging state in the NR frame image from the NR frame image, A third derivation means for deriving a third state information based on the first state information and the second state information, It further possesses, The second state information is information regarding the amount of defocus in the NR frame image, The third derivation means derives the third state information by adding an offset value caused by the NR processing to the first state information obtained from the frame image captured by the imaging unit at the second time, The superposition means generates a superimposed image by superimposing the state information based on the third state information onto the NR frame image generated from the frame image captured by the imaging unit at a first time. The image processing apparatus according to item 5, characterized in that... (Item 7) The system further includes a selection means for selecting one or more first state information from a plurality of state information, which includes one or more first state information derived by the first derivation means and one or more second state information derived by the second derivation means, such that a pair of second state information exists within a predetermined distance. The third derivation means derives the third state information based on one or more first state information selected by the selection means. The image processing apparatus according to item 6, characterized in that (Item 8) The system further includes a means for specifying a sub-region of the frame image, The first derivation means derives the imaging state of the partial region of the frame image as first state information. An image processing apparatus according to any one of items 1 to 7, characterized by the above. (Item 9) An input means for sequentially inputting frame images obtained by imaging by the imaging unit, An NR processing means that applies NR processing to the aforementioned frame image to generate an NR frame image, A first acquisition means for acquiring field-of-view information that defines the shooting range of the frame image, A superimposing means for generating a superimposed image by superimposing field-of-view information for a frame image captured by the imaging unit at a second time following the first time onto an NR frame image generated from a frame image captured by the imaging unit at a first time; Output means that sequentially outputs superimposed images corresponding to frame images input by the input means to the display unit, An image processing apparatus characterized by having (Item 10) The angle of view information includes at least one of the pan information, tilt information, and zoom information of the imaging unit. The image processing apparatus according to item 9, characterized in that (Item 11) A method for controlling an image processing device, An input process in which frame images obtained by imaging by the imaging unit are sequentially input, A noise reduction (NR) processing step is performed on the aforementioned frame image to generate an NR frame image. A first derivation step of deriving first state information relating to the imaging state in the frame image from the frame image, A superposition step is to generate a superimposed image by superimposing state information based on first state information obtained from a frame image captured by the imaging unit at a second time following the first time onto an NR frame image generated from a frame image captured by the imaging unit at a first time; An output step sequentially outputs superimposed images corresponding to the frame images input in the input step to the display unit, A control method characterized by including (Item 12) A method for controlling an image processing device, An input process in which frame images obtained by imaging by the imaging unit are sequentially input, An NR processing step is performed on the aforementioned frame image to generate an NR frame image, A first acquisition step of acquiring field-of-view information that defines the shooting range of the frame image, A superposition step is to generate a superimposed image by superimposing field-of-view information for a frame image captured by the imaging unit at a second time following the first time onto an NR frame image generated from a frame image captured by the imaging unit at a first time; An output step sequentially outputs superimposed images corresponding to the frame images input in the input step to the display unit, A control method characterized by including (Item 13) A program that causes a computer to perform the control methods described in item 11 or 12.

[0079] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0080] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0081] 100 Image processing device; 103 First focus information calculation unit; 105 Overlay unit; 500 Second focus information calculation unit; 501 Third focus information calculation unit; 801 AB image NR processing unit

Claims

1. An input means for sequentially inputting frame images obtained by imaging by the imaging unit, An NR processing means that applies noise reduction (NR) processing to the aforementioned frame image to generate an NR frame image, A first derivation means for deriving first state information relating to the imaging state in the frame image from the frame image, A superimposing means generates a superimposed image by superimposing state information based on first state information obtained from a frame image captured by the imaging unit at a second time following the first time onto an NR frame image generated from a frame image captured by the imaging unit at a first time; Output means that sequentially outputs superimposed images corresponding to frame images input by the input means to the display unit, An image processing apparatus characterized by having

2. The first state information is information relating to the contrast in the frame image. The image processing apparatus according to feature 1.

3. A second derivation means for deriving a second state information relating to the imaging state in the NR frame image from the NR frame image, A third derivation means for deriving a third state information based on the first state information and the second state information, It further possesses, The second state information is information regarding the contrast in the NR frame image, The third derivation means derives the third state information by adding an offset value caused by the NR processing to the first state information obtained from the frame image captured by the imaging unit at the second time, The superposition means generates a superimposed image by superimposing the state information based on the third state information onto the NR frame image generated from the frame image captured by the imaging unit at a first time. The image processing apparatus according to claim 2.

4. The third derivation means determines the offset value based on the difference between the first state information and the second state information corresponding to the frame image captured by the imaging unit at the first time. The image processing apparatus according to claim 3.

5. The input means inputs the frame image as a pair of phase images, The first state information is information regarding the amount of defocus in the frame image. The image processing apparatus according to feature 1.

6. A second derivation means for deriving a second state information relating to the imaging state in the NR frame image from the NR frame image, A third derivation means for deriving a third state information based on the first state information and the second state information, It further possesses, The second state information is information regarding the amount of defocus in the NR frame image, The third derivation means derives the third state information by adding an offset value caused by the NR processing to the first state information obtained from the frame image captured by the imaging unit at the second time, The superposition means generates a superimposed image by superimposing the state information based on the third state information onto the NR frame image generated from the frame image captured by the imaging unit at a first time. The image processing apparatus according to feature 5.

7. The system further includes a selection means for selecting one or more first state information from a plurality of state information, which includes one or more first state information derived by the first derivation means and one or more second state information derived by the second derivation means, such that a pair of second state information exists within a predetermined distance. The third derivation means derives the third state information based on one or more first state information selected by the selection means. The image processing apparatus according to claim 6.

8. The system further includes a means for specifying a sub-region of the frame image, The first derivation means derives the imaging state of the partial region of the frame image as first state information. The image processing apparatus according to feature 1.

9. An input means for sequentially inputting frame images obtained by imaging by the imaging unit, An NR processing means that applies NR processing to the aforementioned frame image to generate an NR frame image, A first acquisition means for acquiring field-of-view information that defines the shooting range of the frame image, A superimposing means for generating a superimposed image by superimposing field-of-view information for a frame image captured by the imaging unit at a second time following the first time onto an NR frame image generated from a frame image captured by the imaging unit at a first time; Output means that sequentially outputs superimposed images corresponding to frame images input by the input means to the display unit, An image processing apparatus characterized by having

10. The angle of view information includes at least one of the pan information, tilt information, and zoom information of the imaging unit. The image processing apparatus according to feature 9.

11. A method for controlling an image processing device, An input process in which frame images obtained by imaging by the imaging unit are sequentially input, An NR processing step is performed on the frame image to generate an NR frame image, A first derivation step of deriving first state information relating to the imaging state in the frame image from the frame image, A superposition step is to generate a superimposed image by superimposing state information based on first state information obtained from a frame image captured by the imaging unit at a second time following the first time onto an NR frame image generated from a frame image captured by the imaging unit at a first time; An output step sequentially outputs superimposed images corresponding to the frame images input in the input step to the display unit, A control method characterized by including

12. A method for controlling an image processing device, An input process in which frame images obtained by imaging by the imaging unit are sequentially input, An NR processing step is performed on the frame image to generate an NR frame image. A first acquisition step of acquiring field-of-view information that defines the shooting range of the frame image, A superposition step is to generate a superimposed image by superimposing field-of-view information for a frame image captured by the imaging unit at a second time following the first time onto an NR frame image generated from a frame image captured by the imaging unit at a first time; An output step sequentially outputs superimposed images corresponding to the frame images input in the input step to the display unit, A control method characterized by including

13. A program for causing a computer to execute the control method described in claim 11 or 12.

Images