Information processing apparatus and method, imaging apparatus and control method thereof, and storage medium

Abstract

The present invention relates to an information processing apparatus and method, a photographing apparatus and a control method thereof, and a storage medium. The apparatus includes: an acquisition unit configured to acquire a first captured image obtained by performing a first image capture using first image capture parameters and motion information related to a subject in the first captured image; a setting unit configured to set second image capture parameters; an estimation unit configured to estimate motion blur of the subject in a second captured image obtained when performing a second image capture using the second image capture parameters based on the motion information and the second image capture parameters; and a notification unit configured to issue a notification of the motion blur.

Description

[0001] (This application is a divisional application of an application with an application date of December 27, 2019, an application number of 2019113775241, and an invention title of "Information Processing Apparatus and Method, Imaging Apparatus and Control Method Thereof, and Storage Medium".) Technical Field

[0002] Aspects of embodiments generally relate to techniques for issuing a notification of subject blurring that occurs in a captured image. Background Art

[0003] Taking an image of a moving subject using a camera without causing subject blurring in the image may require taking the image at an appropriately set shutter speed.

[0004] Japanese Unexamined Patent Application Publication No. 2008-172667 discusses a technique for enabling an operator of an imaging device to visually confirm a moving area in an image during preparation for image capture. As used herein, preparation for image capture refers to an image capture operation in which the operator makes composition adjustments and sets image capture conditions while viewing the electronic viewfinder or the rear liquid crystal display of the imaging device. Japanese Unexamined Patent Application Publication No. 2008-172667 also discusses a technique for detecting a moving area between sequential images captured during preparation for image capture and displaying the detected moving area in an emphasized manner.

[0005] However, even if the operator visually confirms the image displayed in the electronic viewfinder or the rear liquid crystal display of the imaging device during preparation for image capture, it is difficult for the operator to confirm whether motion blur has occurred using the set shutter speed. Specifically, it is difficult for the operator to visually confirm motion blur in small areas such as the limbs of a runner. In addition, in the case where different shutter speeds are set for main image capture and preparation for image capture, since the possible motion blur may be different between main image capture and preparation for image capture, even when the operator has visually confirmed the image captured during preparation for image capture, it is difficult for the operator to confirm the motion blur that occurs in main image capture. For example, when the operator uses a pre-set shutter speed different from the shutter speed set during preparation for image capture to perform main image capture of a runner whose image was not blurred during preparation for image capture, since the shutter speed used for main image capture is lower than the running speed of the runner, the image of the runner captured in main image capture may be recorded in a blurred manner in some cases. This is not limited to the problem that occurs in the relationship between preparation for image capture and main image capture, and the same problem also occurs between multiple image capture operations in which respective image capture parameters can be independently set. Summary of the Invention

[0006] According to an aspect of an embodiment, a device includes one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the information processing device to function as: an acquisition unit configured to acquire a first captured image obtained by performing a first image capture using first image capture parameters and motion information related to a subject in the first captured image; a setting unit configured to set second image capture parameters; an estimation unit configured to estimate motion blur of the subject in a second captured image obtained by performing a second image capture using the second image capture parameters based on the motion information and the second image capture parameters; and a notification unit configured to issue a notification of the motion blur.

[0007] According to another aspect of an embodiment, a device includes an imaging unit and, in a case where an image capture instruction is issued by an operator when first captured images obtained by performing a first image capture using first image capture parameters in the imaging unit are sequentially output, outputs a second captured image obtained by performing a second image capture using second image capture parameters in response to the image capture instruction. The device includes one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the information processing device to function as: a calculation unit configured to calculate motion information based on a plurality of first captured images each corresponding to a first captured image output from the imaging unit; a setting unit configured to set second image capture parameters; an estimation unit configured to estimate motion blur in the second captured image based on the motion information and the second image capture parameters; and a notification unit configured to issue a notification of the estimated motion blur before receiving the image capture instruction.

[0008] According to still another aspect of an embodiment, a method includes: acquiring a first captured image obtained by performing a first image capture using first image capture parameters and motion information related to a subject in the first captured image; setting second image capture parameters; estimating motion blur of the subject in a second captured image obtained by performing a second image capture using the second image capture parameters based on the motion information and the second image capture parameters; and issuing a notification of the motion blur.

[0009] According to another aspect of the embodiment, a method for a device, the device including an imaging unit, and in a case where an image capture instruction is issued by an operator when first captured images obtained by first image capture using first image capture parameters are sequentially output in the imaging unit, a second captured image obtained by second image capture using second image capture parameters is output in response to the image capture instruction. The method includes: calculating motion information based on a plurality of first captured images respectively corresponding to the first captured images output from the imaging unit; setting the second image capture parameters; estimating motion blur in the second captured image based on the motion information and the second image capture parameters; and issuing a notification of the estimated motion blur before receiving the image capture instruction.

[0010] An information processing device according to another aspect of the embodiment includes: an acquisition component configured to acquire a first captured image obtained by first image capture using first image capture parameters and motion information related to a subject in the first captured image; a setting component configured to set second image capture parameters independently of the first image capture parameters; an estimation component configured to estimate motion blur of the subject in a second captured image obtained when second image capture is performed using the second image capture parameters based on the motion information and the second image capture parameters; and a notification component configured to issue a notification of the motion blur.

[0011] An imaging device according to another aspect of the embodiment includes: an imaging component configured to perform first image capture; and the above information processing device.

[0012] An imaging device according to another aspect of the embodiment includes an imaging component, and in a case where an image capture instruction is issued by an operator when first captured images obtained by first image capture using first image capture parameters are sequentially output in the imaging component, a second captured image obtained by second image capture using second image capture parameters is output in response to the image capture instruction. The imaging device includes: a calculation component configured to calculate motion information based on a plurality of first captured images respectively corresponding to the first captured images output from the imaging component; a setting component configured to set the second image capture parameters independently of the first image capture parameters; an estimation component configured to estimate motion blur in the second captured image based on the motion information and the second image capture parameters; and a notification component configured to issue a notification of the motion blur estimated by the estimation component before receiving the image capture instruction.

[0013] An information processing method according to another aspect of an embodiment includes: obtaining a first captured image obtained by first image capturing using first image capturing parameters and motion information related to a subject in the first captured image; setting second image capturing parameters independently of the first image capturing parameters; estimating motion blur of the subject in a second captured image obtained when performing second image capturing using the second image capturing parameters based on the motion information and the second image capturing parameters; and issuing a notification of the motion blur.

[0014] A control method for a imaging device according to another aspect of an embodiment, the imaging device including an imaging component, and in a case where a first captured image obtained by first image capturing using first image capturing parameters is sequentially output from the imaging component and an operator issues an image capturing instruction, outputting a second captured image obtained by second image capturing using second image capturing parameters in response to the image capturing instruction, the control method including: calculating motion information based on a plurality of first captured images each corresponding to the first captured image output from the imaging component; setting the second image capturing parameters independently of the first image capturing parameters; estimating motion blur in the second captured image based on the motion information and the second image capturing parameters; and issuing a notification of the estimated motion blur before receiving the image capturing instruction.

[0015] A non-transitory computer-readable storage medium storing computer-executable instructions, which when executed by a computer cause the computer to perform a method, the method including: obtaining a first captured image obtained by first image capturing using first image capturing parameters and motion information related to a subject in the first captured image; setting second image capturing parameters independently of the first image capturing parameters; estimating motion blur of the subject in a second captured image obtained when performing second image capturing using the second image capturing parameters based on the motion information and the second image capturing parameters; and issuing a notification of the motion blur.

[0016] Other features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a diagram showing an imaging device in a first exemplary embodiment.

[0018] Figure 2 is a flowchart showing an image capturing processing flow of a digital camera in a first exemplary embodiment.

[0019] Figure 3 FIG. Figure 3 is a diagram showing an example of the structure of a motion blur notification image generation unit in the first exemplary embodiment.

[0020] Figure 4 FIG. is a flowchart showing a processing flow for generating a motion blur notification image in the first exemplary embodiment.

[0021] Figure 5A and Figure 5B FIGS. Figure 5A and Figure 5B are diagrams showing a preparation capture image and a motion vector in the first exemplary embodiment, respectively.

[0022] Figure 6 FIG. is a conceptual diagram showing a motion vector and an estimated motion blur in the first exemplary embodiment.

[0023] Figure 7A , Figure 7B and Figure 7C FIGS. Figure 7A , Figure 7B , and Figure 7C are diagrams showing a motion blur notification method in the first exemplary embodiment.

[0024] Figure 8 FIG. is a diagram showing an example of the structure of a motion blur notification image generation unit in the second exemplary embodiment.

[0025] Figure 9 FIG.

[0024] is a flowchart showing a processing flow for generating a motion blur notification image in the second exemplary embodiment.

[0026] Figure 10 FIG. Figure 8 is a diagram showing an example of the structure of a motion blur notification image generation unit in the third exemplary embodiment.

[0027] Figure 11 FIG. is a flowchart showing a processing flow for generating a motion blur notification image in the third exemplary embodiment.

[0028] Figure 12 FIG.

[0025] is a diagram showing a motion vector in the third exemplary embodiment.

[0029] Figure 13A , Figure 13B and Figure 13C FIGS. Figure 9 , , and

[0026] are diagrams showing motion intensity, edge intensity, and motion edge intensity in the third exemplary embodiment, respectively.

[0030] Figure 14 FIG. Figure 10 is a diagram showing the position of a subject and the cross-section of the subject in a preparation capture image in the fourth exemplary embodiment.

[0031] Figure 15A and Figure 15B FIGS. and

[0027] are diagrams each showing the relationship between position coordinates on a preparation capture image and edge intensity in the fourth exemplary embodiment.

[0032] Figure 16 FIG. is a diagram showing an example of the configuration of a motion blur notification image generation unit in the fourth exemplary embodiment.

[0033] Figure 17 FIG. is a flowchart showing a processing flow for second notification plane generation in the fourth exemplary embodiment.

[0034] Figure 18 FIG. is a diagram showing the relationship between a changed edge intensity threshold and an edge intensity in the fourth exemplary embodiment.

[0035] Figure 19 FIG. is a diagram showing a corrected edge intensity in the fourth exemplary embodiment. DETAILED DESCRIPTION

[0036] Various exemplary embodiments, features, and aspects of the present invention will be described in detail below with reference to the drawings.

[0037] The first exemplary embodiment is configured to estimate motion blur that occurs when an image is captured using image capture parameters set independently of a first image capture, based on motion information related to a subject obtained during the first image capture, and issue a notification of the estimated motion blur.

[0038] Figure 1 FIG. is a block diagram of a digital camera equipped with an imaging unit as an example of each of an information processing device and an imaging device according to the first exemplary embodiment. The information processing device described in this exemplary embodiment can also be applied to any electronic device capable of processing captured images. Examples of such electronic devices may include mobile phones, game machines, tablet terminals, personal computers, and timepiece-type or glasses-type information terminals.

[0039] A control unit 101, such as a central processing unit (CPU), reads a control program from a read-only memory (ROM) 102, loads the control program onto a random access memory (RAM) 103 described below, and executes the control program, where the control program is provided to control various blocks included in the digital camera 100. Thus, the control unit 101 controls the operations of the respective blocks included in the digital camera 100.

[0040] In addition to operation programs for the respective blocks included in the digital camera 100, the ROM 102 (electrically erasable and recordable non-volatile memory) also stores parameters required for the operations of the respective blocks, for example.

[0041] The RAM 103 (re-writable volatile storage) is used, for example, to load programs executed by the control unit 101, and temporarily store data generated by the operations of the respective blocks included in the digital camera 100, for example.

[0042] The optical system 104 (configured with a lens group including a zoom lens and a focusing lens) forms a subject image on the imaging surface of the imaging unit 105 described below.

[0043] The imaging unit 105 (which is an image sensor such as a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor) performs photoelectric conversion on the optical image formed by the optical system 104 on the imaging surface of the imaging unit 105, and outputs the obtained analog image signal to the analog-to-digital (A / D) conversion unit 106.

[0044] The A / D conversion unit 106 converts the input analog image signal into digital image data. The digital image data output from the A / D conversion unit 106 is temporarily stored in the RAM 103.

[0045] The image processing unit 107 performs various processing operations on the image data stored in the RAM 103. Specifically, the image processing unit 107 performs various image processing operations for displaying the digital image data and for displaying or recording the displayed digital image data, such as defect correction processing for pixels caused by the optical system 104 or the image sensor, demosaicing processing, white balance correction processing, color interpolation processing, and gamma processing.

[0046] In addition, the image processing unit 107 includes a motion blur notification image generation unit 300 ( Figure 3 ). In the first exemplary embodiment, the motion blur notification image generation unit 300 generates a motion blur notification image for the image data stored in the RAM 103 by generating and superimposing an image plane that enables easy confirmation of motion blur based on the obtained information related to the motion blur of the subject.

[0047] The recording / output unit 108 records the data including the image data on a recording medium such as a removable memory card or outputs the data to an external device via an external interface. The recording / output unit 108 records the image data processed by the image processing unit 107 as a recorded image via the RAM 103.

[0048] The display unit 109, which includes a display device such as a liquid crystal display (LCD), displays the image stored in the RAM 103 or the image recorded on the recording / output unit 108 on the display device. In addition, the display unit 109 also displays, for example, an operation user interface for receiving instructions from the operator. Additionally, the display unit 109 may include a plurality of display devices such as an electronic viewfinder (EVF) and a rear monitor provided on the operator side (rear). The display unit 109 may be configured to allow simultaneous output to the plurality of display devices or selectively output to one of the plurality of display devices by switching.

[0049] The instruction input unit 110 is an input interface that includes various physical operation members such as a touch panel and a shutter button, and receives an input of an instruction from the operator.

[0050] The angular velocity detection unit 111, such as an angular velocity sensor, detects the angular velocity of the main body of the digital camera 100 in the yaw direction and the pitch direction caused by camera shake or camera movement. The angular velocity detection method used in the angular velocity detection unit 111 is assumed to be an existing method, and a detailed description thereof is omitted.

[0051] In addition, the digital camera 100 performs preparatory image capture (live view image capture) under the control of the control unit 101, in which the preparatory image capture sequentially displays the analog image signals sequentially output from the imaging unit 105 on the display device via the A / D conversion unit 106, the RAM 103, the image processing unit 107, and the display unit 109. During the preparatory image capture, the operator can perform preparations for image capture, such as adjusting the composition in preparation for the main image capture (which involves storing on a recording medium or outputting to an external device), or changing image capture parameters such as the exposure time (Tv value), the aperture value (Av value), and the ISO sensitivity for the main image capture.

[0052] Next, refer to Figure 2 the flowchart to describe in detail the processing performed by the digital camera 100 in the first exemplary embodiment. Figure 2 The steps in the flowchart are performed by the control unit 101 or by an applicable unit of the digital camera 100 in response to an instruction from the control unit 101.

[0053] When the digital camera 100 is powered on by the operator, then in step S201, the control unit 101 starts preparing for image capture by controlling the optical system 104 and the imaging unit 105 in response to the digital camera 100 being powered on. During the preparation for image capture, the digital camera 100 sequentially captures images to sequentially obtain captured images, and displays the captured images on the display device of the display unit 109. The operator can perform, for example, composition adjustment while confirming the sequentially displayed preparation captured images. In addition, the processing operations in steps S202, S203, S204, S205, and S206 described below are performed during the preparation for image capture.

[0054] In step S202, the operator inputs image capture parameters for simulation using the instruction input unit 110. In response to the input from the instruction input unit 110, the control unit 101 sets the image capture parameters for simulation independently of the image capture parameters for preparing image capture. Here, the control unit 101 can be configured to automatically set, for example, image capture parameters that seem suitable for the detected subject model by using, for example, known image analysis or subject analysis. In the first exemplary embodiment, the exposure time can be set as the image capture parameter for simulation.

[0055] In addition, in the first exemplary embodiment, after the pressing of the shutter button (instruction for main image capture) described below is detected, the image capture parameters for simulation set by the control unit 101 are assumed to be the image capture parameters for main image capture. However, the first exemplary embodiment is not limited thereto, but a configuration in which the control unit 101 independently sets the image capture parameters for main image capture based on other separate instructions from the operator or in an automatic manner can be adopted.

[0056] In step S203, the control unit 101 determines whether the setting of the motion blur notification is on or off. For example, whether to set the motion blur notification on or off can be set by the operator using the instruction input unit 110, or whether to set the motion blur notification on or off can be automatically set based on certain types of image capture conditions. The following configuration can be adopted: in this configuration, it is possible to use a physical operation element (e.g., a button or a handle) or an icon on a touch device to set on or off, and the operator can set on or off at any timing during the preparation for image capture. In addition, the following configuration can be adopted: the setting displayed when periodically switching between on and off is available.

[0057] In step S203, if it is determined that the setting of the motion blur notification is enabled (Yes in step S203), the control unit 101 causes the process to proceed to step S204. In step S204, the motion blur notification image generation unit 300 generates a motion blur notification image obtained by superimposing a motion blur notification plane on the ready-to-capture image, where the motion blur notification plane is for notifying the motion blur (or no motion blur) that occurs in the subject image when an image is captured using the image capture parameters for main image capture. Then, in step S205, the control unit 101 displays the motion blur notification image on the display device of the display unit 109.

[0058] In step S203, if it is determined that the setting of the motion blur notification is disabled (No in step S203), then in step S205, the control unit 101 displays the ready-to-capture image without the motion blur notification plane superimposed thereon on the display unit 109.

[0059] In step S206, the control unit 101 determines whether the shutter button of the instruction input unit 110 has been pressed by the operation of the operator. Here, in the case where the input of the pressing of the shutter button is configured to receive a two-stage input method (including, for example, a half-press for issuing an instruction for an image capture preparation operation and a full-press for issuing an instruction for a main image capture operation), the control unit 101 determines whether the full-press has been performed. In the case where only a simple single-stage input is received, the control unit 101 determines whether the single-stage input has been performed.

[0060] If it is determined that the shutter button has not been pressed (No in step S206), the control unit 101 causes the process to return to step S202, and then repeats the processing operations in steps S202 to S206. This enables the operator to easily confirm the motion blur that occurs in the subject image when the main image is captured using the currently set image capture parameters even during the preparation for image capture. If the motion blur is confirmed and the confirmed motion blur is not the motion blur preferred by the operator (in the case where the occurrence of motion blur is not desired), the operator resets the shutter speed (exposure time) for main image capture without pressing the shutter button.

[0061] In this way, notifying the motion blur that occurs in the subject image during the preparation for image capture enables the operator to repeat setting the exposure time for main image capture until the desired motion blur is obtained when confirming the motion blur notification image displayed on the display unit 109. Then, the operator can receive the shutter opportunity in a state where the exposure time setting corresponding to the appropriate motion blur has been reached.

[0062] In step S206, if it is determined that the shutter button has been pressed (Yes in step S206), the control unit 101 determines that an image capture instruction for main image capture has been received, and then causes the process to proceed to step S207. In step S207, the control unit 101 performs main image capture by controlling, for example, the optical system 104 and the imaging unit 105 based on the image capture parameters set until the preparation for image capture. In step S208, the control unit 101 outputs the main captured image obtained by the main image capture to the display unit 109 and the recording / output unit 108, thereby displaying the main captured image on the display device of the display unit 109, and recording the main captured image on the recording / output unit 108 or outputting the main captured image from the recording / output unit 108 to an external device.

[0063] Next, Figure 3 A structural example of the motion blur notification image generation unit 300 included in the image processing unit 107, which is a characterization part of the first exemplary embodiment, will be described.

[0064] Figure 3 FIG. is a diagram showing a structural example of the motion blur notification image generation unit 300. The motion blur notification image generation unit 300 includes a motion vector calculation unit 301 and an estimated motion blur calculation unit 302. The motion vector calculation unit 301 calculates the motion vector of the subject by comparing images, and the estimated motion blur calculation unit 302 estimates the motion blur in the subject image that may occur during the main image capture based on the calculated motion vector. In addition, the motion blur notification image generation unit 300 further includes a notification plane generation unit 303 and an image superimposing unit 304. The notification plane generation unit 303 generates data for notifying the motion blur based on the estimated motion blur in the subject image, and the image superimposing unit 304 performs a superimposing process for superimposing the motion blur notification plane on the captured image in the first exemplary embodiment. Details of the operations of the above units will be described below.

[0065] In addition, Figure 3 One or more of the functional blocks shown in can be implemented by hardware such as an application specific integrated circuit (ASIC) or a programmable logic array (PLA), or can be implemented by a programmable processor such as a CPU or a microprocessing unit (MPU) executing software. In addition, one or more functional blocks can be implemented by a combination of software and hardware. Accordingly, in the following description, even when different functional blocks are described as the main body of the operation, the same hardware can be implemented as the main body.

[0066] Next, Figure 4 will be described in detail with reference to the flowchart of Figure 2The process performed in step S204 shown in FIG. The motion blur notification image generation unit 300 performs this process to generate a motion blur notification image. Figure 4 The steps in the flowchart of FIG. are performed by the control unit 101 or by an applicable unit of the digital camera 100 including the motion blur notification image generation unit 300 in response to an instruction from the control unit 101.

[0067] In step S401, the control unit 101 inputs the preparation capture images sequentially captured by the imaging unit 105 and the image capture parameters for main image capture to the motion blur notification image generation unit 300.

[0068] In step S402, the motion vector calculation unit 301 calculates the motion vector between the images as the motion information of the subject by performing a comparison process between the images of the preparation image capture. The motion vector, as a vector, represents the amount of movement of the subject between the images of the preparation image capture. In the first exemplary embodiment, the motion vector calculation unit 301 calculates the motion vector of the subject in the plane between two-dimensional images. However, the first exemplary embodiment is not limited thereto, and the motion vector calculation unit 301 may be configured to, for example, acquire depth information (such as subject distance or defocus amount) in the depth direction of the subject in each preparation capture image, so as to calculate the motion vector of the subject in three-dimensional space.

[0069] The motion vector calculation unit 301 sets a plurality of preparation capture images as the reference frame and the reference frame, and performs a correlation calculation between the reference block in the reference frame and each block of the object region in the reference frame. The motion vector calculation unit 301 calculates the motion vector based on the positional relationship between the block with the highest correlation as the result of the correlation calculation and the reference block. The method of calculating the correlation value is not specifically limited, and may be, for example, a method based on the sum of absolute differences, the sum of squared differences, and the normalized cross-correlation value, and the method of calculating the motion vector itself may be other methods such as the gradient method.

[0070] Figure 5A Shows the preparation capture images obtained by capturing an image of a dog 501 running to the left, and Figure 5B is a schematic diagram of the motion vector calculated at that time. In Figure 5B the example shown, for the running dog 501, a motion vector with a magnitude above a predetermined value is detected as a motion vector to the left, and for another stationary dog 502 and the background fence, the value of the motion vector is zero or less than the predetermined value.

[0071] In addition, instead of calculating the motion vectors of all pixels, the motion vector calculation unit 301 may calculate the motion vectors for each predetermined pixel.

[0072] In Figure 4 step S403 shown, the estimated motion blur calculation unit 302 obtains the exposure time for main image shooting set in step S202 shown in Figure 2 and the time interval (frame rate) between images in preparatory image shooting as image shooting conditions.

[0073] In step S404, the estimated motion blur calculation unit 302 estimates the motion blur of the subject in the main image shooting based on the exposure time for main image shooting and the time interval between images in preparatory image shooting obtained in step S403, according to the motion vectors for each pixel calculated in step S402.

[0074] Refer to Figure 6 for a detailed description of the method for estimating the motion blur of the subject in the main image shooting. Figure 6 is a diagram showing the relationship between the motion vectors in preparatory image shooting and the estimated motion blur estimated as the motion blur in the main image shooting. In Figure 6 , an example is shown in which, as each image shooting condition, the time interval between images in preparatory image shooting is 1 / 60 second, and the exposure times for main image shooting are 1 / 120 second and 1 / 30 second.

[0075] The estimated motion blur calculation unit 302 estimates the motion blur in the main image shooting based on conversion formulas such as the following formulas (1) and (2) according to the motion vectors for each pixel.

[0076] CONV_GAIN = EXP_TIME / INT_TIME (1)

[0077] CONV_BLUR = VEC_LEN × CONV_GAIN (2)

[0078] Here, in formula (1), CONV_GAIN represents the estimated gain used to convert the magnitude of the motion vector in preparatory image shooting into the magnitude of the motion vector in the main image shooting, EXP_TIME represents the exposure time for main image shooting, and INT_TIME represents the time interval between images in preparatory image shooting. In addition, in formula (2), CONV_BLUR represents the motion blur of the subject in the main image shooting, and VEC_LEN represents the magnitude of the motion vector in preparatory image shooting.

[0079] In Equation (1), the estimated gain is calculated by dividing the exposure time used for main image capture by the time interval between the images for which pre-image capture is prepared. Then, in Equation (2), the motion blur of the subject in the main image capture is calculated by multiplying the magnitude of the motion vector by the estimated gain.

[0080] Specifically, as Figure 6 shown, when the length of the motion vector in the pre-image capture is 10 pixels, the estimated motion blur in the main image capture using an exposure time of 1 / 120 second becomes 5 pixels because the estimated gain is 1 / 2 times. Further, the estimated motion blur in the main image capture using an exposure time of 1 / 30 second becomes 20 pixels because the estimated gain is 2 times.

[0081] In step S405, the notification plane generation unit 303 generates an image plane for notifying of the motion blur based on the motion blur for each pixel calculated in step S404. For example, the notification plane generation unit 303 generates an image plane for notifying of the motion blur in a manner that emphasizes and displays the distinguishable pixels corresponding to the motion blur having a predetermined blur amount or more.

[0082] Further, during the period until the shutter press is detected in step S206 as shown in Figure 2 and the main image capture is then performed, the operator is allowed to change image capture parameters such as the shutter speed to obtain a captured image in the main image capture having a motion blur. If the image capture parameters are changed, the estimated motion blur calculation unit 302 re-estimates the motion blur. The notification plane generation unit 303 regenerates the image plane based on the re-estimated motion blur.

[0083] In step S406, the image superimposing unit 304 generates a motion blur notification image by superimposing the notification plane generated in step S405, for example, on the pre-capture image in the RAM 103.

[0084] Here, refer to Figure 7A 、 Figure 7B and Figure 7C for a detailed description of the method for generating the notification plane for notifying of the motion blur of the subject generated by the notification plane generation unit 303, and the method for generating the motion blur notification image obtained by the superimposition of the notification plane. Figures 7A to 7C Three examples of the motion blur notification image are shown. In these examples, the motion blur notification image is displayed on the display unit 109 during the pre-image capture so that the operator can easily confirm the motion blur of the subject.

[0085] Figure 7AAn example of using a display with an icon to issue a motion blur notification is shown. In step S405, the control unit 101 calculates the ratio of the number of pixels with estimated motion blur exceeding a predetermined value in the estimated motion blur for each pixel to the entire image screen. When the calculated ratio is greater than or equal to a predetermined ratio, the notification plane generation unit 303 generates a motion blur notification icon 901 as shown in Figure 7A as a motion blur notification plane, and the image superimposing unit 304 generates a motion blur notification image as shown in Figure 7A by superimposing the motion blur notification icon 901 on the image to be captured.

[0086] Figure 7B An example of using a display with a motion blur box to issue a motion blur notification is shown. Here, a method for generating a motion blur notification image with a motion blur box displayed is described. In step S405, the control unit 101 calculates the ratio of the number of pixels with estimated motion blur exceeding a predetermined value in the estimated motion blur for each pixel within a segmented area to the entire segmented area. For a segmented area where the calculated ratio is greater than or equal to a predetermined ratio, the notification plane generation unit 303 generates a motion blur notification box 902 as shown in Figure 7B as a motion blur notification plane, and the image superimposing unit 304 generates a motion blur notification image as shown in Figure 7B by superimposing the motion blur notification box 902 on the image to be captured. Whether to issue a motion blur notification for each segmented area can be determined based on a statistical value such as the average value or the median value of the estimated motion blur for each pixel in the segmented area.

[0087] Figure 7C An example of issuing a motion blur notification by emphasizing the edges of a subject with motion blur is shown. Here, a method for generating a motion blur notification image using an emphasized display of motion blur edges is described. In step S405, the notification plane generation unit 303 detects the edge intensity of the image to be captured. Detection of the edge intensity can be performed using a known technique such as a Sobel filter, and a detailed description thereof is omitted. Then, the notification plane generation unit 303 extracts pixels with an edge intensity exceeding a predetermined value and an estimated motion blur exceeding a predetermined value (for example, clips the result of the above filter with a predetermined value). For the extracted pixels, the notification plane generation unit 303 generates a motion blur notification plane that emphasizes the edges of the subject with motion blur (such as Figure 7C the motion blur notification edge 903 shown in), and then generates a motion blur notification image as shown in Figure 7C by superimposing the motion blur notification plane on the image to be captured. In Figure 7CIn the example shown, the motion blur notification edge 903 is bolded. Other examples of methods for emphasizing display include methods of changing hue, saturation, or brightness, such as extracting pixels with an edge intensity above a predetermined value and an estimated motion blur above a predetermined value, and turning the extracted pixels red.

[0088] As described above, the first exemplary embodiment is configured to estimate motion blur in a second image capture from motion blur in a first image capture and issue a notification of the estimated motion blur, so that an operator can easily confirm motion blur of a subject that occurs in the second image capture when an image obtained by the first image capture is displayed. The operator can confirm whether it is possible to capture an image with motion blur by using the current image capture parameters used in the second image capture during the first image capture, and can set the image capture parameters for the second image capture.

[0089] Although, in the first exemplary embodiment, an example of issuing a motion blur notification when the estimated motion blur is above a predetermined value has been described, a motion blur notification may also be issued when the estimated motion blur is below a predetermined value. Thereby, in the case where motion blur means long exposure image capture that is expressed as a dynamic feeling, the operator becomes able to easily confirm any insufficient motion blur during the period of preparing for image capture.

[0090] In addition, in the first exemplary embodiment, the motion blur notification is started in response to starting to prepare for image capture after the digital camera 100 is powered on. However, the first exemplary embodiment is not limited thereto, but the following structure may be adopted: In this structure, for example, when the shutter button is half-pressed during the period of preparing for image capture, the motion blur notification is issued during the period when the shutter button is half-pressed or for a predetermined period. In addition, an operation member that can be operated during the period of preparing for image capture to freely switch between turning on and off the motion blur notification may be provided.

[0091] In addition, the first exemplary embodiment is configured to estimate motion blur in the main image capture based on the motion vector obtained in the preparation image capture. However, the first exemplary embodiment is not limited thereto, and for example, it may be configured to predict the motion vector in the main image capture based on the motion vector obtained in the preparation image capture, and then estimate the motion blur in the main image capture based on the predicted motion vector. The method of predicting the motion vector in the main image capture includes, for example, predicting based on the temporal change of the motion vector in the preparation image capture and the time elapsed before the main image capture. This method is particularly effective in cases where the time elapsed from the preparation image capture to the main image capture is predetermined (such as in the case of self-timed image capture or automatic image capture where the imaging device identifies the shutter opportunity and automatically performs image capture).

[0092] In addition, the first exemplary embodiment is configured to estimate the motion blur in the second image capture (main image capture) by converting the motion blur in the first image capture (preparation image capture) into the motion blur in the second image capture by considering the exposure time as a different image capture parameter. However, the image capture parameter to be considered is not limited to the exposure time, and since a brighter image makes motion blur more noticeable, the following structure may also be adopted: in this structure, based on the exposure value (Ev value) of each image, compared with the case of a low-exposure image, in the case of a high-exposure image, the threshold for determining whether the current motion blur is a motion blur to be notified is set lower.

[0093] In addition, although, in the first exemplary embodiment, three examples of the emphasized display (i.e., the motion blur notification icon display, the motion blur notification box display, and the motion blur notification edge display) have been described as the method for motion blur notification, the method for motion blur notification is not limited thereto. For example, the area where motion blur occurs including the flat area may be displayed in an emphasized manner. Specifically, the notification plane generation unit 303 performs an emphasized display that makes the pixels for which the estimated motion blur is above a predetermined value turn red for each pixel. In this way, not only the edge area is emphasized, but also the area other than the edge area is emphasized so that the entire subject is displayed in an emphasized manner, making it easy for the operator to confirm the motion blur.

[0094] In addition, although in the first exemplary embodiment, an example of a method of notifying motion blur by displaying on the display unit 109 has been described as a method of motion blur notification, the method for motion blur notification is not limited thereto. For example, the notification of motion blur can be issued by using sound, light, or vibration. Specifically, when the ratio of the number of pixels with an estimated motion blur exceeding a predetermined value in the estimated motion blur for each pixel to the entire image screen is greater than or equal to a predetermined ratio, a motion blur notification sound, a motion blur notification light, or a motion blur notification vibration is generated. In this modified example, the structures of the notification plane generation unit 303 and the image superimposing unit 304, and the processing flows of step S405 and step S406 become unnecessary. Instead, a speaker can be installed in the digital camera 100, and in parallel with the display of the ready-to-capture image on the display unit 109 in step S205, the control unit 101 can generate a notification sound through the speaker, can cause the notification light to be emitted, or can generate a notification vibration.

[0095] In addition, although in the first exemplary embodiment, an example of notifying the motion blur of a subject occurring in the main captured image when the ready-to-capture image is displayed has been described, the relationship between the two captured images or the image capture parameters is not limited thereto. In other words, the following exemplary embodiment can be adopted: In this exemplary embodiment, under the condition that a plurality of image sensors are used for image capture, one image sensor outputs a first captured image by performing a first image capture using a first image capture parameter, and another image sensor outputs a second captured image by performing a second image capture using a second image capture parameter. In this case, the motion blur of the subject in the second captured image is also estimated based on the motion information related to the first captured image, and the motion blur is also notified. The plurality of image sensors can be respectively installed on a plurality of imaging devices.

[0096] In addition, although in the first exemplary embodiment, an example of notifying the estimated motion blur has been described, the first exemplary embodiment is not limited thereto, and the estimated motion blur can also be used for image capture control such as for determining an image capture instruction. For example, control such as not receiving an image capture start instruction when the amount of motion blur has not reached can be considered.

[0097] Considering the fact that motion blur also prevents an image from being obtained due to the movement of an imaging device caused by, for example, camera shake, the second exemplary embodiment described below is configured to estimate the motion blur of a subject caused by the movement of the imaging device in a second image capture based on motion information related to the imaging device in a first image capture and then issue a notification of the estimated motion blur. In addition, elements designated by the same reference numerals as those in the respective drawings of the first exemplary embodiment are assumed to perform the same actions and processing operations as those in the first exemplary embodiment, and thus are omitted in this description.

[0098] Figure 8 FIG. is a diagram showing a structural example of a motion blur notification image generation unit 800 included in an image processing unit 107. The motion blur notification image generation unit 800 includes an image motion information conversion unit 801, an estimated motion blur calculation unit 302, a notification plane generation unit 303, and an image superimposing unit 304. Actions and processing operations other than those of the image motion information conversion unit 801 are the same as those in the first exemplary embodiment, and thus are omitted in this description.

[0099] Next, with reference to Figure 9 the flowchart, the processing performed by the motion blur notification image generation unit 800 to generate a motion blur notification image will be described in detail. Figure 9 The steps in the flowchart are performed by a control unit 101 or by an applicable unit of a digital camera 100 including the motion blur notification image generation unit 800 in response to an instruction from the control unit 101.

[0100] In step S901, the image motion information conversion unit 801 converts the angular velocity information detected by the angular velocity detection unit 111 into motion information in the image. The approximate conversion formulas for converting the angular velocity into motion information are shown in the following formulas (3) and (4).

[0101]

[0102]

[0103] MOV_yaw represents the amount of movement in the yaw direction, and MOV_pitch represents the amount of movement in the pitch direction. In addition, f represents the focal length, ω_yaw represents the angular velocity in the yaw direction, ω_pitch represents the angular velocity in the pitch direction, fps represents the frame rate for preparing image capture, and pp represents the pixel pitch of the imaging unit 105. The conversion formulas shown in formulas (3) and (4) function to calculate the amount of movement on the imaging plane based on the angle and focal length generated by the movement during the time interval between images in preparing image capture, and to calculate the amount of movement on the image (the number of pixels corresponding to the movement) by dividing the calculated amount of movement on the imaging plane by the pixel pitch in each of the yaw and pitch directions. In addition, the amount of movement on the image calculated here is not an amount of movement that varies for each pixel, but an amount of movement that is the same for all pixels.

[0104] The image motion information conversion unit 801 regards the amount of movement in the yaw direction as the amount of movement in the horizontal direction and regards the amount of movement in the pitch direction as the amount of movement in the vertical direction, and thus outputs the amount of movement as a motion vector that is the same for all pixels to the estimated motion blur calculation unit 302.

[0105] In step S404, similar to the first exemplary embodiment, the estimated motion blur calculation unit 302 estimates the motion blur in the main image capture by using the motion vector calculated in step S901. And, in step S405, it notifies the plane generation unit 303 to generate a motion blur notification plane. Then, in step S406, the image superposition unit 304 generates a motion blur notification image. As described above, the second exemplary embodiment is configured to estimate the motion blur of the subject caused by the movement of the imaging device in the second image capture based on the motion information related to the imaging device in the first image capture, and to issue a notification of the motion blur based on the estimated motion blur. This enables the operator to confirm the motion blur caused by the movement of the imaging device such as camera shake when acquiring the first captured image. After confirmation, the operator is allowed to change the image capture parameters in a manner that enables acquisition of a captured image with motion blur (including a captured image without motion blur).

[0106] In addition, although in the second exemplary embodiment, an example has been described in which the amount of movement on the imaging surface is calculated based on the angle and focal length resulting from the movement during the time interval between the images captured as preparation images, and the amount of movement on the image is calculated by dividing the calculated amount of movement on the imaging surface by the pixel pitch, the method for calculating the amount of movement on the image is not limited thereto. The method for calculating the amount of movement on the image may include calculating the amount of movement on the imaging surface based on the angle and focal length resulting from the movement during the exposure period of the image in the preparation image capture, and dividing the calculated amount of movement on the imaging surface by the pixel pitch. Specifically, the angular velocity detection unit 111 detects the angular velocity during the exposure period of the image in the preparation image capture, and the motion blur notification image generation unit 800 generates a motion blur notification image based on the detected angular velocity. In addition, in this case, the estimated motion blur calculation unit 302 estimates the motion vector for each pixel as the motion blur in the main image capture based on the exposure time in the main image capture and the exposure time of the image in the preparation image capture.

[0107] The third exemplary embodiment described below relates to solving the following problem: when the resolution (number of pixels) of the motion blur notification plane is low (small), it is difficult for the operator to visually confirm the motion blur of small areas such as limbs. Therefore, the third exemplary embodiment is configured to obtain the motion blur estimated based on the motion vector for each region of the subject in the first image capture, divide each region for which the motion vector is obtained into a plurality of block regions, obtain the motion edge intensity for each block region, and issue a notification of the motion blur based on the estimated motion blur and the obtained motion edge intensity. In addition, elements designated by the same reference numerals as those in the first and second exemplary embodiments are assumed to perform the same actions and processing operations as those in the first and second exemplary embodiments, and thus are omitted from this description.

[0108] Figure 10 FIG. is a diagram showing a structural example of the motion blur notification image generation unit 1000 included in the image processing unit 107. The motion blur notification image generation unit 1000 includes a motion vector calculation unit 1001, a motion intensity calculation unit 1002, an estimated motion blur calculation unit 302, a notification plane generation unit 1003, and an image superposition unit 304. In addition, the actions and processing operations of the estimated motion blur calculation unit 302 and the image superposition unit 304 are the same as those in the first exemplary embodiment, and thus are omitted from this description.

[0109] Next, with reference to Figure 11 the flowchart, the processing performed by the motion blur notification image generation unit 1000 to generate a motion blur notification image will be described in detail. Figure 11The steps in the flowchart are performed by the control unit 101 or by applicable units of the digital camera 100 including the motion blur notification image generation unit 1000 in response to an instruction from the control unit 101.

[0110] In step S1101, the motion vector calculation unit 1001 calculates the motion vector between images, which is the motion information of the subject, by performing a comparison process between the images for preparing image capture for each unit area. Although the minimum unit of the unit area can be set to one pixel, in the third exemplary embodiment, in order to reduce the processing load applied to the motion vector calculation, during normal operation, a plurality of pixels are set as the unit area. Through the operation of the operator via the instruction input unit 110, the unit area can be specified for an operation to determine the detection accuracy or processing speed of the motion blur. For example, compared with the case where the detection accuracy of the motion blur is set to low accuracy, in the case where the detection accuracy of the motion blur is set to high accuracy, the unit area becomes smaller.

[0111] Furthermore, in step S1101, the motion intensity calculation unit 1002 calculates the motion intensity by performing a comparison process between the captured images for each block area, where each block area is obtained by further dividing each unit area used to calculate the motion vector into a plurality of block regions. The minimum unit of the block region is also set to one pixel, and in the third exemplary embodiment, it is assumed that the motion intensity calculation unit 1002 calculates the motion intensity for each pixel. In the first exemplary embodiment, an example has been described in which the motion vector calculation unit 301 calculates the motion vector for each pixel as the motion information. On the other hand, in the third exemplary embodiment, the following example is described: In this example, the motion vector calculation unit 1001 calculates the motion vector for each unit area as the motion information, and the motion intensity calculation unit 1002 calculates the motion intensity for each pixel.

[0112] First, Figure 12 the motion vector calculation unit 1001 is described in detail.

[0113] Figure 12 is a diagram showing Figure 5A the motion vector calculated by the motion vector calculation unit 1001 in the image for preparing image capture shown in Figure 12 and the unit area for calculating the motion vector. In Figure 12 the example shown in

[0114] The motion vector calculation unit 1001 calculates one motion vector for each unit area 1300 (for which a motion vector is calculated), and assigns the same motion vector as the calculated one motion vector to all the pixels included in the unit area. According to such motion vector calculation processing being performed, for example, in a unit area such as the unit area 1300 that calculates a motion vector indicating leftward movement, both the face of the running dog and the stationary ground have a motion vector indicating leftward movement.

[0115] In this way, compared with calculating motion vectors for all pixels as in the first exemplary embodiment, calculating one motion vector for each unit area and assigning the same motion vector as the calculated one motion vector to all the pixels included in the unit area reduces the processing amount even more.

[0116] For the specific motion vector calculation method, the same actions and processing operations as those in the first exemplary embodiment are performed, and thus are omitted in this description.

[0117] Next, refer to Figure 13A 、 Figure 13B and 13C to describe the motion intensity calculation unit 1002 in detail.

[0118] Figure 13A is a diagram showing the motion intensity calculated by the motion intensity calculation unit 1002 in the image to be captured as shown in Figure 5A . The motion intensity calculation unit 1002 calculates the difference in pixel values at the same coordinates between the images to be captured as the motion intensity. In the example shown in Figure 13A , for the running dog 501 in the image to be captured as shown in Figure 5A , since the difference in pixel values between the images is large, a large motion intensity is obtained. On the other hand, for the stationary standing dog 502 and the background fence other than the running dog, since the difference in pixel values between the images is small, a small motion intensity is obtained. In this way, compared with the processing for calculating motion vectors for each pixel, the processing for calculating the motion intensity for each pixel based on the difference in pixel values between the images enables the detection of the motion area with a smaller processing amount.

[0119] Furthermore, although in the third exemplary embodiment, an example in which the motion intensity calculation unit 1002 calculates the motion intensity for each pixel is described, the unit for calculating the motion vector is not limited to this, and it can be a region smaller than the unit area for calculating the motion vector.

[0120] In step S1102, the notification plane generation unit 1003 generates an image plane for issuing a motion blur notification based on the motion blur for each pixel calculated in step S404. For example, the notification plane generation unit 1003 generates an image plane for issuing a motion blur notification in a distinguishable manner that emphasizes and displays pixels corresponding to a motion blur with a predetermined blur amount or more.

[0121] Here, refer to Figure 13B and Figure 13C A method for generating a notification plane (generated by the notification plane generation unit 1003) for issuing a motion blur of a subject is described in detail. Figure 13B is a diagram showing Figure 5A the edge intensity in the preparation capture image shown in, and Figure 13C is a diagram showing the motion edge intensity therein. In the third exemplary embodiment, an example of issuing a motion blur notification by emphasizing and displaying the edge of a subject that has undergone a motion blur as shown in Figure 7C is described.

[0122] The notification plane generation unit 1003 detects the edge intensity of the preparation capture image. Figure 13B The edge intensity detected by the notification plane generation unit 1003 is shown in. The specific detection method for the edge intensity is the same as the detection method in the first exemplary embodiment, and thus is omitted in this description. Then, the notification plane generation unit 1003 calculates the motion edge intensity by multiplying the motion intensity and the edge intensity at the same coordinates for each pixel. Figure 13C The motion edge intensity calculated by the notification plane generation unit 1003 is shown in. For pixels with a greater edge intensity and a greater motion intensity, the motion edge intensity obtains a greater value. Then, the notification plane generation unit 1003 extracts pixels with a motion edge intensity of a predetermined value or more and an estimated motion blur of a predetermined value or more. For the extracted pixels, the notification plane generation unit 1003 generates a motion blur notification plane that emphasizes and displays the edge of a subject that is undergoing a motion blur (such as the motion blur notification edge 903 shown in Figure 7C ), and then generates a motion blur notification image such as the motion blur notification image shown in Figure 7C by superimposing the motion blur notification plane on the preparation capture image. In this way, using the motion edge intensity to generate the motion blur notification plane enables only the edge of the running dog to be emphasized and displayed even if the same leftward motion vector is assigned to the running dog and the stationary ground in the unit area 1300.

[0123] In addition, although in the third exemplary embodiment, an example of calculating the motion edge intensity by multiplying the motion intensity and the edge intensity at the same coordinates for each pixel has been described, the method for calculating the motion edge intensity is not limited thereto. For example, the calculation method may include calculating the motion edge intensity by adding the motion intensity and the edge intensity at the same coordinates for each pixel.

[0124] In addition, when the unit area in which the motion vector calculation unit 1001 calculates the motion vector is small enough, a motion blur notification may be issued based on the motion information related only to the motion vector as in the first exemplary embodiment. This is because, when the unit area for calculating the motion vector is small enough, the operator can visually confirm the motion blur in a small area such as the limbs.

[0125] The third exemplary embodiment described above is configured to calculate the motion edge intensity by multiplying the detected edge intensity and the motion intensity at the same coordinates for each pixel. However, in such an image capturing scene where motion blur occurs, the edge intensity on the prepared capture image may decrease in some cases, and thus a motion blur notification may not be appropriately issued.

[0126] Reference Figure 14 、 Figure 15A And Figure 15B Describe specific examples of the above-mentioned problems. Figure 14 Illustrate Figure 5A And Figure 5B The image of the dog 501 in the prepared capture image shown in Figure 15A And Figure 15B Illustrate Figure 14 The transition of the edge intensity in the cross-section 1401 shown in

[0127] In Figure 15A And Figure 15B The horizontal axis represents the coordinates of the position in the horizontal direction on the cross-section 1401, and the vertical axis represents the detected edge intensity.

[0128] When no blur occurs on the prepared capture image, such an edge intensity as shown in Figure 15A Is detected on the cross-section 1401.

[0129] However, when blur occurs on the prepared capture image, as shown in Figure 15B The detected edge intensity on the cross-section 1401 decreases.

[0130] In view of the problems mentioned above, the fourth exemplary embodiment described below is configured to set an edge intensity threshold for identifying the above-mentioned edge region based on the motion vector of the subject, correct the edge intensity based on the identification result, and issue a notification of motion blur.

[0131] Reference Figure 16 、 Figure 17 and Figure 18 describe the fourth exemplary embodiment.

[0132] In addition, elements designated by the same reference numerals as those in the third exemplary embodiment are assumed to perform the same operations and processing operations as those in the third exemplary embodiment, and thus these elements are omitted from the description herein.

[0133] In addition, since there are no differences in elements other than the motion blur notification image generation unit in the third exemplary embodiment and the second motion blur notification image generation unit in the fourth exemplary embodiment, other elements are omitted from the description herein.

[0134] First, Figure 16 is a diagram showing a structural example of the second motion blur notification image generation unit 1500 included in the image processing unit 107.

[0135] The second motion blur notification image generation unit 1500 includes a motion vector calculation unit 1001, a motion intensity calculation unit 1002, an estimated motion blur calculation unit 302, an image superposition unit 304, an edge intensity calculation unit 1501, an edge region discrimination unit 1502, a motion edge intensity calculation unit 1503, and a specific pixel extraction unit 1504.

[0136] The operations and processing operations of the motion vector calculation unit 1001, the motion intensity calculation unit 1002, the estimated motion blur calculation unit 302, and the image superposition unit 304 are the same as those in the first exemplary embodiment and the third exemplary embodiment, and thus are omitted from the description herein.

[0137] The edge intensity calculation unit 1501 acquires a ready-to-capture image and calculates the edge intensity of the acquired ready-to-capture image. The specific calculation method of the edge intensity is the same as that in the first exemplary embodiment, and thus is omitted from the description herein.

[0138] The edge region discrimination unit 1502 sets an edge intensity threshold for identifying the above-mentioned edge region, uses the edge intensity threshold to determine whether the edge intensity is greater than or equal to the edge intensity threshold, and if the edge intensity is greater than or equal to the edge intensity threshold, corrects the edge intensity. The detailed operation of the edge region discrimination unit 1502 is described below.

[0139] The motion edge intensity calculation unit 1503 calculates the motion edge intensity by multiplying or adding the edge intensity and the motion intensity together. The specific calculation method of the motion edge intensity is the same as that in the third exemplary embodiment, and thus is omitted from the description herein.

[0140] The specific pixel extraction unit 1504 extracts pixels whose motion edge intensity is above a predetermined value and whose estimated motion blur is above a predetermined value. The specific extraction method is the same as that in the third exemplary embodiment, and thus is omitted from the description herein.

[0141] Next, with reference to Figure 17 the flowchart of, the process performed by the edge region discrimination unit 1502 for discriminating and correcting the edge region by using an edge intensity threshold with respect to the detected edge intensity is described in detail.

[0142] Figure 17 The steps in the flowchart of are performed by the control unit 101 or by an applicable unit of the digital camera 100 including the edge region discrimination unit 1502 in response to an instruction from the control unit 101.

[0143] In the fourth exemplary embodiment, an example of calculating the edge intensity threshold based on the maximum value in the magnitudes of the motion vectors is described.

[0144] In step S1601, the edge region discrimination unit 1502 acquires the edge intensity calculated by the edge intensity calculation unit 1501, and thus obtains a transition of the edge intensity such as that shown in Figure 14 the cross section 1401 shown in with respect to Figure 15B the edge intensity shown in.

[0145] In step S1602, the edge region discrimination unit 1502 acquires the motion vectors for each unit region calculated by the motion vector calculation unit 1001 based on Figure 14 the cross section 1401 shown in.

[0146] In step S1603, the edge region discrimination unit 1502 calculates the edge intensity threshold Th according to the acquired motion vectors for each unit region.

[0147] First, the edge region discrimination unit 1502 calculates the maximum value in the magnitudes of the motion vectors according to the acquired motion vectors for each unit region.

[0148] In addition, although in the fourth exemplary embodiment, the maximum value in the magnitudes of the motion vectors acquired for each unit region is calculated, the value to be calculated is not limited thereto, and may be a statistical value of the motion vectors or a statistical value of the magnitudes of the motion vectors.

[0149] For example, in a case where a notification of appropriate motion blur is issued for a subject that occupies a large proportion of the entire image to be captured, the mode value of the motion vectors for each unit area can be used.

[0150] Next, the edge region discrimination unit 1502 calculates an edge intensity threshold Th for discriminating an edge region based on the maximum value of the calculated motion vectors.

[0151] The edge intensity threshold Th is calculated by the following formula (5) according to a linear function relationship.

[0152] Th = A · v + Th0 (5)

[0153] Th: Edge intensity threshold

[0154] A: Negative constant

[0155] v: Maximum value of the motion vector

[0156] Th0: Reference edge intensity threshold

[0157] Here, the negative constant A in formula (5) is a coefficient representing a proportional constant determined by the degree of relationship between the motion vector and the decrease in edge intensity. In the fourth exemplary embodiment, the negative constant A is assumed to be previously stored in the ROM 102.

[0158] In addition, although an example of calculating the edge intensity threshold has been described using a linear function in the fourth exemplary embodiment, the calculation method for the edge intensity threshold is not limited to this. The calculation method can approximate the degree of relationship between motion blur and the decrease in edge intensity. For example, the calculation method may include calculating the edge intensity threshold according to a quadratic function.

[0159] In step S1604, the edge region discrimination unit 1502 discriminates an edge region based on the edge intensity threshold calculated in step S1603. The edge region discrimination unit 1502 performs such discrimination by determining whether the edge intensity obtained in step S1601 is greater than or equal to the edge intensity threshold calculated in step S1603. If it is determined that the edge intensity is greater than or equal to the edge intensity threshold (yes in step S1604), the edge region discrimination unit 1502 proceeds to step S1605 and then to step S1606.

[0160] If it is determined that the edge intensity is less than the edge intensity threshold (no in step S1604), the edge region discrimination unit 1502 causes the process to proceed to step S1606.

[0161] Reference Figure 18Describe the discrimination of the edge region in step S1604.

[0162] Figure 18 shows the application of the discrimination of the edge region to the Figure 15B edge strength shown in. As in Figure 15A and Figure 15B the same, in Figure 18 the vertical axis represents the detected edge strength, and the horizontal axis represents the Figure 14 position coordinates on the preparation capture image in the cross-section 1401 shown in. Even when the edge strength has decreased due to the occurrence of motion blur such as that shown in Figure 15B the edge region is discriminated.

[0163] Next, in step S1605, the edge region discrimination unit 1502 performs a correction process on the edge strength that has been determined to be greater than or equal to the edge strength threshold.

[0164] Correct the edge strength according to the following formula (6).

[0165]

[0166] E’: Corrected edge strength

[0167] E: Edge strength not yet corrected

[0168] Th0: Reference edge strength threshold

[0169] Th: Threshold calculated in step S1603

[0170] Figure 19 shows the result obtained by performing the correction expressed by formula (6) for the case shown in Figure 18 . According to the ratio between the edge strength thresholds represented in formula (6), only the edge strength exceeding the edge strength threshold is corrected to become a strength equivalent to the edge strength obtained in the case where no motion blur occurs.

[0171] In addition, although in the fourth exemplary embodiment, the correction is performed according to the ratio between the edge strength thresholds, the correction formula is not limited thereto.

[0172] For example, in the case where almost all the edge strengths not yet corrected are approximately equal to each other, the difference between the edge strength thresholds may be added to the edge strength.

[0173] Next, in step S1606, the edge region discrimination unit 1502 determines whether the discrimination of the edge regions for all pixels and the correction of the edge intensities have been completed. If it is determined that such a processing operation has not been completed (in step S1606, "No"), the edge region discrimination unit 1502 returns the processing to step S1604, and if it is determined that such a processing operation has been completed (in step S1606, "Yes"), the edge region discrimination unit 1502 ends the processing.

[0174] In this way, based on the motion blur occurring in the image to be captured, the edge intensity threshold for discriminating the edge regions is set, and the edge intensity is corrected based on the discrimination result so that even for a subject whose edge intensity has decreased due to motion blur, a notification of motion blur can be issued.

[0175] In addition, although in the fourth exemplary embodiment, the edge intensity threshold is determined based on the magnitude of the motion vector, for example, in the case of a translational image capture where the subject moves only in one direction, the edge intensity threshold can be determined based at least on the direction component of the motion vector of the subject.

[0176] In addition, in the fourth exemplary embodiment, an edge intensity threshold is determined based on the motion vector for each unit region. However, in the case where there are a plurality of subjects performing different motions in the image to be captured, the motion vectors calculated in the respective unit regions can be used to determine the edge intensity thresholds for the respective unit regions.

[0177] Aspects of the embodiments can also be obtained by performing the following processing. Specifically, a recording medium recording a program code of software having a process for realizing the functions of the above-described exemplary embodiments described herein is provided to a system or device. Then, the computer (or CPU or microprocessing unit (MPU)) of the system or device is caused to read out and execute the program code stored in the storage medium.

[0178] In this case, the program code read out from the storage medium itself implements the novel functions of the aspects of the embodiments, and the storage medium and the program having the program code stored in the storage medium can constitute the aspects of the embodiments.

[0179] In addition, storage media for providing program code include, for example, floppy disks, hard disks, optical disks, and magneto-optical disks. In addition, storage media for providing program code also include, for example, compact disc read-only memory (CD-ROM), recordable compact disc (CD-R), rewritable compact disc (CD-RW), digital versatile disc read-only memory (DVD-ROM), digital versatile disc random access memory (DVD-RAM), rewritable digital versatile disc (DVD-RW), recordable digital versatile disc (DVD-R), magnetic tapes, non-volatile memory cards, and read-only memory (ROM).

[0180] In addition, executable program code read by a computer implements the functions of the exemplary embodiments described above. Additionally, for example, an operating system (OS) running on the computer may also perform part or all of the actual processing based on instructions from the program code to implement the functions of the exemplary embodiments described above.

[0181] In addition, the following processing may be included. First, the processing writes the program code read from the storage medium into a memory, where the memory is included in a function expansion board inserted into the computer or a function expansion unit connected to the computer. After that, for example, a CPU included in the function expansion board or the function expansion unit performs part or all of the actual processing based on instructions from the program code.

[0182] Other Embodiments

[0183] Embodiments of the present invention may also be implemented by the following method, that is, software (program) for performing the functions of the above-described embodiments is provided to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0184] Although the present invention has been described with reference to exemplary embodiments, it should be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications as well as equivalent structures and functions.

Claims

1. An information processing device, comprising: An acquisition component, configured to acquire a first captured image obtained by first image capturing using first image capturing parameters and motion information related to a subject in the first captured image; A setting component, configured to set second image capturing parameters independently of the first image capturing parameters; And An estimation component, configured to estimate motion blur of a subject in a second captured image obtained by second image capturing using the second image capturing parameters, based on the motion information and a difference between the first image capturing parameters and the second image capturing parameters.

2. The information processing device according to claim 1, further comprising a notification component for issuing a motion blur notification.

3. The information processing device according to claim 1, wherein, The acquisition component calculates and acquires the motion information by comparing images among a plurality of first captured images respectively corresponding to the first captured image.

4. The information processing device according to claim 2, further comprising: An indication component, configured to issue an image capturing indication when successively outputting the first captured images obtained by first image capturing using the first image capturing parameters in an image sensor, wherein the image sensor outputs a second captured image obtained by second image capturing using the second image capturing parameters in response to the image capturing indication. Wherein, the notification component is configured to issue a motion blur notification before receiving the image capturing indication.

5. The information processing device according to claim 1, wherein, The estimation component estimates the motion blur in the second captured image based on the motion information, a time interval between images in multiple first image capturings, and an exposure time set as the second image capturing parameters.

6. The information processing device according to claim 2, wherein, The notification component issues a motion blur notification by displaying information corresponding to the motion blur on a display unit.

7. The information processing device according to claim 2, wherein, The notification component issues a motion blur notification by superimposing information corresponding to the motion blur on the first captured image and displaying the first captured image with the superimposed information on the display unit.

8. The information processing device according to claim 2, wherein, The acquisition component acquires motion information corresponding to a plurality of regions of the first captured image. Wherein, the estimation component estimates motion blur in a plurality of regions of the second captured image according to the motion information corresponding to the plurality of regions of the first captured image, and Wherein, the notification component issues a motion blur notification for each of the plurality of corresponding regions.

9. The information processing device according to claim 8, wherein, The notification component issues a motion blur notification by displaying a frame for each region in the first captured image where the motion blur in the second captured image has been estimated to have a predetermined blur amount or more.

10. The information processing device according to claim 8, wherein, The notification component issues a motion blur notification by visibly displaying an edge region for each of the following regions in the first captured image, wherein in each of the regions, the motion blur in the second captured image has been estimated to have a predetermined blur amount or more.

11. The information processing device according to claim 2, wherein, The first image capturing is performed before the second image capturing, and Wherein, while the acquisition component successively acquires the first captured images, the notification component issues a motion blur notification.

12. The information processing device according to claim 2, wherein, In a case where the setting component changes the second image capturing parameters, the estimation component performs estimation of motion blur again. Among them, the notification component issues a notification of the motion blur re-estimated by the estimation component.

13. The information processing device according to claim 1, wherein, The setting component sets the second image shooting parameter based on the first captured image.

14. The information processing device according to claim 2, wherein, The acquisition component calculates the motion vector in each region of a plurality of regions of the first captured image corresponding to the first captured images obtained by performing the first image shooting multiple times, divides each region of the plurality of regions into a plurality of block regions, and calculates the difference between the plurality of first captured images for each block region of the block regions, and Among them, the estimation component estimates the motion blur of the subject for each block region of the block regions based on the motion vector and the difference between the plurality of first captured images.

15. The information processing apparatus according to claim 14, wherein, The notification component issues a notification of motion blur by displaying an edge region in a distinguishable manner for each of the following regions in the first captured image, where the motion blur in the second captured image has been estimated to have a predetermined blur amount or more in each of the regions.

16. The information processing apparatus according to claim 14, wherein, When the size of each region of the plurality of regions in which the motion vector is calculated is equal to or greater than a threshold value, the estimation component estimates the motion blur of the subject for each block region of the block regions based on the motion vector and the difference between the plurality of first captured images, and When the size of each region of the plurality of regions in which the motion vector is calculated is less than the threshold value, the estimation component estimates the motion blur of the subject for each of the plurality of regions based on the motion vector.

17. The information processing apparatus according to claim 15, wherein, The notification component switches the threshold value used to determine the edge region according to the motion information.

18. The information processing apparatus according to claim 17, wherein, The notification component changes the threshold value used to determine the edge region based on the statistical value calculated according to the motion vector.

19. The information processing apparatus according to claim 18, wherein, The statistical value is calculated at least based on the maximum value of the magnitude of the motion vector.

20. The information processing apparatus according to claim 18, wherein, The statistical value is calculated at least based on the mode value of the magnitude of the motion vector.

21. The information processing apparatus according to claim 18, wherein, The notification component changes the threshold value used to determine the edge region based on at least one direction component of the motion vector.

22. The information processing apparatus according to claim 18, wherein, The notification component changes the threshold value used to determine the edge region in each region of the plurality of regions of the first captured image.

23. An imaging apparatus, comprising: An imaging component for performing first image shooting; And The information processing apparatus according to any one of claims 1 to 22.

24. An imaging apparatus, comprising an imaging unit, and when first captured images obtained by first image capturing using first image capturing parameters are sequentially output in the imaging unit and an image capturing instruction is issued by an operator, second captured images obtained by second image capturing using second image capturing parameters are output in response to the image capturing instruction, the imaging apparatus comprising: A calculation component for calculating motion information based on a plurality of first captured images respectively corresponding to the first captured images output from the imaging component; A setting component for setting the second image shooting parameter independently of the first image shooting parameter; And An estimation component for estimating the motion blur in the second captured image based on the motion information and the difference between the first image shooting parameter and the second image shooting parameter.

25. An information processing method, comprising: Obtain a first captured image obtained by performing first image shooting using a first image shooting parameter and motion information related to a subject in the first captured image; Set a second image shooting parameter independently of the first image shooting parameter; And Estimate the motion blur of a subject in a second captured image obtained when capturing a second image using the second image capture parameter, based on the motion information and the difference between the first image capture parameter and the second image capture parameter.

26. A control method for a imaging device, the imaging device including an imaging component, and in a case where when successively outputting first captured images obtained by first image capturing using first image capturing parameters in the imaging component and an operator issues an image capturing instruction, outputting second captured images obtained by second image capturing using second image capturing parameters in response to the image capturing instruction, the control method includes: Calculate motion information based on a plurality of first captured images each corresponding to the first captured image output from the imaging component; Set the second image capture parameter independently of the first image capture parameter; And Estimate the motion blur in the second captured image based on the motion information and the difference between the first image capture parameter and the second image capture parameter.

27. A non-transitory computer-readable storage medium storing computer-executable instructions, which when executed by a computer, cause the computer to perform a method, the method including: Obtain a first captured image obtained by first image capture using a first image capture parameter and motion information related to a subject in the first captured image; Set a second image capture parameter independently of the first image capture parameter; And Estimate the motion blur of a subject in a second captured image obtained when capturing a second image using the second image capture parameter, based on the motion information and the difference between the first image capture parameter and the second image capture parameter.

Images