Information processing apparatus, imaging apparatus, information processing method, and method for controlling imaging apparatus

Abstract

To provide an image processing apparatus that allows a user, during preparatory photographing, to easily check a motion blur occurring in main photographing by restricting a notification area.SOLUTION: An image processing apparatus has: acquisition means that acquires a plurality of first photographed images obtained by performing first photographing according to a first photographing parameter and information on a subject's motion in the plurality of first images; estimation means that, when second photographing is performed according to a second photographing parameter set independently of the first photographing parameter, estimates a motion blur of the subject in a second photographed image obtained through the second photographing from the motion information and the second photographing parameter; notification means that performs notification processing corresponding to information on the motion blur; designation means that designates a subject for which the notification means performs notification of the motion blur; and determination means that determines an area for which notification of the motion blur is performed based on the subject designated by the designation means. The notification means performs the notification processing for the area determined by the determination means.SELECTED DRAWING: Figure 14

Description

Technical Field

[0001] The present invention relates to a technique for notifying subject blur in an image to be captured.

Background Art

[0002] Some imaging devices such as digital still cameras have a shooting mode that prioritizes the shutter speed (hereinafter referred to as "shutter speed priority mode"). In the shutter speed priority mode, the photographer sets the desired shutter speed, and the imaging device automatically sets the exposure setting values other than the shutter speed, such as the aperture value and ISO sensitivity. By using such a shutter speed priority mode, the photographer can take a picture at a preferred shutter speed. For example, by setting a high shutter speed before shooting and taking a picture in the shutter speed priority mode, an image with less motion blur can be taken.

[0003] Patent Document 1 discloses a technique for detecting a motion area between time-series images captured during preparatory shooting and highlighting the motion area. Here, preparatory shooting is shooting in which the composition is adjusted and shooting conditions are set while looking at the electronic viewfinder or the rear liquid crystal of the imaging device before actual shooting. Actual shooting is shooting that is executed by the imaging device based on the composition and shooting conditions determined during preparatory shooting, triggered by an action such as the photographer pressing the shutter button. According to Patent Document 1, the photographer can visually confirm the motion area during preparatory shooting.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0005] According to Patent Document 1, it is possible to detect motion regions between time-series images captured during preparation shooting and to encourage the user to take actions to reduce blur by emphasizing those motion regions. However, Patent Document 1 does not consider shooting with appropriate frame rates and shutter speeds according to the speed and amount of movement of the subject in order to extract motion regions between time-series images.

[0006] This invention has been made in view of the above-mentioned problems, and aims to make it easy to confirm motion blur occurring during the main shoot while the preparation shoot is in progress, by limiting the notification area when notifying motion blur based on motion blur estimated to be equivalent to that of the main shoot from the motion blur of the preparation shoot. [Means for solving the problem]

[0007] To solve the above problems, the information processing device of the present invention includes: acquisition means for acquiring a plurality of first captured images obtained by performing a first shooting with first shooting parameters and motion information of a subject in the plurality of first images; estimation means for estimating motion blur of a subject in a second captured image obtained by a second shooting using second shooting parameters set independently of the first shooting parameters, based on the motion information and the second shooting parameters; notification means for performing notification processing corresponding to the motion blur information; designation means for designating a subject for which the motion blur will be notified by the notification means; and determination means for determining a region for which the motion blur will be notified based on the subject designated by the designation means, wherein the notification means performs the notification processing on the region determined by the determination means.

[0008] Furthermore, the information processing method of the present invention comprises: an acquisition step of acquiring a plurality of first captured images obtained by performing a first shooting with first shooting parameters and motion information of a subject in the plurality of first images; an estimation step of estimating motion blur of a subject in a second captured image obtained in a second shooting from the motion information and the second shooting parameters when a second shooting is performed with second shooting parameters set independently of the first shooting parameters; a notification step of performing notification processing corresponding to the motion blur information; a designation step of designating a subject for which motion blur notification is to be performed in the notification step; and a determination step of determining a region for which motion blur notification is to be performed based on the subject designated in the designation step, wherein the notification step performs the notification processing on the region determined in the determination step. [Effects of the Invention]

[0009] According to the present invention, by limiting the notification area when notifying motion blur based on motion blur estimated to be equivalent to that of the main shoot during the preparation shoot, it is possible to easily check the motion blur that occurs during the main shoot while the preparation shoot is in progress. [Brief explanation of the drawing]

[0010] [Figure 1] Block diagram of an imaging device according to an embodiment of the present invention. [Figure 2] Cross-sectional view of the housing of an imaging device according to an embodiment of the present invention. [Figure 3] Principle diagram of a line-of-sight detection method provided in an imaging device according to an embodiment of the present invention [Figure 4] Schematic diagram of the eyeball image projected onto the eyeball image sensor 119 and output intensity diagram of the CCD in the eyeball image sensor 119. [Figure 5] Outline flow routine of gaze detection in an imaging device according to an embodiment of the present invention [Figure 6] A diagram showing the image capture processing flow according to the first embodiment of the present invention. [Figure 7] This figure shows an example configuration of the motion blur notification image generation unit 300. [Figure 8] Figure showing the processing flow of the motion blur notification image generation unit 300 [Figure 9] Figure showing the preparatory captured image and the motion vector [Figure 10] Figure showing the processing flow of the motion vector calculation unit 301 [Figure 11] Figure showing the method for calculating the motion vector [Figure 12] Figure showing the motion vector and the estimated motion blur [Figure 13] Figure showing the motion blur notification method according to the first embodiment of the present invention [Figure 14] ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​Embodiments of the present invention will be described below with reference to the attached drawings. Note that the following embodiments do not limit the invention to the claims. While multiple features are described in the embodiments, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] As mentioned above, Patent Document 1 does not consider the subject's speed or movement when extracting motion regions between time-series images. For example, when photographing a runner in a foot race in shutter-priority mode to minimize motion blur, the photographer predicts the runner's speed during the preparation shot and sets a shutter speed that is expected to minimize motion blur. However, even if the photographer visually checks the image displayed in the electronic viewfinder or rear LCD during the preparation shot, it is extremely difficult to confirm whether or not motion blur will occur at the set shutter speed. Specifically, it is difficult to visually confirm motion blur in small areas such as the runner's hands and feet during the preparation shot. Furthermore, if the shutter speeds for the main shot and the preparation shot are different, the motion blur that occurs in the main shot and the preparation shot will also be different, making it difficult to confirm the motion blur of the main shot even if the image during the preparation shot is visually checked.Therefore, the following describes an embodiment of the present invention that makes it easy to confirm the motion blur that occurs in the main shot during the preparation shot by limiting the notification area when notifying motion blur based on motion blur estimated to be equivalent to that of the main shot.

[0013] Figure 1 is a block diagram of an imaging device (digital camera 100) shown as an example of an information processing device according to an embodiment of the present invention. The embodiments described below are imaging devices, and examples of applying the present invention to a digital camera as an example of an imaging device will be described. The information processing device referred to in the present invention is also applicable to any electronic device capable of processing captured images. These electronic devices may include, for example, mobile phones, game consoles, tablet terminals, personal computers, and information terminals such as watches, glasses, and head-mounted displays.

[0014] The imaging optical unit 101 is composed of multiple lens groups, including a zoom lens, a focus lens, and an image stabilization lens, and is equipped with an aperture. During shooting, the imaging optical unit 101 adjusts the focus using its focus adjustment circuit 208, adjusts the exposure using its aperture control circuit 207, corrects for blur, and forms an optical image on the imaging surface of the imaging unit 102.

[0015] The imaging unit 102 has a photoelectric conversion function that converts an optical image into an electrical signal (analog image signal) and is composed of a CCD or CMOS sensor, etc. The optical image formed on the imaging surface of the imaging unit 102 is photoelectrically converted and the resulting analog signal is output to the A / D conversion unit 103.

[0016] The A / D conversion unit 103 converts the input analog image signal into digital image data. The digital image data is temporarily stored in the DRAM 107, which will be described later.

[0017] The image processing unit 104 consists of various image processing units and buffer memory, and performs various image processing on the image data stored in the DRAM 107. For example, it appropriately performs processes such as chromatic aberration correction, development processing, noise reduction processing, geometric transformation, and resizing. In addition, the image processing unit 104 also includes an imaging correction unit that appropriately performs pixel correction, black level correction, shading correction, and scratch correction on the image data converted by the A / D converter 103. Furthermore, the image processing unit 104 includes a motion blur notification image generation unit 300, which will be described later. The motion blur notification image generation unit 300 generates an image that notifies motion blur (motion blur notification image) by generating and superimposing an image plane on the image stored in the DRAM 107, based on the acquired motion blur information of the subject, which allows for easy confirmation of motion blur.

[0018] Furthermore, the image processing unit 104 has a subject area detection means (subject detection means) and detects the main subject area from the captured image. In addition, semantic region segmentation can be performed on the subject. For example, it is possible to segment the image into specific parts (specific regions) such as the torso, arms, and legs of a human, as well as various other subjects such as animals and vehicles. Note that existing methods such as machine learning are used for the subject area detection means and the semantic region segmentation means, and their explanation is omitted.

[0019] The data transfer unit 105 is composed of multiple DMACs (Direct Memory Access Controllers) that perform data transfer.

[0020] The DRAM (memory) 107 is a memory for storing data, and has sufficient storage capacity to store a predetermined number of still images, predetermined duration of video, audio, and other data, as well as constants for the operation of the control unit 114 and programs. It is also used for program deployment performed by the control unit 114, etc., as described later.

[0021] The memory control unit 106 writes data to and reads data from the DRAM 107 in response to instructions from the control unit 114 or the data transfer unit 105.

[0022] The non-volatile memory control unit 108 writes and reads data from the ROM (non-volatile memory) 109 in response to instructions from the control unit 114.

[0023] ROM109 is an electrically erasable and recordable memory, such as an EEPROM. ROM109 stores constants for the operation of the control unit 114, programs, and the like.

[0024] The recording medium 111 is a recording medium such as an SD card, and is controlled by the recording medium control unit 110 to record image data and read recorded data.

[0025] The display unit 113 includes a display device such as an LCD and displays images stored in the DRAM 107 or recorded on the recording medium 111 under the control of the display control unit 112. The display unit 113 also displays an operation user interface for receiving instructions from the user. Furthermore, the display unit 113 may have multiple display devices, such as an electronic viewfinder (EVF) or a rear monitor provided on the photographer's side (rear). The display unit 113 is controlled by the display control unit 112 and can process and display image data input from the A / D conversion unit 103 in real time before still image shooting or during video recording.

[0026] The operation unit 115 is an input interface that includes various physical operating elements such as switches, buttons, and touch panels that are operated by the user, and accepts instructions from the user.

[0027] The control unit 114 is, for example, a CPU, which reads control programs for each functional block of the digital camera 100 from the ROM 109, loads them into the DRAM 107, and executes them. The control unit 114 also performs calculations necessary for various control processes. The control unit 114 controls the image processing unit 104, data transfer unit 105, memory control unit 106, non-volatile memory control unit 108, recording media control unit 110, display control unit 112, operation unit 115, and imaging unit 102 via the bus 116. The microcomputer executes the programs recorded in the ROM 109 to realize each process in this embodiment. Furthermore, the control unit 114 controls the lens and aperture of the imaging optical unit 101 and acquires information such as focal length.

[0028] Bus 116 is a system bus mainly used to transmit control signals from the control unit 114 and other components to each block, while bus 117 is a data bus mainly used to transfer image data.

[0029] Furthermore, the digital camera 100, under the control of the control unit 114, performs preparatory shooting (live view shooting), which sequentially displays the analog image signals output from the imaging unit 102 on a display device via the A / D conversion unit 103, DRAM 107, image processing unit 104, and display unit 113. Here, preparatory shooting refers to shooting before the main shooting, in which the photographer adjusts the composition and sets the shooting conditions while looking at the electronic viewfinder or rear LCD of the imaging device. The main shooting refers to shooting for recording images, triggered by an action such as the photographer pressing the shutter button, which causes the imaging device to execute based on the composition and shooting conditions set in the preparatory shooting. Patent Document 1 allows the photographer to visually confirm the motion area during preparatory shooting. During preparatory shooting, the photographer can adjust the composition for the main shooting, which is intended for recording to a recording medium or output to an external device, and change the shooting parameters for the main shooting, such as exposure time (Tv value), aperture value (Av value), and ISO sensitivity, to prepare for shooting.

[0030] Figure 2 is a cross-sectional view of the housing of a digital camera 100 according to an embodiment of the present invention, and is an explanatory diagram illustrating the general structure. In Figures 1 and 2, corresponding parts are indicated by the same numbers.

[0031] In Figure 2, 1A represents the photographic lens in an interchangeable lens camera. In this embodiment, for convenience, the inside of the photographic lens 1A is represented by two lenses, 211 and 212, but in reality, it may be composed of many more lenses. 1B represents the housing of the camera body, and the configuration of the units contained inside is as follows. The image sensor 102 is positioned at the intended imaging plane of the photographic lens 1A of the digital camera 100. Also, an eyepiece lens 12 for observing the subject image displayed on the display unit 113 is positioned.

[0032] 13a to 13b are light sources for illuminating the photographer's eyeball 14 to detect the direction of line of sight from the relationship between the reflected image from the corneal reflection of the light source and the pupil. They consist of infrared light-emitting diodes and are arranged around the eyepiece lens 12. The illuminated eyeball image and the images from the corneal reflection of light sources 13a and 13b pass through the eyepiece lens 12, are reflected by the light divider 15, and are imaged onto the eyeball image sensor 119, which has a two-dimensional arrangement of photoelectric elements such as a CCD, by the light-receiving lens 16. The light-receiving lens 16 positions the pupil of the photographer's eyeball 14 and the eyeball image sensor 119 in a conjugate imaging relationship. The control unit 114 detects the direction of line of sight from the positional relationship between the eyeball imaged on the eyeball image sensor 119 and the images from the corneal reflection of light sources 13a and 13b using a predetermined algorithm described later.

[0033] The photographic lens 1A includes an aperture 201, an aperture control circuit 207, a lens drive motor 202, a lens drive component 203 consisting of drive gears, and a photocoupler 204. The photocoupler 204 detects the rotation of a pulse plate 205 that is linked to the lens drive component 203 and transmits this information to the focus adjustment circuit 208. The focus adjustment circuit 208 then drives the lens drive motor 202 by a predetermined amount based on this information and information on the amount of lens drive from the camera, moving the photographic lens 1a to the focus position. The mount contact 206 serves as a known interface between the camera and the lens.

[0034] Furthermore, the aforementioned control unit 115 is equipped with operating components such as a touch-panel LCD, shooting assist buttons, and a button-type directional pad, and is used for control via the shooting assist operation described later. The touch-panel LCD allows the user to specify the image area and move the AF (autofocus) frame by touching the LCD. Similar settings can also be made using the button-type directional pad.

[0035] Figure 3 is a diagram illustrating the principle of the gaze detection method and corresponds to a summary diagram of the optical system for gaze detection. In Figure 2, light sources 13a and 13b are light sources such as light-emitting diodes that emit infrared light that is insensitive to the observer. Each light source is positioned approximately symmetrically with respect to the optical axis of the light-receiving lens 16 and illuminates the observer's eyeball 14. A portion of the illumination light reflected by the cornea of ​​the eyeball 14 is focused by the light-receiving lens 16 onto the image sensor 119 for the eyeball, and the direction of the gaze can be detected from the positional relationship.

[0036] Figure 4(A) is a schematic diagram of the eyeball image projected onto the eyeball image sensor 119, and Figure 4(B) is a diagram of the output intensity of the CCD in the eyeball image sensor 119. Figure 5 shows a schematic flowchart of gaze detection.

[0037] The gaze detection means (gaze area detection means) will be explained below using Figures 3 to 5.

[0038] <Explanation of eye-tracking operation> In Figure 5, when the gaze detection routine starts, in step S501, the light sources 13a and 13b irradiate the observer's eyeball 14 with infrared light. The image of the observer's eyeball, illuminated by the infrared light, is formed on the eyeball image sensor 119 through the light-receiving lens 16, and the eyeball image sensor 119 performs photoelectric conversion, making the eyeball image processable as an electrical signal.

[0039] In step S502, the eyeball image signal obtained from the eyeball image sensor 119 as described above is sent to the control unit 114.

[0040] In step S503, the eyeball image signal information sent to the control unit in S502 is acquired by a gaze detection circuit 118 (not shown), and the coordinates of the points corresponding to the corneal reflection images Pd, Pe and the pupil center c of the light sources 13a and 13b shown in Figure 3 are determined. The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the observer's eyeball 14. At this time, the corneal reflection images Pd, Pe, formed by a portion of the infrared light reflected from the surface of the cornea 142 are focused by the light-receiving lens 16 and imaged onto the eyeball image sensor 119 (points Pd', Pe' shown). Similarly, the light beams from the ends a and b of the pupil 141 are also imaged onto the eyeball image sensor 119. In Figure 4, Figure 4(a) shows an example of a reflection image obtained from the eyeball image sensor 119, and Figure 4(b) shows an example of brightness information obtained from the eyeball image sensor 119 in region α of the above image example. As shown in the figure, the horizontal direction is defined as the X-axis and the vertical direction as the Y-axis. In this case, the coordinates in the X-axis direction (horizontal direction) of the images Pd' and Pe' formed by the corneal reflections of light sources 13a and 13b are defined as Xd and Xe. Also, the coordinates in the X-axis direction of the images a' and b' formed by the light beams from the ends a and b of the pupil 14b are defined as Xa and Xb. In the example of luminance information in (b), extremely high levels of luminance are obtained at positions Xd and Xe, which correspond to the images Pd' and Pe' formed by the corneal reflections of light sources 13a and 13b. In the region between coordinates Xa and Xb, which corresponds to the area of ​​the pupil 141, extremely low levels of luminance are obtained, except for the positions Xd and Xe. In contrast, in the region with an X-coordinate value lower than Xa and a X-coordinate value higher than Xb, which corresponds to the area of ​​the iris 143 outside the pupil 141, intermediate values ​​between the two types of luminance levels are obtained. From the luminance level variation information with respect to the above X coordinate position, the X coordinates Xd and Xe of the images Pd' and Pe' formed by the corneal reflection images of light sources 13a and 13b, and the X coordinates Xa and Xb of the images a' and b' at the pupillary tip can be obtained. Furthermore, when the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light-receiving lens 16 is small, the coordinate Xc of the point corresponding to the pupillary center c (let's call it c') that is imaged on the eyeball image sensor 119 can be expressed as Xc ≈ (Xa + Xb) / 2. From the above, the X coordinate of c' corresponding to the pupillary center that is imaged on the eyeball image sensor 119, and the coordinates of the corneal reflection images Pd' and Pe' of light sources 13a and 13b could be estimated.

[0041] Furthermore, in step S504, the imaging magnification β of the eyeball image is calculated. β is a magnification determined by the position of the eyeball 14 relative to the light-receiving lens 16, and can essentially be determined as a function of the interval (Xd-Xe) between the corneal reflection images Pd' and Pe'.

[0042] Furthermore, in step S505, the X-coordinate of the midpoint of the corneal reflection images Pd and Pe and the X-coordinate of the curvature center O of the cornea 142 are approximately coincidental. Therefore, if we denote the standard distance from the curvature center O of the cornea 142 to the center c of the pupil 141 as Oc, then the rotation angle θX of the optical axis of the eyeball 14 in the ZX plane is: β*Oc*Sinθx≈{(Xd+Xe) / 2}-Xc This can be determined from the given relationship. Furthermore, while Figures 3 and 4 show an example of calculating the rotation angle θX when the observer's eyeball rotates in a plane perpendicular to the Y-axis, the method for calculating the rotation angle θy when the observer's eyeball rotates in a plane perpendicular to the X-axis is similar.

[0043] In the previous step, the rotation angles θx and θy of the optical axis of the observer's eyeball 14 are calculated. In step S506, θx and θy are used to determine the position of the observer's line of sight on the display element 10 (the position of the point of focus; hereinafter referred to as the point of focus). Assuming that the point of focus is the coordinate (Hx, Hy) corresponding to the center c of the pupil 141 on the display unit 113, Hx = m × (Ax × θx + Bx) Hy = m × (Ay × θy + By) This can be calculated as follows. Here, the coefficient m is a constant determined by the configuration of the camera's viewfinder optical system, and is a conversion coefficient that converts the rotation angles θx and θy into position coordinates corresponding to the center c of the pupil 141 on the display unit 113. It is assumed that the coefficient m is predetermined and stored in memory 107. In addition, Ax, Bx, Ay, and By are gaze correction coefficients that correct for individual differences in the observer's gaze, and are obtained by performing the calibration work described later, and are assumed to be stored in memory 107 before the gaze detection routine starts.

[0044] As described above, after the gaze detection circuit 118 calculates the coordinates (Hx, Hy) of the center c of the pupil 141 on the display unit 113, the coordinates are stored in the memory 107 in step S507, and the gaze detection routine ends. In addition, the control unit 114 measures how long the gaze remained in a certain area and stores this as the gaze time in the memory 107.

[0045] The above describes a method for acquiring the coordinates of the point of gaze on a display element using corneal reflection images from light sources 13a and 13b, but the present invention is not limited to this method. Any method for acquiring the rotation angle of the eyeball from an captured eyeball image is applicable to this invention.

[0046] Furthermore, while the method for obtaining the gaze point coordinates was explained above, the gaze area may also be defined as the region included within a certain distance from the obtained gaze point coordinates.

[0047] Based on the gaze point coordinates obtained above, gaze markers indicating the gaze detection result may be displayed on the display device of the display unit 113 based on the control of the display control unit 112. For example, it is possible to superimpose gaze markers so that the position of the gaze point is updated for image data that is successively displayed on the display unit 113. That is, the gaze detection routine described above is repeated, and the display control unit 112 displays on the display unit 113 an image with gaze markers superimposed on the image data that is successively input from the A / D conversion unit 103.

[0048] The color, shape, and size of the gaze marker can be anything you like. For example, it could be a circle centered on the gaze point coordinates calculated by the gaze detection routine described above, but it is not limited to this.

[0049] Furthermore, when superimposing gaze markers onto the image displayed on the display unit 113, the superposition position does not have to be the gaze point coordinates described above. For example, one method is to use the average value of multiple gaze point coordinates calculated by the gaze detection circuit 118 using eyeball image signals obtained from the eyeball image sensor 119 within a certain time period. This can suppress errors in gaze detection and variations in the superposition position of the gaze markers due to flickering of the observer's gaze. The time required for one of the gaze detection routines described above can be made shorter than the time (update time) for updating the image displayed on the display unit 113. If the calculation of gaze point coordinates is performed multiple times while the image displayed on the display unit 113 is being updated, the gaze markers based on the average value of the gaze point coordinates described above can be sequentially updated and superimposed on the image displayed on the display unit 113. The method for determining the position in which the gaze markers are superimposed is not limited to those described above, and it is preferable to use a method that suppresses the effects of errors in gaze detection and flickering of the observer's gaze. For example, one possible method is to exclude, from the calculation of determining the display position of the gaze marker, gaze point coordinates that are located more than a certain distance away from other gaze point coordinates among multiple gaze point coordinates acquired within a certain time period.

[0050] The above is a description of the gaze detection means (gaze area detection means) according to the present invention. However, the gaze detection means is not limited to this, and other existing methods may be used.

[0051] [First Embodiment] Next, the processing of the digital camera 100 in the first embodiment of the present invention will be described in detail with reference to the flowchart in Figure 6. The following processing is achieved by the control unit 114 controlling each part of the digital camera 100 according to the program stored in the ROM 109.

[0052] In step S601, the user turns on the power to the digital camera 100. In response to the power being turned on to the digital camera 100, the control unit 114 controls the imaging optical unit 101 and the imaging unit 102 to start preparatory shooting. During this preparatory shooting period, the digital camera 100 captures and acquires images sequentially, and the acquired images are displayed on the display device of the display unit 113. The user can adjust the composition while viewing the images being taken during preparatory shooting, which are displayed sequentially. The processes in steps S602, S603, S604, S605, S606 and S607, which will be described later, are performed during the preparatory shooting period. Here, images taken during preparatory shooting are defined as preparatory shooting images.

[0053] In step S602, the user inputs simulation shooting parameters using the operation unit 115. The control unit 114 sets the simulation shooting parameters independently of the shooting parameters for preparation shooting, according to the input from the operation unit 115. Here, the control unit 114 may use known image analysis, subject analysis, etc., to automatically set shooting parameters that are deemed suitable for the detected subject model, for example. In this embodiment, exposure time can be set as a simulation shooting parameter.

[0054] In this embodiment, the simulation shooting parameters set by the control unit 114 are used as the shooting parameters for the actual shooting after the pressing of the shutter button (instruction for actual shooting), as described later, is detected. However, the control unit 101 is not limited to this, and may be configured to set the parameters for the actual shooting separately and independently based on user instructions or automatically.

[0055] In step S603, the control unit 114 determines whether motion blur notification is set to ON or OFF. The setting for ON or OFF of motion blur notification may be set by the user using the operation unit 115, or it may be set automatically based on some shooting conditions. The ON / OFF setting may be made possible by a single physical operation element (button, bar, etc.) or a single icon on a touch device, allowing the user to set ON or OFF at any time during preparation shooting. Furthermore, it may be possible to set the display to periodically switch ON and OFF.

[0056] If the control unit 114 determines in step S603 that motion blur notification is set to ON, the process proceeds to step S604. In step S604, the image processing unit 104 detects a region based on a subject detected using the subject detection means, a region based on a subject specified via the operation unit 115, or a main subject region based on a subject specified via the operation unit 115, and decides to estimate and notify the motion blur of that region. In that region, the motion blur notification image generation unit 300 generates a motion blur notification image by superimposing a motion blur notification plane onto the prepared captured image. Then, in step S606, the control unit 114 displays the motion blur notification image on the display device of the display unit 113.

[0057] If the control unit 114 determines in step S603 that motion blur notification is set to OFF, the process proceeds to step S606. However, in this case, the control unit 114 displays the prepared image on the display device of the display unit 113.

[0058] In step S607, the control unit 114 determines whether the shutter button on the operation unit 115 has been pressed down by user operation. Here, if the shutter button input is configured to accept a two-stage input method, such as a half-press to indicate a shooting preparation operation and a full press to indicate actual shooting, the control unit 114 determines whether a full press has been performed. If only a simple one-stage input is accepted, the control unit 114 determines whether that one-stage input has been performed.

[0059] If the control unit 114 determines that the shutter button is not pressed, it returns to step S602 and repeats the process from step S602 to step S606. This allows the user to easily check the motion blur that would occur in the subject if the actual shooting were performed with the currently set shooting parameters, even while preparation shooting is in progress. If motion blur is detected and it is not to the user's liking (i.e., motion blur is undesirable), the user can simply reset the shutter speed (exposure time) for the actual shooting without pressing the shutter button.

[0060] In this way, by notifying the user of motion blur during the preparation shooting, the user can check the motion blur notification image displayed on the display unit 113 and repeat the exposure time setting for the main shoot until the desired level of motion blur is achieved. After that, the user can capture the shot when the exposure time setting is appropriate for the motion blur.

[0061] If the control unit 114 determines in step S607 that the shutter button has been pressed, it recognizes that it has received a shooting instruction for the main shooting and proceeds to step S608. In step S608, the control unit 114 controls the imaging optical unit 101, the imaging unit 102, etc., to perform the main shooting based on the shooting parameters set up during the preparation shooting. In step S609, the image obtained by the main shooting is output by the control unit 114 to the display unit 113 and the recording media control unit 110, and is displayed on the display device of the display unit 113, recorded to the recording media 111 by the recording media control unit 110, or output to an external device via a communication unit (not shown), etc.

[0062] Next, an example of the configuration of the motion blur notification image generation unit 300, which is included in the image processing unit 104 that is a feature of the present invention, will be described with reference to Figure 7.

[0063] Figure 7 shows an example of the configuration of the motion blur notification image generation unit 300. The motion blur notification image generation unit 300 includes a motion vector calculation unit 301 that calculates the motion vector of the subject from a comparison between images, and an estimated motion blur calculation unit 302 that estimates the motion blur of the subject during the actual shooting based on the calculated motion vector. Furthermore, it consists of a motion blur notification plane creation unit 303 that creates data for notifying motion blur based on the estimated motion blur of the subject, and an image superposition unit 304 that performs superposition processing to superimpose the motion blur notification plane onto the captured image.

[0064] Furthermore, one or more of the functional blocks shown in Figure 3 may be implemented by hardware such as an ASIC or a programmable logic array (PLA), or by a programmable processor such as a CPU or MPU executing software. They may also be implemented by a combination of software and hardware. Therefore, even if different functional blocks are described as the primary operating components in the following explanation, the same hardware may be the primary implementing component.

[0065] Next, the process by which the motion blur notification image generation unit 300 generates motion blur notification images will be explained in detail with reference to the flowchart in Figure 8. Each step in this flowchart is executed by the control unit 101 or by the various parts of the digital camera 100, including the motion blur notification image generation unit 300, at the instruction of the control unit 101.

[0066] In step S801, the control unit 114 inputs the preparatory shooting images sequentially acquired by the imaging unit 102 and the shooting parameters to be used during the actual shooting to the motion blur notification image generation unit 300. An example of a preparatory shooting image is shown in Figure 9(a). In this invention, we will explain using an example in which a car 901 moving from right to left is being photographed, as shown in Figure 9(a).

[0067] In step S802, the motion vector calculation unit 301 of the motion blur notification image generation unit 300 calculates motion vectors between the prepared images as motion information. A motion vector is a vector representation of the horizontal and vertical movement of the subject between the prepared images. The method for calculating motion vectors will be explained in detail with reference to Figures 10 and 11.

[0068] Figure 10 is a flowchart illustrating the motion vector calculation process performed by the motion vector calculation unit 301. In this invention, the block matching method is used as an example of the motion vector calculation method, but the motion vector calculation method is not limited to this example, and other methods such as the gradient method may also be used. Each step in this flowchart is executed by the control unit 114 or by the parts of the digital camera 100, including the motion blur notification image generation unit 300, at the instruction of the control unit 114.

[0069] In step S1001, the motion vector calculation unit 301 receives two temporally adjacent preparatory images. The motion vector calculation unit 301 then sets the M-th frame preparatory image as the reference frame and the M+1-th frame preparatory image as the reference frame.

[0070] In step S1002, the motion vector calculation unit 301 places an N×N pixel reference block 702 in the reference frame 701, as shown in Figure 10.

[0071] In step S1003, the motion vector calculation unit 301 sets the search range 705 for the reference frame 703 as follows, using the (N+n) × (N+n) pixels surrounding the center coordinates 704 of the reference block 702 of the reference frame 701, as shown in Figure 11.

[0072] In step S1004, the motion vector calculation unit 301 performs a correlation calculation between the reference block 702 of the reference frame 701 and the reference block 706 of N×N pixels at different coordinates within the search range 705 of the reference frame 703, and calculates a correlation value. The correlation value is calculated based on the sum of absolute values ​​of inter-frame differences for the pixels of the reference block 702 and the reference block 706. In other words, the coordinate with the smallest sum of absolute values ​​of inter-frame differences is the coordinate with the highest correlation value. Note that the method for calculating the correlation value is not limited to calculating the sum of absolute values ​​of inter-frame differences; for example, a method of calculating the correlation value based on the sum of squares of inter-frame differences or the normal cross-correlation value may also be used. In the example in Figure 11, it is assumed that the reference block 706 has the highest correlation.

[0073] In step S1005, the motion vector calculation unit 301 calculates a motion vector based on the reference block coordinates showing the highest correlation value obtained in step S1004. In the example in Figure 10, within the search range 705 of the reference frame 703, the motion vector is determined based on the same coordinate 704 corresponding to the center coordinate of the reference block 702 of the reference frame 701 and the center coordinate of the reference block 706. In other words, the distance and direction between the same coordinate 704 and the center coordinate of the reference block 706 are determined as the motion vector.

[0074] In step S1006, the motion vector calculation unit 301 determines whether or not it has calculated motion vectors for all pixels of the reference frame 701. If the motion vector calculation unit 301 determines in step S1006 that it has not calculated motion vectors for all pixels, it returns to step S1002. In step S1002, an N×N pixel reference block 702 is placed in the reference frame 701, centered on the pixels for which motion vectors have not been calculated, and the processing from steps S1003 to 1005 is performed as described above. That is, the motion vector calculation unit 301 calculates motion vectors for all pixels of the reference frame 701 by repeating the processing from steps S1002 to S1005 while moving the reference block 702 shown in Figure 11. An example of these motion vectors is shown in Figure 9(b). Figure 9(b) is a diagram showing an example of motion vectors for the prepared image captured in Figure 9(a). It also shows the main subject area 902 detected by the subject area detection of the image processing unit 104.

[0075] Figure 9(a) shows an example of a preparatory image where car 501 is moving to the left. A typical example of a motion vector when the subject is moving is shown in Figure 9(b). In the example in Figure 9(b), the moving car 501 is detected as a motion vector to the left, while the stationary background fence is detected as a motion vector of 0, and therefore its motion vector is not shown.

[0076] Furthermore, the motion vector calculation unit 301 may calculate motion vectors for predetermined pixels, such as those around the subject area, rather than calculating motion vectors for all pixels.

[0077] Through the above process, the motion vector calculation unit 301 calculates motion vectors between temporally adjacent preparation images.

[0078] The process by which the motion vector calculation unit 301 calculates motion vectors has been explained above.

[0079] In step S803, the estimated motion blur calculation unit 302 acquires the exposure time for the main shoot and the time interval between images in the preparation shoot, which were set in step S602, as shooting conditions.

[0080] In step S804, the estimated motion blur calculation unit 302 estimates the motion blur of the main shoot from the pixel-by-pixel motion vector calculated in step S302, based on the exposure time of the main shoot and the time interval between images in the preparation shoot acquired in step S803. The method for estimating the motion blur of the main shoot will be explained in detail with reference to Figure 12. Figure 12 is a diagram showing the motion vector in the preparation shoot and the estimated motion blur estimated for the main shoot. In Figure 12, the shooting conditions are shown as follows: the time interval between images in the preparation shoot is 1 / 60 second, and the exposure times for the main shoot are 1 / 120 second and 1 / 30 second as examples.

[0081] The estimated motion blur calculation unit 302 estimates the motion vector for each pixel as the motion blur of the actual shooting based on the estimation formulas shown in equations (1) and (2). CONV_GAIN=EXP_TIME / INT_TIME···Formula (1) CONV_BLUR=VEC_LEN×CONV_GAIN...Equation (2) Here, in equation (1), CONV_GAIN represents the estimated gain for estimating the motion vector of the preparatory shot into the motion vector of the main shot, EXP_TIME represents the exposure time of the main shot, and INT_TIME represents the time interval between preparatory shots. Also, in equation (2), CONV_BLUR represents the estimated motion blur of the main shot, and VEC_LEN represents the length of the motion vector in the preparatory shot.

[0082] In equation (1), the estimated gain is calculated by dividing the exposure time of the main shot by the time interval between the preparatory shots. Then, in equation (2), the estimated motion blur of the main shot is calculated by multiplying the length of the motion vector by the estimated gain.

[0083] Specifically, as shown in Figure 12, if the length of the motion vector in the preparation shot is 10 pixels, the estimated motion blur for the main shot with an exposure time of 1 / 120 second will be 5 pixels because the estimated gain is halved. Similarly, the estimated motion blur for the main shot with an exposure time of 1 / 30 second will be 20 pixels because the estimated gain is doubled.

[0084] In step S805, the motion blur notification plane creation unit 303 creates an image plane for notifying motion blur based on the estimated motion blur for each pixel calculated in step S804. Note that the image plane is created only for the area determined based on the designated area of ​​the main subject area acquired in step S604. In other words, the image processing unit 104 in step S604 also serves as a designation means for designating the main subject area as the area to be notified.

[0085] In step S806, the image superposition unit 304 superimposes the motion blur notification plane created in step S805 onto the prepared captured image to generate a motion blur notification image.

[0086] Here, the method for generating motion blur notification images will be explained in detail with reference to Figure 13. Figure 13 shows three examples of motion blur notification images. By displaying the motion blur notification image on the display unit 113 during preparation shooting, the user can easily check for motion blur.

[0087] Figure 13(a) shows an example of notifying motion blur using a motion blur frame display. Here, the method for generating a motion blur notification image using a motion blur frame display will be explained. In step S805, the motion blur notification plane creation unit 303 calculates the ratio of the number of pixels showing estimated motion blur of a predetermined value or more to the total number of pixels in the divided region within the specified area. For the divided region where this ratio is greater than or equal to the predetermined ratio, a motion blur frame 902 as shown in Figure 13(a) is created as a motion blur notification plane, and by superimposing it on the prepared captured image, a motion blur notification image for the main subject region 902 as shown in Figure 13(a) is generated.

[0088] Figure 13(b) shows an example of notifying motion blur by highlighting edges where motion blur occurs. Here, the method for generating a motion blur notification image by highlighting motion blur edges will be explained. In step S805, the motion blur notification plane creation unit 303 detects the edge intensity of the prepared image within a specified area. The calculation of edge intensity is done using an existing method such as a Sobel filter, and the explanation will be omitted. The motion blur notification plane creation unit 303 then extracts pixels in which the edge intensity is greater than or equal to a predetermined value and the estimated motion blur is greater than or equal to a predetermined value. For the extracted pixels, a motion blur notification plane is created that highlights the motion blur edges as shown in 903 of Figure 13(b), and by superimposing this plane onto the prepared image, a motion blur notification image like the one in Figure 13(b) is generated. In the example of 903 in Figure 13(b), an example is shown in which the motion blur edges of the main subject area 902 are made thicker. Another example of a highlighting method is to extract pixels where the edge strength is above a predetermined value and the estimated motion blur is above a predetermined value, and then highlight the extracted pixels in red.

[0089] The above describes the process by which the motion blur notification image generation unit 300 generates motion blur notification images. In this text, these processes will be collectively referred to as "notification processing."

[0090] In this invention, motion blur is notified for the main subject area, and motion blur for the background, which is estimated when the camera is panning left or right, is hidden, making it easier to check for motion blur by displaying only the motion blur of the main subject area.

[0091] [Second Embodiment] Next, the processing of the digital camera 100 in the second embodiment will be explained in detail with reference to the flowchart in Figure 14. Note that the processing of steps S1401 to S1408, other than step S1404, is the same as the processing of steps S601 to S608 in Figure 6, so the explanation will be omitted. Each step in this flowchart is executed by the control unit 114 or by the parts of the digital camera 100 at the instruction of the control unit 114.

[0092] Figure 15 shows a motion blur notification method 1 according to a second embodiment of the present invention, and shows a preparatory captured image in which motion blur notification is performed only within a specified area. In step S1404, the user specifies an area by touch operation on the shooting screen displayed on the touch panel-compatible LCD of the operation unit 115. The selected area 1501 is used as the specified area, and each process is performed in the motion blur notification plane creation unit 303 of Figure 3, and in step S1405, motion blur notification is performed only for area 1501 by highlighting the motion blur edge. It is also possible to notify only the area within a predetermined range from a position specified by the user. Furthermore, it is also possible to perform motion blur notification on a specific area within a predetermined range from a specified position.

[0093] In the above embodiment, the area 1501 was specified using a touch-panel compatible LCD as an example of the operation unit 115, but it is also possible to specify the area by setting the AF frame using the directional keys on the operation unit 115. Here, Figure 16 shows a motion blur notification method 2 according to a second embodiment of the present invention. Figure 16(a) is a preparatory shooting image in which nine AF frames are displayed overlapping the subject 1601.

[0094] First, the upper left frame 1602 of the AF frame is set by operating the control unit 115. When the AF frame 1602 overlaps the left foot of the subject 1601, the image processing unit 104 performs semantic region division. Then, as shown in Figure 16(b), the motion blur notification plane creation unit 303 executes each process with the left foot portion 1603 of the subject 1601 as the designated region. As a result, in step S1405, motion blur notification is performed by highlighting the motion blur edge only for the left foot portion 1603 of the subject 1601, as shown in Figure 16(b).

[0095] In this invention, by limiting the motion blur notification area through operation of the control unit 115, motion blur notification for the background, which is estimated when the camera is panning left or right, can be hidden. Then, by displaying only the motion blur in the specified area, it becomes easier to check for motion blur.

[0096] [Third Embodiment] Next, the processing of the digital camera in the third embodiment will be explained in detail with reference to the flowchart in Figure 17. Note that the processing of steps S1701 to S1708, other than step S1704, is the same as the processing of steps S601 to S608 in Figure 6, so the explanation will be omitted. Each step in this flowchart is executed by the control unit 114 or by the parts of the digital camera 100 at the instruction of the control unit 114.

[0097] Figure 18 shows a motion blur notification method according to a third embodiment of the present invention. In step S1704, the gaze detection circuit 118 detects the user's gaze area 1802 according to the gaze detection schematic flow routine shown in Figure 5 above. Similar to Embodiment 2, when the gaze area 1802 overlaps with the left foot of the subject 1801, the image processing unit 104 performs semantic region division and designates the left foot area 1803 of the subject 1801 as the specified region, and the motion blur notification plane creation unit 303 in Figure 3 performs each process. In step S1705 in Figure 17, motion blur notification is performed by highlighting the motion blur edge only for the left foot area 1803 of the subject 1801.

[0098] In this invention, by limiting the motion blur notification area to the gaze area detected by the gaze detection means, motion blur notification for the background, which is estimated when the camera is panning left or right, can be hidden. Then, by displaying only the motion blur of the gaze area detected by the gaze detection means, it becomes easier to check for motion blur.

[0099] [Fourth Embodiment] Next, the processing of the digital camera 100 in the fourth embodiment will be described. In this embodiment, the same processing as the flowchart in Figure 17 described for the image value 100 in the third embodiment is performed.

[0100] Figure 19 illustrates the operation of changing the visibility of the motion blur notification area according to the movement range of the gaze area. The processing details will be explained below using Figure 19. The gaze detection circuit 118 detects how much the gaze area has moved when it captures each subject with its gaze. This allows the gaze detection circuit 118 to calculate the movement range of the gaze. Here, a reference number of judgment pixels is set for the amount of gaze movement. The number of judgment pixels is a value used to determine whether a subject included in the gaze area is the main subject or not based on the number of gaze movement pixels, and is variable depending on the shooting scene and image resolution. In the motion blur notification plane creation unit 303 of Figure 7, if the movement range of the gaze relative to the subject is smaller than the number of judgment pixels, the motion blur notification area is displayed with high visibility as the subject is considered the main subject. Conversely, if the movement range of the gaze relative to the subject is larger than the number of judgment pixels, it is determined that it is not the main subject, and the motion blur notification area is displayed with low visibility.

[0101] To differentiate the visibility of the motion blur notification area, if the image area displaying the motion blur notification area is white, the colors used to display the motion blur edge in order of highest visibility may be, for example, red, yellow, orange, blue, pink, light green, light purple, green, and purple. If the image area displaying the motion blur notification area is black, the colors used to display the motion blur edge in order of highest visibility may be, for example, yellow, orange, red, light green, pink, blue, green, light purple, and purple. The colors used to display the motion blur edge in order of highest visibility are not limited to these, and there are no particular restrictions on the colors or the number of colors used.

[0102] In addition, while the visibility of the motion blur notification area was expressed using the color of the motion blur edge, other methods are also possible. For example, in Figure 19, the movement amount 1921 of the gaze area 1911 relative to subject 1901 is 250 pixels, the movement amount 1922 of the gaze area 1912 relative to subject 1902 is 1500 pixels, and the movement amount 1923 of the gaze area 1932 relative to subject 1903 is 1800 pixels. Here, when the number of judgment pixels is 500 pixels, the motion blur notification plane creation unit 303 in Figure 7 changes the thickness of the motion blur edge for subject 1901 captured by gaze area 1911 and displays it accordingly.

[0103] In this invention, when the range of eye movement (sway) relative to the subject is smaller than the number of detection pixels, the main subject is improved by increasing the visibility of the motion blur notification area, making it easier to confirm the motion blur of the subject.

[0104] [Fifth Embodiment] Next, the processing of the digital camera 100 in the fifth embodiment will be described. In this embodiment, the same processing as in the flowchart of Figure 17 described for the digital camera 100 in the third embodiment is performed.

[0105] Figure 20 shows the operation of changing the visibility of the motion blur notification area according to the gaze duration of the gaze area. The processing details will be explained below using Figure 20. The gaze detection circuit 118 detects how long the gaze duration is when each subject is captured by the gaze. Here, a standard judgment gaze duration is set for the length of the gaze duration. The judgment gaze duration is the length of time used to determine whether the subject being gazed on is the main subject, and is variable depending on the shooting scene and shooting conditions. In the motion blur notification plane creation unit 303 of Figure 7, if the gaze duration for the subject is longer than the judgment gaze duration, the subject is considered the main subject, and the motion blur notification area is displayed with high visibility. Conversely, if the gaze duration for the subject is shorter than the judgment gaze duration, it is determined that the subject is not the main subject, and the motion blur notification area is displayed with low visibility. The means for adjusting the visibility of the motion blur notification area are explained in the fourth embodiment, so they will be omitted here.

[0106] In Figure 20, the gaze time for the gaze region 2011 relative to subject 2001 is 6400ms, the gaze time for the gaze region relative to subject 2002 is 500ms, and the gaze time for the gaze region relative to subject 2003 is 3300ms. In the motion blur notification plane creation unit 303 in Figure 7, the thickness of the motion blur edge is changed and displayed for subject 2001, which is captured by the gaze region 2011 with the longest gaze time.

[0107] In this invention, when the gaze is fixed on a subject for a long period of time, the visibility of the motion blur notification area is improved for the main subject, making it easier to confirm the motion blur of the main subject.

[0108] [Sixth Embodiment] Next, the processing of the digital camera 100 in the sixth embodiment will be described. In this embodiment, the same processing as in the flowchart of Figure 17 described for the digital camera 100 in the third embodiment is performed.

[0109] Figure 21 illustrates the operation of changing the visibility of the motion blur notification area according to its distance from the gaze area. The processing details will be explained below using Figure 21. The gaze detection circuit 118 detects how far other subjects are from the gaze area in which the main subject is captured by the gaze. In the motion blur notification plane creation unit 303 of Figure 7, the visibility of the motion blur notification area of ​​other subjects is changed according to the distance between the gaze area in which the main subject is captured by the gaze and the other subjects. The closer the distance to the other subjects, the more visible the motion blur notification area of ​​those subjects will be displayed, and the farther the distance, the less visible the display will be. The means for adjusting the visibility of the motion blur notification area was explained in the fourth embodiment and will be omitted here.

[0110] In Figure 21, the distance to subject 2101 relative to the gaze area is 0 pixels, the distance to subject 2102 relative to the gaze area is 1200 pixels, and the distance to subject 2102 relative to the gaze area is 1500 pixels. In the motion blur notification plane creation unit 303 of Figure 7, the thickness of the motion blur edge is changed and displayed for subjects that are closer to the gaze area 2111.

[0111] In this invention, when the distance of the subject to the gaze area is short, the motion blur of the main subject is made easier to identify by improving the visibility of the motion blur notification area.

[0112] [Seventh Embodiment] Next, the processing of the digital camera 100 in the seventh embodiment will be described. In this embodiment, the same processing as the flowchart in Figure 17 described for the digital camera 100 in the third embodiment is performed.

[0113] Figure 22 shows a method for displaying the motion blur area relative to the gaze area. The subject 2201 is moving to the left 2241 (arrow) in the drawing. Here, the gaze area 2211 is also moving to the left 2231 (arrow) in the drawing while following the subject 2201. Generally, the tracking motion of the eyeball lags behind that of a moving subject, so the gaze area 2211 is expected to be located at a distance from the subject 2201. In other words, the display area of ​​the gaze area will be shifted relative to the subject. To cancel this difference caused by the delay in eyeball movement, the motion blur notification plane creation unit 303 in Figure 7 displays a motion blur edge 2221 for the subject 2201 which is ahead in the direction of movement of the gaze area.

[0114] In this invention, by canceling the delay in the eye's tracking motion towards the subject, the motion blur notification area can be appropriately displayed relative to the subject. Since the tracking delay varies from person to person, it may be possible to perform calibration or similar procedures in advance to measure the tracking delay and adjust the display position of the motion blur notification area relative to the subject being tracked by the gaze area. Alternatively, the display of the gaze area may be adjusted to follow the speed of the subject being tracked by the gaze area.

[0115] (Other embodiments) Although the present invention has been described above based on embodiments, the present invention is not limited to these embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention.

[0116] The object of the present invention can also be achieved as follows: a storage medium containing program code for software describing the procedures for realizing the functions of each embodiment described above is supplied to a system or device. The computer (or control unit, MPU, etc.) of the system or device then reads and executes the program code stored on the storage medium.

[0117] In this case, the program code read from the storage medium itself realizes the novel function of the present invention, and the storage medium and program that store that program code constitute the present invention.

[0118] Furthermore, storage media for supplying program code include, for example, flexible disks, hard disks, optical disks, and magneto-optical disks. CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-Rs, magnetic tapes, non-volatile memory cards, and ROMs can also be used.

[0119] Furthermore, the functions of each of the embodiments described above are realized by making the program code read by the computer executable. In addition, this also includes cases in which the OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program code, and the functions of each of the embodiments described above are realized through that processing.

[0120] Furthermore, the following cases are also included: First, program code read from a storage medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer. Then, based on the instructions of that program code, the control unit and other components of that function expansion board or function expansion unit perform some or all of the actual processing. [Explanation of symbols]

[0121] 101 Imaging Optics Section 102 Imaging Unit 103 A / D Conversion Unit 104 Image Processing Unit 105 Data Transfer Section 106 Memory Control Unit 107 DRAM (memory) 108 Non-volatile memory control unit 109 ROM (Non-Volatile Memory) 110 Recording media control unit 111 Recording media 112 Display Control Unit 113 Display section 114 CPU 115 Operation section 116 Bus (System Bus) 117 Bus 118 Eye-tracking circuit 119 Image sensor for the eye 201 aperture 202 Lens drive motor 203 Lens driving component 204 Photocoupler 205 Pulse Plate 206 Mount contacts 207 Aperture control circuit 208 Focus adjustment circuit 301 Motion Vector Calculation Unit 302 Estimated motion deviation calculation unit 303 Motion Blur Notification Plane Creation Unit 304 Image Overlay Area

Claims

1. An acquisition means for acquiring a plurality of first captured images obtained by performing a first capture with first capture parameters and movement information of a subject in the plurality of first images; An estimation means for estimating a motion blur of a subject in a second captured image obtained by a second capture based on the first capture parameters, the movement information, and the second capture parameters when the second capture is performed with second capture parameters; A notification means for performing a notification process corresponding to the information on the motion blur; A designation means for designating a subject for which the notification process is to be performed by the notification means; A determination means for determining an area for performing the notification process based on the subject designated by the designation means, and having: The information processing apparatus, wherein the notification means performs the notification process on the area determined by the determination means.

2. The determination means includes a subject detection means, The information processing apparatus according to claim 1, wherein the notification means performs the notification process on a subject area detected by the subject detection means as an area for performing the notification.

3. The designation means is a touch panel, The information processing apparatus according to claim 1, wherein the notification means performs the notification process on the area determined by the determination means based on a subject designated by an operation of the touch panel by a user.

4. The designation means is an operation member for designating the position of an AF frame, The information processing apparatus according to claim 1, wherein the notification means performs the notification process on the area determined by the determination means based on the position of the AF frame designated by an operation of the operation member.

5. The information processing apparatus according to claim 1, wherein the designation means is a fixation area detection means for detecting a fixation area of a user.

6. The information processing apparatus according to claim 5, wherein the notification means changes the visibility of the notification process according to at least one of a range in which the fixation area moves, a fixation time, and a distance from the fixation area.

7. The information processing apparatus according to claim 5, wherein the notification means changes a display area of the notification process according to a moving direction and a speed of the fixation area.

8. The estimation means estimates the motion blur for a specific area in the designated subject, The information processing apparatus according to any one of claims 1 to 7, wherein the notification means performs the notification process on the specific area based on the estimated motion blur.

9. The information processing apparatus according to claim 8, wherein the specific area includes at least one of a human, an animal, and a part of a vehicle.

10. The specifying means specifies the subject by specifying the position of the first captured image, The determination means determines, as an area for performing the notification process, the specific area detected within a predetermined range from the position specified by the specifying means. The information processing apparatus according to claim 8.

11. An imaging means, The information processing apparatus according to any one of claims 1 to 7, An imaging device having the above.

12. An acquisition step of acquiring a plurality of first captured images obtained by performing a first capture with a first imaging parameter and motion information of a subject in the plurality of first images; When a second capture is performed with a second imaging parameter, an estimation step of estimating motion blur of a subject in a second captured image obtained by the second capture based on the first imaging parameter, the motion information, and the second imaging parameter; A notification step of performing a notification process corresponding to the motion blur information; A specifying step of specifying a subject for which the motion blur is notified in the notification step; A determination step of determining an area for performing the notification process based on the subject specified in the specifying step, and The notification step performs the notification process on the area determined in the determination step. A control method for an information processing apparatus.

13. An imaging device having an imaging step, wherein when a shooting instruction is given by a user while a first captured image is sequentially output by a first imaging with a first imaging parameter in the imaging step, a second captured image is output by a second imaging with a second imaging parameter in response to the shooting instruction, A calculation step of calculating motion information of a subject in the first captured image based on the plurality of first captured images output in the imaging step; A setting step of setting the second imaging parameter; An estimation step of estimating motion blur in the second captured image based on the first imaging parameter, the motion information, and the second imaging parameter; A notification step of performing a notification process corresponding to the motion blur information; A specifying step of specifying a subject for which the notification process is performed in the notification step; A determining step of determining an area for performing the notification process based on the subject specified in the specifying step, and having: The notification step is a control method for an imaging device, characterized in that the notification process is performed on the area determined in the determining step.

14. A program for causing a computer to execute as each means of the information processing apparatus according to any one of Claims 1 to 7.

15. A program for causing a computer to execute as each means of the information processing apparatus according to Claim 8.

16. A computer-readable storage medium storing a program for causing a computer to execute as each means of the information processing apparatus according to any one of Claims 1 to 7.

17. A computer-readable storage medium storing a program for causing a computer to execute as each means of the information processing apparatus according to Claim 8.