Information processing apparatus, imaging apparatus, information processing method and control method, and program

Abstract

To appropriately select frames of a photographed image to easily check a motion shake that may occur in main photographing.SOLUTION: An information processing apparatus comprises: acquisition means that acquires a plurality of first photographed images obtained by performing first photographing according to a first photographic parameter and motion information on a subject in the plurality of first photographed images; estimation means that, when second photographing is performed according to a second photographic parameter set independently of the first photographic parameter, estimates a motion shake of the subject in a second photographed image obtained through the second photographing from the motion information and the second photographic parameter; setting means that sets the first photographic parameter and the second photographic parameter; and notification means that notifies information on the motion shake. The acquisition means acquires the motion information from the plurality of first photographed images obtained according to the first photographic parameter that is changed in correspondence with an exposure time in the second photographing related to the second photographic parameter.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a technique for notifying a subject blur occurring in a captured image.

Background Art

[0002] Some imaging devices such as digital still cameras have a shooting mode that prioritizes the shutter speed (hereinafter referred to as "shutter speed priority mode"). In the shutter speed priority mode, the photographer sets a desired shutter speed, and the imaging device automatically sets exposure setting values other than the shutter speed, such as the aperture value and the ISO sensitivity. By using the shutter speed priority mode, the photographer can shoot at a preferred shutter speed. For example, by setting a high shutter speed before shooting and performing shooting in the shutter speed priority mode, an image with less motion blur can be captured.

[0003] Patent Document 1 discloses a technique for detecting a motion area between time-series images captured during preparatory shooting and highlighting the motion area. Here, preparatory shooting refers to shooting in which, before actual shooting, while looking at the electronic viewfinder or the rear liquid crystal of the imaging device, the composition of the image and the setting of shooting conditions are adjusted. Actual shooting refers to shooting that is executed by the imaging device based on the composition and shooting condition settings determined during preparatory shooting, triggered by an action such as the photographer pressing the shutter button. According to Patent Document 1, the photographer can visually confirm the motion area during preparatory shooting.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to Patent Document 1, it is possible to detect motion regions between time-series images captured during preparation shooting and to encourage the user to take actions to reduce blur by emphasizing those motion regions.

[0006] However, Patent Document 1 does not consider capturing images at an appropriate frame rate and shutter speed according to the subject's speed and movement in order to extract motion regions between time-series images.

[0007] For example, when taking portraits at night, the amount of light that can be captured by the image sensor is less than during the day, resulting in a longer exposure time. This means that even small movements or tremors of the subject can cause motion blur. In this case, in order to properly notify the user of motion blur, it is necessary to detect the subject's movement that affects the exposure time (corresponding to the set shutter speed). However, the frame rate value may not be set in advance to a value sufficient to detect small movements of the subject, resulting in the problem of not being able to properly notify the user of the occurrence of motion blur.

[0008] This invention was made in view of the above-mentioned problems, and aims to provide an information processing device, imaging device, information processing method, control method, and program that enable the confirmation of motion blur occurring during actual shooting by selecting an image that affects motion blur (also called "motion blur detection") according to the exposure time of shooting. [Means for solving the problem]

[0009] To solve the above problems, one of the information processing devices of the present invention includes: an acquisition means for acquiring a plurality of first captured images obtained by performing a first shooting with a first shooting parameter and motion information of a subject in the plurality of first captured images; an estimation means for estimating motion blur of a subject in a second captured image obtained in a second shooting from the motion information and the second shooting parameter when a second shooting is performed with a second shooting parameter set independently of the first shooting parameter; a setting means for setting the first shooting parameter and the second shooting parameter; and a notification means for notifying the motion blur information, wherein the acquisition means acquires the motion information from the plurality of first captured images obtained with the first shooting parameter changed in accordance with the exposure time in the second shooting related to the second shooting parameter.

[0010] Furthermore, one of the information processing devices of the present invention includes an acquisition means for acquiring a plurality of first captured images obtained by performing a first shooting with a first shooting parameter and motion information of a subject in the plurality of first captured images; an estimation means for estimating motion blur of a subject in a second captured image obtained in a second shooting from the motion information and the second shooting parameter when a second shooting is performed with a second shooting parameter set independently of the first shooting parameter; a setting means for setting the first shooting parameter and the second shooting parameter; and a notification means for notifying the motion blur information, wherein the acquisition means determines an image to be used to acquire the motion information from the plurality of first captured images based on the exposure time in the second shooting related to the second shooting parameter. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an information processing device, an imaging device, an information processing method, a control method, and a program that can display a blur notification image that allows for accurate confirmation of motion blur occurring during the actual shooting by appropriately selecting frames of preparatory images for detecting motion. [Brief explanation of the drawing]

[0012] [Figure 1] Figure showing a configuration example for implementing the present invention [Figure 2] Flowchart showing the shooting process of imaging device 100 [Figure 3] Figure showing a configuration example of the motion blur notification image generation unit [Figure 4] Figure for explaining the process of S202 in FIG. 2 of Example 1 [Figure 5] Figure for explaining the relationship between the amount of movement of the subject and the allowable amount of movement [Figure 6] Figure showing the preparatory shooting image and the motion vector [Figure 7] Flowchart showing the method for calculating the motion vector [Figure 8] Figure showing the method for calculating the motion vector [Figure 9] Figure showing the motion blur notification method [Figure 10] Timing chart for obtaining the preparatory shooting image and the actual shooting image [Figure 11] Figure for explaining the exposure time of the actual shooting and the frame interval of the preparatory shooting [Figure 12] Figure for explaining the integration of the amount of movement between frames [Figure 13] Figure for explaining the control of the frame rate [Figure 14] Figure for explaining the process of S208 in FIG. 2 of Example 2

Embodiments for Carrying Out the Invention

[0013] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

Examples

[0014] In Example 1 of the present invention, motion information calculated from an image during preparatory shooting is used to estimate motion blur corresponding to actual shooting, and the motion blur expected to occur during actual shooting is displayed as a motion blur notification image to notify the user. An imaging device will be described.

[0015] FIG. 1 is a block diagram showing a configuration example of an imaging device as an example of an information processing device and an imaging device in Embodiment 1 of the present invention. Hereinafter, the configuration example of Embodiment 1 of the present invention will be described with reference to FIG. 1. The information processing device referred to in the present invention is applicable to any electronic device capable of processing a captured image. These electronic devices may include, for example, mobile phones, game machines, tablet terminals, personal computers, watch-type or glasses-type information terminals, and the like.

[0016] The control unit 101 is, for example, a CPU, reads a control program for each block included in the imaging device 100 (e.g., a camera) from the ROM 102 described later, and develops and executes it in the RAM 103 described later. Thereby, the control unit 101 controls the operations of each block included in the imaging device 100. The ROM 102 is an electrically erasable and recordable non-volatile memory, and stores operation programs for each block included in the imaging device 100, as well as parameters and the like necessary for the operations of each block. The RAM 103 is a rewritable volatile memory, and is used for developing programs executed by the control unit 101 and the like, and temporarily storing data generated by the operations of each block included in the imaging device 100.

[0017] The optical system 104 is composed of a lens group including a zoom lens and a focus lens, and forms an object image on the imaging surface of the imaging unit 105 described later. The imaging unit 105 is an imaging element such as a CCD or a CMOS sensor, photoelectrically converts the optical image formed on the imaging surface of the imaging unit 105 by the optical system 104, and outputs the obtained analog image signal to the A / D conversion unit 106. The A / D conversion unit 106 converts the input analog image signal into digital image data. The digital image data output from the A / D conversion unit 106 is temporarily stored in the RAM 103.

[0018] The image processing unit 107 applies various image processing functions, such as white balance adjustment, color interpolation, and gamma processing, to the image data stored in the RAM 103. The image processing unit 107 also includes a motion blur notification image generation unit 300 (described later), which superimposes an image plane that allows for easy confirmation of motion blur onto the image stored in the RAM 103 to generate a motion blur notification image. The recording unit 108 is a removable storage medium such as a memory card. The recording unit 108 records the image data processed by the image processing unit 107 as a recorded image via the RAM 103. The display unit 109 is a display device such as an LCD, which displays images stored in the RAM 103 and images recorded in the recording unit 108, and displays an operation user interface for receiving user instructions. The display unit 109 also displays images captured by the imaging unit 105 during preparation shooting for composition adjustment, etc. The instruction input unit 110 is a touch panel or mouse, which receives user instructions.

[0019] Next, Embodiment 1 will be described in detail with reference to the flowchart in Figure 2. The following processes are realized by the control unit 101 controlling each part of the device according to the program stored in the ROM 102. The flowchart in Figure 2 is executed when preparation imaging is started.

[0020] In S201, the user turns on the power to the imaging device 100. In response to the power being turned on to the imaging device 100, the control unit 101 controls the optical system 104 and the imaging unit 105 to start preparatory shooting, such as composing the shot. During this preparatory shooting period, the control unit 101 maintains a predetermined frame rate, and the imaging device 100 sequentially captures images, displaying the acquired images on the display unit 109. The user can then perform composing and other actions while viewing the displayed preparatory shooting images. Here, images captured during preparatory shooting are defined as preparatory shooting images.

[0021] In S202, the imaging device 100 sets the shooting conditions (shooting parameters) for the preparatory images to be captured to detect the amount of movement of the subject within the composition. These shooting conditions primarily refer to the frame rate and shutter speed. The highest possible frame rate and shutter speed are set, within a range that does not affect the calculation of evaluation values ​​used for controlling auto functions such as AE (auto exposure) and AF (autofocus) in typical imaging devices. Furthermore, even when a high shutter speed is set, the imaging device 100 controls the optical system 104 to adjust the lens aperture and the ISO sensitivity setting of the imaging unit 105 to ensure proper exposure. The imaging device 100 captures a series of images sequentially during the preparatory shooting process using the shooting conditions set in S202. The faster the shutter speed, the less accumulated blur there is in the captured images of a moving subject, and the smaller the amount of subject movement between captured images, allowing for detailed capture of the subject's movement.

[0022] In S203, the user uses the instruction input unit 110 to set the shooting parameters for the main shoot. The control unit 101 can set the exposure time as a shooting parameter for the preparation shoot according to the input from the instruction input unit 110. The shooting parameters for the main shoot may be configured to be set automatically by the imaging device 100. The shooting parameters for the main shoot may also be set independently of the shooting parameters for the preparation shoot. In this embodiment, the set shooting parameters for the main shoot are used as the shooting parameters for the main shoot after the pressing of the shutter button (instruction for the main shoot), which will be described later, is detected.

[0023] In S204, the control unit 101 determines whether motion blur notification is set to ON or OFF. The setting for ON or OFF of motion blur notification is set by the user using the instruction input unit 110, for example, and once set, the setting value is retained. If the control unit 101 determines in S204 that motion blur notification is set to ON, the motion blur notification image generation unit 300, described later, generates a motion blur notification image in S205 by superimposing a motion blur notification plane onto the prepared image. Then, in S206, the control unit 101 displays the generated motion blur notification image on the display unit 109.

[0024] If the control unit 101 determines in S204 that motion blur notification is set to OFF, no motion blur notification image is generated. Therefore, in S206, the preparatory image without any motion blur notification is displayed on the display unit 109.

[0025] In S207, the control unit 101 determines whether the shutter button of the instruction input unit 110 has been pressed down by the user. Here, if the shutter button input is configured to accept a two-stage input method, such as a half-press to instruct the shooting preparation operation and a full press to instruct the actual shooting, the control unit 101 determines whether a full press has been performed. If only a simple one-stage input is accepted, the control unit 101 determines whether that one-stage input has been performed.

[0026] If the control unit 101 determines that the shutter button is not pressed, it returns to S203 and repeats the process from S203 to S206. This allows the user to easily check the motion blur that will occur in the subject if the actual shooting is performed with the currently set shooting parameters during the preparation shooting phase, provided that the motion blur notification image is displayed on the display unit 109. If the confirmed motion blur is not to the user's liking, the user may return to S202 without pressing the shutter button and reset the shutter speed (exposure time) for the actual shooting phase again in S203. In this way, during the preparation shooting phase, the user can check the motion blur notification image displayed on the display unit 109 and repeatedly set the exposure time for the actual shooting phase until the desired motion blur is achieved, and then press the shutter button at the opportune moment to take a picture.

[0027] If the control unit 101 determines in S207 that the shutter button has been pressed, it receives a shooting instruction for the main shoot and proceeds to S208, where it controls the optical system 104, imaging unit 105, etc., to perform the main shoot based on the shooting parameters set up during the preparation shoot. The main image obtained by the main shoot is output by the control unit 101 to the display unit 109 and the recording unit 108, where it is displayed on the display device of the display unit 109 and recorded on the recording medium of the recording unit 108 or output to an external device.

[0028] Next, an example of the configuration of the motion blur notification image generation unit 300, which is included in the image processing unit 107 that is a feature of the present invention, will be described with reference to Figure 3.

[0029] Figure 3 shows an example of the configuration of the motion blur notification image generation unit 300. The motion blur notification image generation unit 300 includes a motion vector calculation unit 301 that calculates the motion vector of the subject from a comparison between images, and an estimated motion blur calculation unit 302 that estimates the motion blur of the subject during the actual shooting based on the calculated motion vector. Furthermore, it includes a notification plane generation unit 303 that creates data for notifying motion blur based on the estimated motion blur of the subject, and in this embodiment, an image superposition unit 304 that performs superposition processing to superimpose the motion blur notification plane onto the captured image. Details of the operation of each unit will be described later.

[0030] Furthermore, one or more of the functional blocks shown in Figure 3 may be implemented by hardware such as an ASIC or a programmable logic array (PLA), or by a programmable processor such as a CPU or MPU executing software. They may also be implemented by a combination of software and hardware. Therefore, even if different functional blocks are described as the primary operating components in the following explanation, the same hardware may be the primary implementing component.

[0031] Next, the process performed by the motion blur notification image generation unit 300 at S205 in Figure 2 will be explained with reference to the flowchart in Figure 4. Each step in this flowchart is executed by the control unit 101 or by each part of the imaging device 100, including the motion blur notification image generation unit 300, at the instruction of the control unit 101.

[0032] In S401, the control unit 101 sets the shooting conditions set in S202. The control unit 101 then controls the optical system 104 and the imaging unit 105, and the imaging device 100 continuously captures preparatory images to detect motion blur of the subject when performing the actual shooting.

[0033] In S402, the control unit 101 acquires shutter speed information that has been automatically set by the user or the camera in order to perform the actual shooting.

[0034] In S403, the control unit 101 selects a preparation image frame to be used to detect the subject's movement (described later in S404) from the frame rate information of the preparation shot and the shutter speed of the main shot. The method for selecting the preparation image frame will be explained in detail using Figures 10 and 11.

[0035] Figure 10 is a timing chart showing the acquisition timing of the preparatory image frame group (1001) and the main image (1011) with the horizontal axis representing time T.

[0036] The horizontal axis represents the shooting time, and 1021 in Figure 10(a) and 1022 in Figure 10(b) represent the selection of preparatory shooting image frames used when calculating the amount of motion of the subject (the amount of motion at that moment is calculated using the two frames connected by the arrow). 1021 represents the control to select the two frames before and after when detecting motion, and 1022 is an example of selecting the preparatory image frame from the frame rate information of the preparatory shooting and the shutter speed of the main shooting in this embodiment. The selection of the shooting image frame in 1021 is a method used when calculating the amount of motion of a subject in general, and in this embodiment, the shooting image frame is selected as in 1022.

[0037] The method for selecting preparation image frames in this embodiment will be explained using Figure 11. Figure 11 shows the relationship between the frame intervals of multiple preparation images (1111 to 1114) and the exposure time of the main image (1101) taken with the shutter speed of the main shooting. The following explanation will use specific numerical values. If the shutter speed set for the main shooting is set to 1 / 30, an image with an exposure time of 1 / 30 seconds will be acquired in the main shooting. In other words, if the movement of the subject in 1 / 30 seconds is small, an image with less accumulated blur can be acquired. On the other hand, preparation images are taken with a shutter speed that does not easily cause accumulated blur between frames in order to analyze the movement. For example, suppose multiple frames are taken with an exposure time shorter than 1 / 120 seconds. In this case, the imaging frame rate of the preparation images to be taken in continuous shooting is set to 120fps (continuous shooting of 120 frames per second). The control unit 101 selects frames in the preparation images where the frame interval is close to the exposure time of the main images in order to detect the movement of the subject during the exposure time of the main images. Since the intervals between adjacent preparatory image frames 1111 to 1114 are uniform, the interval of 4 frames (1111 and 1114) is 1 / 30th of a second, which is equivalent to the exposure time of the main image. In other words, it means that frames containing subject movement for a duration equivalent to 1 / 30th of the exposure time of the main image are selected. The smaller the difference between the selected frame interval of the preparatory images and the shutter speed of the main image, the smaller the error in the processing of S405 described later, and the more accurate the estimation. Therefore, the control unit 101 selects frames in which the frame interval of the preparatory images is as close as possible to the shutter speed set for the main shot. The control unit 101 selects preparatory images with frame intervals that satisfy the following equation (2). F_Period(frame interval) ≥ (shutter speed of the main shoot × frame rate of the preparation shoot) × α (F_Period is the smallest integer that satisfies the equation, and α is a constant) ... Equation (2)

[0038] By selecting preparatory image frames that satisfy equation (2) to calculate the amount of motion of the subject, it becomes possible to select frames with a time interval close to the same as the shutter speed set for the actual shooting.

[0039] The relationship between the frame interval of the preparation images and the shutter speed of the main shots has been explained using Figure 10. In this embodiment, the control unit 101 controls the frame interval of the preparation images to select the frame 1022 in Figure 10(b). Furthermore, it is desirable to set α in equation (2) to decrease as the ISO sensitivity increases. By using α, it is possible to prevent the frame interval from becoming too long when shooting conditions are severe. Also, α may be changed for each shutter speed of the main shots. For example, if the shutter speed of the main shots is long and the frame interval of the preparation images becomes too long, complex subject movements are more likely to be included between frames, which may reduce the accuracy of subject movement detection. Here, by changing α for each shutter speed of the main shots, the decrease in the accuracy of subject movement detection can be suppressed.

[0040] In S404 of Figure 4, the frame of the preparatory image selected in S403 is input to the motion blur notification image generation unit 300, and the motion vector calculation unit 301 uses the input image to calculate the amount of motion of the subject. The process of calculating the amount of motion of the subject by the motion vector calculation unit 303 will be explained using an example as shown in Figure 6(a), in which a scene of a dog 601 running to the left and a dog 602 standing still is captured.

[0041] The motion vector calculation unit 301 calculates a motion vector between sequentially occurring preparatory images to represent the amount of movement of the subject. A motion vector is a vector representation of the amount of horizontal and vertical movement of the subject between preparatory images. The method for calculating the motion vector will be explained in detail with reference to Figures 7 and 8.

[0042] Figure 7 is a flowchart showing the process for calculating motion vectors. In this invention, the block matching method is used as an example of a motion vector calculation method, but the method for calculating motion vectors is not limited to this example, and other methods such as the gradient method may also be used. Each step in this flowchart is executed by the control unit 101 or by the parts of the imaging device 100, including the motion blur notification image generation unit 300, at the instruction of the control unit 101.

[0043] In S701, the control unit 101 receives two temporally adjacent preparatory images acquired during preparatory imaging, sets the M-th frame of the preparatory imaging as the reference frame, and sets the M+1 frame of the preparatory imaging as the reference frame.

[0044] In S702, the control unit 101 arranges an N×N pixel reference block 802 in the reference frame 801, as shown in Figure 8.

[0045] In S703, as shown in Figure 8, the control unit 101 sets the search range 805 for the reference frame 803 to be the area surrounding the center coordinates of the reference block 802 of the reference frame 801 and the same coordinates 804, which is (N+n) × (N+n) pixels.

[0046] In S704, the control unit 101 controls the motion vector calculation unit 301 to perform a correlation calculation between the reference block 802 of the reference frame 801 and the reference block 806 of different coordinates (N×N pixels) located within the search range 805 of the reference frame 803, and calculates a correlation value. The correlation value is calculated based on the sum of absolute differences (SAD = Sum of Absolute Difference) between frames for the pixels of the reference block 802 and the reference block 806. In other words, the coordinate with the smallest sum of absolute differences between frames is the coordinate with the highest correlation value. Note that the method for calculating the correlation value is not limited to calculating the sum of absolute differences between frames; for example, a method of calculating the correlation value based on the sum of squared differences between frames or the normal cross-correlation value may also be used. In the example in Figure 8, it is assumed that the reference block 806 has the highest correlation.

[0047] In S705, the control unit 101 controls the motion vector calculation unit 301 and calculates a motion vector based on the reference block coordinates showing the highest correlation value obtained in S704. In the example in Figure 8, within the search range 805 of the reference frame 803, the motion vector is determined based on the same coordinate 804 corresponding to the center coordinate of the reference block 802 of the reference frame 801 and the center coordinate of the reference block 806. In other words, the distance and direction between the same coordinate 804 and the center coordinate of the reference block 806 are determined as the motion vector.

[0048] In step S706, the control unit 101 determines whether or not motion vectors have been calculated for all pixels of the reference frame 801. If it determines in S706 that motion vectors have not been calculated for all pixels, the process returns to S702. Then, in step 602, an N×N pixel reference block 802 is placed in the aforementioned reference frame 801, centered on the pixels for which motion vectors have not been calculated, and the processing from S703 to S705 is performed as described above. That is, the motion vector calculation unit 301 calculates the motion vectors for all pixels of the reference frame 801 by repeating the processing from S702 to S705 while moving the reference block 802 shown in Figure 8. An example of these motion vectors is shown in Figure 6(b). Figure 6(b) is a diagram showing an example of the motion vectors of the prepared image captured in Figure 6(a).

[0049] Figure 6(a) shows an example of a preparatory image where dog 601 is running to the left. A typical example of a motion vector when the subject is moving is shown in Figure 6(b). In the example in Figure 6(b), the running dog 601 is detected as having a motion vector to the left, while the stationary dog ​​602 and the background fence are detected as having a motion vector of 0, and therefore their motion vectors are not shown.

[0050] Furthermore, the motion vector calculation unit 301 may calculate a motion vector for each predetermined pixel rather than calculating the motion vector for all pixels. Through the above process, the motion vectors between time-adjacent frames of the prepared images are calculated.

[0051] In S405 of Figure 4, the estimated motion blur calculation unit 302 calculates the relationship between the calculated amount of motion of the subject and the target amount of motion blur. At this time, if the time between frames of the preparation image selected in S403 is not equal to the shutter speed (exposure time) of the actual shooting, the subject blur in the actual shooting cannot be calculated accurately. Therefore, in S404, the amount of motion of the subject detected is multiplied by the correction gain obtained by equation (4) to convert the amount of motion and estimate the motion blur of the actual shooting. Correction gain = Shutter speed of the main shot / Interval between frames of the selected preparatory shots... Equation (4)

[0052] Next, the control unit 101 performs a determination process to determine whether the estimated amount of motion blur in the actual shooting, i.e., the estimated amount of motion blur, which is obtained by multiplying it by the correction gain of equation (4), is greater than or less than the target amount of motion blur. The target amount of motion blur here refers to the allowable amount of motion blur that is not noticeable when shooting at a predetermined shutter speed. The allowable amount of motion blur is determined by the size of the image sensor, such as a CCD or CMOS sensor, the number of pixels, and the resolution of the display used for display. For example, if the image sensor is APS-C, the number of pixels is 200,000, and the allowable amount of motion blur is 5 pixels or less on a Full HD (1920 x 1080 pixels) PC display, then it is necessary to be able to detect movement of 5 pixels or less between frames. In other words, a resolution sufficient to detect movement of 5 pixels or less is required. A specific example will be explained using Figure 5.

[0053] Figures 5(a) and 5(b) show the relationship between the amount of subject movement calculated from the preparatory captured image and the acceptable amount of motion blur, expressed using motion vectors. Figure 5(a) shows the case where the amount of subject movement exceeds the acceptable amount of motion blur, while Figure 5(b) shows the case where the amount of subject movement is less than or equal to the acceptable amount of motion blur. The relationship between the estimated amount of subject movement and the target amount of motion blur can be expressed by equation (1). n = amount of subject movement / allowable amount of motion blur ... Equation (1)

[0054] At this time, if n is greater than 1, the control unit 101 determines that subject blur occurs because the allowable amount of motion blur has been exceeded. Conversely, if n is 1 or less, the control unit 101 determines that the allowable amount of motion blur has not been exceeded. In other words, it can be determined that the subject blur is within an acceptable range. The target allowable amount of motion blur can also be determined by the user's settings.

[0055] In S406 of Figure 4, based on instructions from the control unit 101, the idle plane generation unit 303 generates an image that notifies motion blur based on the relationship between the amount of motion of the subject and the target amount of motion blur obtained in S405. The process in S406 will be explained using Figure 9. Motion blur notification is a process that informs the user of the degree of motion blur when taking the actual shot. Therefore, the amount of motion calculated from the preparation shot image is estimated as the amount of motion (estimated amount of motion blur) when converted under the exposure conditions of the actual shot, and a blur notification image is created that allows the estimated amount of motion to be visually confirmed.

[0056] First, for each amount of motion of the subject, the notification plane generation unit 303 uses the result of equation (1) calculated in S405 to create an image plane for notifying motion blur. Then, the image superposition unit 304 superimposes the motion blur notification plane onto the prepared image to generate a motion blur notification image. An example of a blur notification image is shown in Figure 9.

[0057] Figure 9 shows three examples of motion blur notification images. By displaying the motion blur notification image on the display unit 109 during preparation shooting, the user can easily check for motion blur.

[0058] First, Figure 9(a) shows an example of notifying motion blur using an icon display. In S405, the control unit 101 calculates the ratio of the amount of motion blur of a subject where n in equation (1) is greater than 1 to the total amount of motion blur of the subject. If this ratio is greater than or equal to a predetermined ratio, the motion blur notification image generation unit 300 creates a motion blur icon 901 as shown in Figure 9(a) as a motion blur notification plane and draws it on the prepared image to generate a motion blur notification image as shown in Figure 9(a).

[0059] Figure 9(b) shows an example of notifying motion blur using motion blur frame display. Here, the method for generating a motion blur notification image using motion blur frame display will be explained. In S405, the control unit 101 divides the image into regions and calculates a number where n is greater than 1 for the amount of motion blur of the subject contained in each divided region using equation (1). That is, based on the instruction of the control unit 101, the notification plane generation unit 303 creates a motion blur frame 902 as a motion blur notification plane, as shown in Figure 9(b), for the divided regions where the proportion is greater than or equal to a predetermined proportion, and generates a motion blur notification image as shown in Figure 9(b) by drawing it on the prepared captured image.

[0060] Figure 9(c) shows an example of notifying motion blur by highlighting the edges where motion blur occurs. Here, we will explain how to generate a motion blur notification image by highlighting motion blur edges. First, in step S405, the notification plane generation unit 303 detects the edge intensity of the preparation image. An existing method such as a Sobel filter is used to calculate the edge intensity, and a detailed explanation is omitted. Then, the notification plane generation unit 303 creates a motion blur notification plane that highlights the motion blur edges, as shown in 903 of Figure 9(c), in the region where the edge intensity is greater than or equal to a predetermined value and n in equation (1) is greater than 1. The image superimposition unit 304 superimposes the created motion blur notification plane onto the preparation image to generate a motion blur notification image like the one in Figure 9(c). In the example of 903 in Figure 9(c), an example of making the motion blur edges thicker is shown. By highlighting such motion blur edges, it becomes possible to visually identify motion blur in the actual shooting for fine details of the subject. Other examples of highlighting methods include extracting pixels where the edge strength is above a predetermined value and the estimated motion blur is above a predetermined value, and then highlighting the extracted pixels by changing their hue, saturation, and brightness, such as by making them red.

[0061] The process by which the motion blur notification image generation unit 300 generates a motion blur notification image by performing the processes from S401 to S406 in Figure 4 has been explained. By having the control unit 101 or each part of the imaging device 100, including the motion blur notification image generation unit 300, perform the flowchart in Figure 2 at the instruction of the control unit 101, it is possible to suitably determine the shooting conditions for the preparatory shooting to check for motion blur that occurs in the main shooting.

[0062] According to the present invention, by appropriately selecting preparatory images for checking motion blur that occurs during the main shooting and detecting the motion, it is possible to confirm appropriate motion blur that occurs during the main shooting.

[0063] In this invention, we have described a control method for detecting motion by selecting frames according to the frame intervals of the preparatory images in order to estimate the amount of subject blur that occurs during the exposure time of the main shooting. To detect motion in more detail, it is also possible to detect the amount of subject motion using all preparatory images contained within the frame intervals of the preparatory images corresponding to the exposure time of the main shooting image. For example, as explained in Figures 11 and 12, in the case of the exposure time of the main shooting (1 / 30 second) and the frame rate of the preparatory shooting of 120fps as explained in Figure 11, the preparatory images will have a 1 / 30 second interval with 4 frames. Therefore, in Example 1, the motion vector calculation unit 301 calculated the amount of subject motion using 1111 and 1114 in Figure 11 (1201 in Figure 12). Here, the control unit 101 integrates the amount of subject motion obtained from 1111 and 1112, 1112 and 1113, and 1113 and 1114 respectively (1202 in Figure 12). This allows for accurate calculation of the subject's movement during 1 / 30 second (from 1111 to 1114) even in pre-extracted images, when the subject moves in a complex manner during that 1 / 30 second period (such as rotational motion or accelerated motion).

[0064] Furthermore, if the photographer has not intentionally determined the acceptable amount of motion blur in advance, the magnitude of the motion vector within the range that does not exceed the limit range of motion detection may be set as the acceptable amount of motion blur. In this case, the control unit 101 determines whether the movement of the subject has exceeded the range of motion detection. This determination can be made by checking whether the change in the SAD value at the time of motion detection is smaller than a predetermined value, or by measuring the movement of the imaging device 100 itself, such as gyro information. If the control unit 101 determines that the acceptable amount of motion blur has been exceeded, it controls the shutter speed and frame rate to increase.

[0065] In this invention, an example of notifying motion blur has been described in which a display unit 109 is used to display motion blur, but this is not the only method of notifying motion blur. For example, motion blur may be notified by sound, light, or vibration. Specifically, when the proportion of pixels showing estimated motion blur of a predetermined value or more among the estimated motion blur of each pixel to the entire screen is a predetermined proportion or more, a motion blur notification sound, notification light, or notification vibration is generated. In this embodiment, the configuration of the notification plane generation unit 303 and the image superposition unit 304, and the processing flow of steps S405 and S406 are unnecessary. Instead, a speaker may be provided in the imaging device 100, and in step S205, the control unit 101 may sound a notification sound through the speaker, light up a notification light, or generate a notification vibration in conjunction with the display of the prepared captured image on the display unit 109.

[0066] In the present invention, the control unit 101 changed the selection of frames for motion detection from the preparatory image according to the shutter speed of the main shot. However, the selection of frames in the preparatory image for motion detection may be changed only when the shutter speed of the main shot is stable. By controlling the frames of the preparatory image for motion detection only when the shutter speed of the main shot is stable, it is possible to prevent unwanted motion blur notification images from being displayed on the display unit 109 when the user is unsure about controlling the shutter speed or when the evaluation value calculated by the control unit 101 is unstable.

[0067] One way to determine whether a subject's movement is stable is to change the shooting conditions during the initial setup and then measure the change in the motion vector to determine if the subject's movement is stable. If the change in the motion vector per unit time becomes large, it is possible to change the shooting conditions during the initial setup again.

[0068] In the present invention, the control unit 101 changed the selection of frames for detecting motion from the preparatory image according to the shutter speed of the main shot. However, it is also possible to change the shutter speed of the preparatory shot in addition to selecting the frames for the preparatory shot. For example, if the shutter speed of the main shot becomes longer than a predetermined value, and the ISO sensitivity of the main shot is higher than a predetermined value, the shutter speed of the preparatory shot is changed to be longer. Conversely, if the ISO sensitivity of the main shot is low, the shutter speed of the preparatory shot is changed to be shorter. This makes it possible to capture an appropriate preparatory image for detecting motion by considering the shooting conditions of the preparatory shot for motion detection as a trade-off between the effect of noise and accumulated blur. [Examples]

[0069] Next, we will explain an example in which the amount of subject blur in the main shoot is estimated and a motion blur notification image is generated by changing the frame rate of the preparation shoot to match the shutter speed of the main shoot. By implementing this embodiment, it becomes possible to efficiently control the frame rate of the preparation shoot image in order to provide subject blur notification for the main shoot.

[0070] In this embodiment, the same configuration of the imaging device 100 as in Figure 1 of Embodiment 1 can be used, so the explanation of the block diagram of the imaging device 100 is omitted. Also, the overall processing performed by the control unit 101 in Embodiment 2 is the same as the processing shown in the flowchart of Figure 2, so the explanation is omitted.

[0071] Next, the processes S202 and S205 in Figure 2, which are features of the present invention, will be described in detail.

[0072] In the S202 process, the control unit 101 changes the frame rate of the preparation shot to match the shutter speed of the main shot. This will be explained using Figure 13.

[0073] Figure 13(a) shows a timing chart for general preparation and main shooting, and (b) shows an example in Embodiment 2 where the frame rate of the preparation shooting is changed to match the shutter speed of the main shooting. As an example, the shutter speed of the main shooting is 1 / 30 second, and the frame rate of the preparation shooting before changing the frame rate as shown in Figure 13(a) is 120 fps. In this case, the main shooting image 1311 will have an exposure time of 1 / 30 second because the shutter speed is 1 / 30 second. In this embodiment, the control unit 101 changes the frame rate of the preparation shooting to approach the exposure time of the main shooting image, as shown in Figure 13(b). For example, when the frame rate of the preparation shooting is 120 fps, the interval between adjacent frames is 1 / 120 second. Here, if the exposure time of the main shooting image is 1 / 30 second, the control unit 101 changes the frame rate of the preparation shooting from 120 fps to 30 fps. In other words, the frame rate of the preparation shooting is given by the following equation (3). Frame rate after preparation shooting = β / shutter speed of the main shoot ... Equation (3) (β is a constant)

[0074] β is a predetermined constant that changes depending on the ISO sensitivity and shutter speed of the main shooting, similar to α in equation (2) of Example 1. The control unit 101 can set the frame interval to an appropriate level for detecting subject blur at the shutter speed of the main shooting by changing the frame rate of the preparation shooting to satisfy the above equation (3). In Figure 13(b) after changing the frame rate, comparing the case where the movement of the subject is detected using adjacent frames when preparation shooting is performed with the frame rate of Figure 13(a) with the frame interval (1321) in Figure 13(b), it can be seen that the frame interval of the preparation shooting (1321) has widened, and the time interval is closer to the exposure time of the main shooting (1311). Therefore, the movement of the subject equivalent to the exposure time of the main shooting (1311) can be estimated by controlling the frame rate of the preparation shooting as shown in Figure 13(b). Note that the frame rate does not have to match the exposure time of the main shooting, and the frame rate may be changed to a value greater than or equal to the exposure time of the main shooting.

[0075] Next, the process of S205 will be explained using the flowchart in Figure 14. Each step in this flowchart is executed by the control unit 101 or by the parts of the imaging device 100, including the motion blur notification image generation unit 300, at the instruction of the control unit 101.

[0076] In S1401, the control unit 101 performs the same processing as in S401 to set the shooting conditions. Then, the control unit 101 controls the optical system 104 and the imaging unit 105, and the imaging device 100 continuously captures preparatory images.

[0077] In S1402, the control unit 101 performs the same processing as in S402.

[0078] In S1403, the control unit 101 performs the same processing as in S404, and the motion vector calculation unit 301 calculates the amount of motion of the subject from the preparation image captured in S1401.

[0079] In S1404, the estimated motion blur calculation unit 302 performs the same processing as in S405, and the imaging device 100 calculates the relationship between the calculated amount of motion of the subject and the target amount of motion blur.

[0080] In S1305, the control unit 101 performs the same processing as in S406 in Figure 4. Based on the estimated amount of motion blur of the captured image processed in S1404, the notification plane generation unit 303 creates an image plane for notifying motion blur, and the image superposition unit 304 superimposes the motion blur notification plane onto the prepared captured image to generate a motion blur notification image.

[0081] Regarding the overall processing of the flowchart in Figure 2 controlled by the control unit 101 of the above embodiment 2, the characteristic processes S202 and S205 were specifically explained.

[0082] By implementing the configuration of Embodiment 2, the control unit 101 can control the frame rate of the preparatory shots used to detect the subject's movement, which affects the exposure time of the main shot. Therefore, it becomes possible to accurately detect motion blur that occurs during the main shot. Furthermore, since it is no longer necessary to continuously capture preparatory images at a high frame rate to detect the subject's movement, accurate motion detection can be achieved with low power consumption.

[0083] In this invention, the control unit 101 changes the selection of frames from which motion is detected from the preparation images according to the shutter speed of the main shot. However, if the shutter speed of the main shot is greater than the maximum frame rate that can be set in the imaging device 100, the frame rate in the preparation shot may be fixed at the maximum value, and the preparation image with the closest frame interval may be selected to detect motion. In this condition, almost no motion blur occurs during the main shot, so it is not necessary to estimate motion blur. Thus, if the shutter speed of the main shot is greater than a certain value, that is, if the exposure time of the main shot is shorter than a certain time, the frame rate may be fixed to a specific value.

[0084] Alternatively, a threshold can be set for the shutter speed of the main shot, and the control unit can control the frame rate during the preparation shot according to that value. For example, a first threshold and a second threshold greater than the first threshold can be set for the shutter speed of the main shot. In this case, if the shutter speed of the main shot is less than the first threshold, the control unit 101 controls at the first frame rate. If the shutter speed of the main shot is greater than or equal to the first threshold and less than the second threshold, the control unit 101 may control at the second frame rate, and if the shutter speed of the main shot is greater than or equal to the second threshold, it may control at the third frame rate.

[0085] This invention describes an example in which motion blur is notified when the estimated motion blur exceeds a predetermined value. However, it can also be applied to cases where multiple motion blur displays are performed depending on the magnitude of the motion blur. For example, it is conceivable that the estimated magnitude of motion blur can be represented by three colors (red, blue, and yellow). In that case, the allowable motion amount is set to the narrowest range of motion that can be detected so that the range of motion magnitudes represented by each color can be expressed. For example, if the motion blur representation colors for the magnitude of the vector are yellow: 0-10 pixels, blue: 11-15 pixels, and red: 16 pixels or more, the blue range represents the narrowest interval of motion. In other words, this can be achieved by setting the 5 pixels in the blue range (11-15) as the allowable motion blur amount.

[0086] In this invention, an acceptable amount of motion blur is set, and the frame rate, which is the shooting condition for the preparatory image, is controlled so that motion below the acceptable amount of motion blur can be detected. However, the shooting conditions may also be changed if the detected motion blur is below a predetermined value (e.g., sports priority mode).

[0087] The shooting conditions for the preparatory images may be changed when it is detected that the subject's movement is stable. Alternatively, after changing the shooting conditions for the preparatory images once, the stability of the subject's movement can be determined by the change in the motion vector, and if the change in the motion vector per unit time becomes large, the shooting conditions for the preparatory images may be changed again.

[0088] If it is not possible to set an acceptable amount of motion blur in advance, the magnitude of the motion vector within the range that does not exceed the limit range of motion detection may be set as the acceptable amount of motion. In this case, the control unit 101 determines whether the movement of the subject has exceeded the range of motion detection. This determination can be made by checking whether the change in the SAD value at the time of motion detection is smaller than a predetermined value, or by measuring the movement of the imaging device 100 itself, such as gyro information. If the control unit 101 determines that the acceptable amount of motion blur has been exceeded, it controls the shutter speed and frame rate to increase.

[0089] In this invention, an example of notifying motion blur has been described in which a display unit 109 is used to display motion blur, but this is not the only method for notifying motion blur. For example, motion blur may be notified by sound, light, or vibration. Specifically, a motion blur notification sound may be generated when the proportion of pixels showing estimated motion blur of a predetermined value or more among the estimated motion blur of each pixel to the entire screen is a predetermined proportion or more. In that case, a means for generating a blur notification sound, such as a microphone, should be provided as a means for notifying motion blur.

[0090] (Other embodiments) Although the present invention has been described above based on embodiments, the present invention is not limited to these embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention.

[0091] The object of the present invention can also be achieved as follows: a storage medium containing program code for software describing the procedures for realizing the functions of each embodiment described above is supplied to a system or device. The computer (or CPU, MPU, etc.) of the system or device then reads and executes the program code stored on the storage medium.

[0092] In this case, the program code read from the storage medium itself realizes the novel function of the present invention, and the storage medium and program that store that program code constitute the present invention.

[0093] Furthermore, storage media for supplying program code include, for example, flexible disks, hard disks, optical disks, and magneto-optical disks. CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-Rs, magnetic tapes, non-volatile memory cards, and ROMs can also be used.

[0094] Furthermore, the functions of each of the embodiments described above are realized by making the program code read by the computer executable. In addition, this also includes cases in which the OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program code, and the functions of each of the embodiments described above are realized through that processing.

[0095] Furthermore, the following cases are also included: First, program code read from a storage medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer. Then, based on the instructions of that program code, the CPU or other components of that function expansion board or function expansion unit perform some or all of the actual processing. [Explanation of symbols]

[0096] 100 Imaging device 101 Control Unit 102 ROM 103 RAM 104 Optical system 105 Imaging Unit 106 A / D Conversion Unit 107 Image Processing Unit 108 Records Section 109 Display section 110 Instruction Input Section

Claims

1. A setting means for setting a first shooting parameter and a second shooting parameter; an acquisition means for acquiring a plurality of first photographed images obtained by performing a first photographing operation with the first photographing parameters and motion information of a subject in the plurality of first photographed images; an estimation means for estimating, based on the second shooting parameters and the motion information, a motion blur of a subject in a second captured image obtained by performing a second shooting with the second shooting parameters; The setting means changes a frame rate included in the first photographing parameter based on an exposure time of the second photographing related to the second photographing parameter, The information processing apparatus according to claim 1, wherein the acquisition means acquires the movement information from the plurality of first photographed images based on the changed first photographing parameters.

2. 2 . The information processing apparatus according to claim 1 , wherein the acquisition means changes the frame rate to a value equal to or greater than an exposure time in the second shooting.

3. 2 . The information processing apparatus according to claim 1 , wherein the acquisition means acquires the motion information from two chronologically consecutive images out of the plurality of first images acquired at the changed frame rate.

4. Further comprising: a notification means for notifying information about the motion blur; 2 . The information processing apparatus according to claim 1 , wherein the notification means notifies the user of the motion blur by displaying information corresponding to the motion blur on a display unit.

5. 5 . The information processing apparatus according to claim 4 , wherein the notification means notifies the user of the motion blur by displaying information corresponding to the motion blur on a display unit in a manner superimposed on the first captured image.

6. 6. The information processing apparatus according to claim 4, wherein, when the estimating means estimates a motion blur larger than a predetermined value, the notifying means notifies the user of the motion blur.

7. The notification means includes:

7. The information processing apparatus according to claim 6, wherein the motion blur is notified by displaying an image of an icon on the display unit.

8. The notification means includes:

7. The information processing apparatus according to claim 6, wherein the motion blur is notified by displaying an image in which a motion blur frame is drawn on the display unit.

9. The notification means includes:

7. The information processing apparatus according to claim 6, wherein the display unit displays an image in which edges of motion blur are emphasized, thereby notifying the user of motion blur.

10. 6. The information processing apparatus according to claim 4, wherein when the notification by said notification means is OFF, said notification means does not notify said motion blur.

11. 2 . The information processing apparatus according to claim 1 , wherein the first photographing is a preparatory photographing for capturing an image at a time different from that of the second photographing.

12. 12. The information processing apparatus according to claim 11, wherein the second photographing is actual photographing, and the first photographing is preparatory photographing carried out before the actual photographing.

13. A motion blur notification sound generating means is provided for generating a motion blur notification sound, 2. The information processing apparatus according to claim 1, wherein said motion blur warning means warns of motion blur by sound.

14. A setting means for setting a first shooting parameter and a second shooting parameter; an acquisition means for acquiring a plurality of first photographed images obtained by performing a first photographing operation with the first photographing parameters and motion information of a subject in the plurality of first photographed images; an estimation means for estimating, based on the second shooting parameters and the motion information, a motion blur of a subject in a second captured image obtained by performing a second shooting with the second shooting parameters; The information processing device according to claim 1, wherein the acquisition means determines an image to be used for acquiring the motion information from the plurality of first captured images based on an exposure time in the second shooting associated with the second shooting parameter.

15. The information processing device according to claim 14, characterized in that the acquisition means determines an image to be used to acquire the motion information from the plurality of first images depending on a difference between a frame rate, which is one of the first shooting parameters, and an exposure time value in the second shooting.

16. 16 . The information processing apparatus according to claim 15 , wherein the acquisition means selects two images from the plurality of first images so that the time interval between the two images corresponds to an exposure time in the second photographing.

17. A notification means for notifying information about the motion blur, 15. The information processing apparatus according to claim 14, wherein the notification means notifies the user of the motion blur by displaying information corresponding to the motion blur on a display unit.

18. 18. The information processing apparatus according to claim 17, wherein the notification means notifies the user of the motion blur by displaying information corresponding to the motion blur on a display unit in a manner superimposed on the first captured image.

19. 18. The information processing apparatus according to claim 17, wherein, when the estimating means estimates a motion blur larger than a predetermined value, the notifying means notifies the user of the motion blur.

20. The notification means includes:

20. The information processing apparatus according to claim 19, wherein the motion blur is notified by displaying an image of an icon on the display unit.

21. The notification means includes:

20. The information processing apparatus according to claim 19, wherein the motion blur is notified by displaying an image in which a motion blur frame is drawn on the display unit.

22. The notification means includes:

20. The information processing apparatus according to claim 19, wherein the display unit displays an image in which edges of motion blur are emphasized, thereby notifying the user of motion blur.

23. 18. The information processing apparatus according to claim 17, wherein when the notification by said notification means is OFF, said notification means does not notify said motion blur.

24. 18. The information processing apparatus according to claim 17, wherein the first photographing is a preparatory photographing for photographing an image at a time different from that of the second photographing.

25. 25. The information processing apparatus according to claim 24, wherein the second photographing is actual photographing, and the first photographing is preparatory photographing carried out before the actual photographing.

26. A motion blur notification sound generating means is provided for generating a motion blur notification sound, 18. The information processing apparatus according to claim 17, wherein the motion blur warning means warns of motion blur by sound.

27. an imaging means for capturing an object image formed via an optical system and outputting the image; The information processing device according to claim 1 ; An imaging device comprising:

28. an imaging means for capturing an object image formed via an optical system and outputting the image; An information processing device according to claim 14; An imaging device comprising:

29. A setting step of setting a first shooting parameter and a second shooting parameter; an acquiring step of acquiring a plurality of first photographed images obtained by performing a first photographing operation with the first photographing parameters and motion information of a subject in the plurality of first photographed images; an estimation step of estimating a motion blur of a subject in a second captured image obtained by performing a second photographing operation with the second photographing parameters, based on the second photographing parameters and the motion information; having In the setting step, a frame rate included in the first photographing parameter is changed based on an exposure time of the second photographing related to the second photographing parameter; A control method for an image processing apparatus, comprising: acquiring the motion information from the plurality of first photographed images based on the first photographing parameters changed in the acquiring step.

30. A setting process for setting a first shooting parameter and a second shooting parameter; an acquiring step of acquiring a plurality of first photographed images obtained by performing a first photographing operation with the first photographing parameters and motion information of a subject in the plurality of first photographed images; an estimation step of estimating a motion blur of a subject in a second captured image obtained by performing a second photographing operation with the second photographing parameters, based on the second photographing parameters and the motion information; having A control method for an information processing device, characterized in that in the acquisition process, an image to be used to acquire the movement information is determined from the plurality of first captured images based on the exposure time in the second shooting related to the second shooting parameter.

31. A program for causing a computer to execute each step of the control method according to claim 29 or 30.

32. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 31.

Images