Photography method

Abstract

To provide an imaging method that detects a change in the direction in which eyes close or open as a subject blinks, and can capture an image of the subject with suitable shutter timing.SOLUTION: The imaging method of an imaging apparatus that includes imaging elements that output image data sequentially by photoelectric conversion of a subject image formed through an optical system includes the steps of capturing image data, calculating the full eye size, calculating the reference eye area, setting readout conditions, storing imaging conditions, calculating the capturing timing, determining the capturing timing, and providing capturing instructions.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a photographing method capable of detecting the blink of a subject's eyes and performing imaging at a suitable timing.

Background Art

[0002] Conventionally, a photographing method and an imaging device have been disclosed in which a face region of a subject is detected from through-image data, an eye image of the subject is detected from the face region, it is determined whether the line of sight of the subject is directed toward the camera from the obtained eye image, and photographing is immediately performed when the subject is determined to be in the camera line of sight.

[0003] In Patent Document 1, an imaging device that determines photographing timing based on line-of-sight transition information indicating the transition of the line of sight of a subject is disclosed.

[0004] According to the invention disclosed in Patent Document 1, it is possible to automatically photograph an image in the camera line of sight even for a subject such as an infant who is difficult to align with the camera line of sight, and it is possible to avoid missing a shutter chance.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in Patent Document 1, even if the photographed image of the subject is in the camera line of sight, there is a possibility that due to the subject blinking at the timing of the camera line of sight, image data in which the eyes are closed or not fully open (the opening and closing state of the eyes is not as desired by the photographer) may be recorded.

[0007] Generally, the time it takes for a human to blink (close and then open their eyes) is said to be approximately 100 to 200 milliseconds. In an imaging device, in order to detect a change in the direction of eye closing or opening due to a subject's blink from through-image data, it is necessary to read out the through-image data in a time shorter than the time it takes to blink.

[0008] This invention has been made in view of the above circumstances, and aims to provide a shooting method that can detect changes in the direction in which the subject's eyes close or open due to blinking, and photograph the subject at a suitable shutter timing. [Means for solving the problem]

[0009] To solve the aforementioned problems, the first imaging method of the present invention is an imaging method for an imaging device equipped with an image sensor that sequentially outputs continuous image data by photoelectric conversion of a subject image formed through an optical system, and comprises an image data acquisition step for acquiring image data; an eye full-open size calculation step for calculating the full-open size of the subject's eyes from the image data; a reference eye area determination step for determining a reference eye area on the image sensor using the full-open size of the eyes; a readout condition setting step for setting readout conditions for the reference eye area; an imaging condition storage step for storing the full-open size of the eyes, the reference eye area, and the readout conditions; an imaging timing calculation step for acquiring the state of the eye area on the sequentially output image data obtained by reading out the reference eye area; an imaging timing determination step for determining whether the size of the eye area on the sequentially output image data obtained by reading out the reference eye area from the state acquired in the imaging timing calculation step will satisfy the full-open size of the eyes within a certain period of time; and an imaging instruction step for instructing the image sensor to read out all pixels when it is determined in the imaging timing determination step that the size of the eye area on the sequentially output image data obtained by reading out the reference eye area will satisfy the full-open size of the eyes within a certain period of time.

[0010] The second invention of the present invention relates to a method for capturing images, characterized in that, in the readout condition setting step, when the frame rate for reading the reference image area is less than a predetermined frame rate, a decimation readout process is set for the reference image area to be equal to or greater than the predetermined frame rate.

[0011] The third invention of the present invention is characterized in that, in the shooting timing calculation step, the amount and rate of change of the size of the eye region on sequentially output image data are detected, and the blinking time and blinking direction are calculated from the amount and rate of change of the size of the eye region on sequentially output image data.

[0012] The fourth invention of the present invention is characterized in that, in the shooting timing determination step, it is determined whether the size of the eye region on the image data to be output sequentially in the next frame is equal to the fully open size of the eye, using the rate of change of the size of the eye region on the sequentially output image data and the blinking direction. [Effects of the Invention]

[0013] The present invention provides a shooting method that prevents the recording of image data in which the eyes are closed or not fully open due to blinking, and allows for the capture of the subject at a more suitable shutter timing. [Brief explanation of the drawing]

[0014] [Figure 1] This is a block diagram showing the main components of an imaging device, which is one embodiment of the present invention. [Figure 2] This is a flowchart illustrating each step of the imaging operation of an imaging device, which is one embodiment of the present invention. [Figure 3] This is a flowchart explaining the process for calculating shooting conditions. [Figure 4] This is a flowchart explaining the process for calculating the shooting timing. [Modes for carrying out the invention]

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited by these embodiments.

[0016] FIG. 1 is a block diagram showing the main configuration of the imaging device 100 of the present embodiment. The imaging device 100 includes a camera body 110 and a lens optical system 120 included in an interchangeable lens that is detachable from the camera body 110.

[0017] The camera body 110 includes an imaging element 130, a signal processing unit 141, an image processing unit 142, a CPU 150, a ROM 161, a RAM 162, a recording medium interface 170, a recording medium 171, a user interface 180, and an image display unit 190.

[0018] The imaging element 130 receives light rays from a subject condensed by the lens optical system 120, performs photoelectric conversion, and outputs an image signal of the subject image. In the present embodiment, a CMOS image sensor is used for the imaging element 130. The light receiving surface of the imaging element 130 is composed of a large number of pixels.

[0019] Furthermore, the imaging element 130 includes a buffer memory (not shown) inside, is connected to the CPU 150, receives a signal for determining the horizontal drive and vertical drive timings for each pixel of the imaging element 130 output from the CPU 150, and performs control.

[0020] The signal processing unit 141 performs various processes on the image signal output from the imaging element 130 and outputs it as image data. Also, the image signals continuously output from the imaging element 130 are acquired as continuous image data. The continuous image data is acquired as through-image data.

[0021] The image processing unit 142 performs various image processes on the image data input from the signal processing unit 141. Examples of the image processing include white balance correction, shading correction, and the like.

[0022] The CPU 150 performs comprehensive control of the entire imaging device 100. For example, it performs read control of the imaging element 130. By outputting a signal for the CPU 150 to determine the driving timing of the imaging element 130, the horizontal drive and vertical drive for each pixel are controlled, and an image signal is read from each pixel.

[0023] The ROM 161 stores various programs executed by the CPU 150, various characteristic information, setting information, etc. used by the CPU 150 to execute processing. As the characteristic information, for example, ratio data of the sizes of a pre-determined face region and eye region is included.

[0024] The RAM 162 is used as a work area when the CPU 150 executes processing according to a program. Also, it can temporarily store detection results and calculation results that each processing unit needs to read out at high speed.

[0025] The recording medium interface 170 writes and reads image data to and from the recording medium 171. This recording medium 171 is a detachable recording medium such as a semiconductor memory.

[0026] The user interface 180 has operation members such as a release button, power button, command dial, cross keys, etc. When the user operates these operation members, the CPU 150 issues an instruction to perform corresponding operations.

[0027] The image display unit 190 displays through image data, image data read from the recording medium 171, etc.

[0028] The lens optical system 120 is composed of a plurality of lens groups (not shown) including a focus lens group and a zoom lens group. In FIG. 1, it is described as one lens as an example. Also, it may be a single-focus lens without a zoom lens.

[0029] Figure 2 is a flowchart illustrating the processing flow of the CPU 150 in the camera body 110 of the imaging device according to the present invention, from the time the blink detection mode is set until all pixels are read out.

[0030] First, the user performs a blink detection mode setting on the camera body 110, a signal is input from the user interface 180 to the CPU 150, and this flowchart begins.

[0031] In step S001, the signal processing unit 142 outputs through image data of the subject, and each processing unit in the CPU 150 that is involved in calculating the shooting conditions detects the subject's face information on the through image data, calculates the size of the face region from the detected face information, calculates the fully open eye size from the calculated face region size, calculates the reference eye region on the image sensor 130 from the calculated fully open eye size, sets the reading conditions for the reference eye region, and stores this information in the RAM 162.

[0032] Here, using Figure 3, we will explain the processing steps of the face detection unit 201, face region calculation unit 202, eye full-open size calculation unit 203, reference eye region determination unit 204, readout condition setting unit 205, user notification unit 206, and shooting condition storage unit 207 within the CPU 150 in the calculation of shooting conditions in step S001.

[0033] In step S101, the face detection unit 201 detects the subject's face information from the through-image data sequentially output by the signal processing unit 141. The specific method for face detection in this invention can be any known method, such as detecting parts like the eyes, mouth, and nose, or detecting skin-colored areas. Face information is detected by finding the skin-colored area and eye area as coordinates on the through-image data. The coordinates on the through-image data are the coordinates of the boundary portions of the rectangular areas surrounding the skin-colored area and the eye area, respectively. If no face information is detected from the through-image data, face detection is repeated. If face information is detected, the process proceeds to step S102.

[0034] In step S102, the face area calculation unit 202 calculates the size of the face area from the face information detected in step S101. The size of the face area is calculated from the coordinates of the boundary portion of the rectangular area surrounding the skin color area on the through-image data in terms of the pixel size in the horizontal direction and the pixel size in the vertical direction for the internal area, respectively.

[0035] In step S103, the fully open eye size calculation unit 203 calculates the fully open eye size from the size of the face area calculated in step S102. The method for calculating the fully open eye size in the present invention uses the ratio data of the sizes of the predetermined face area and eye area stored in the ROM 161, and calculates the size of the eye area in the state where the eye area is maximized (the state where the eyelids are fully open) in the size of the face area calculated in step S102, and sets it as the fully open eye size. Thereby, it becomes possible to calculate the size in the state where the eyes are fully open regardless of the expression of the subject at the time of face detection. The fully open eye size is represented by the pixel size in the vertical direction and the pixel size in the horizontal direction.

[0036] In step S104, the reference eye area determination unit 204 determines the eye area on the imaging device 130 from the fully open eye size calculated in step S103. The eye area on the imaging device 130 determined here is defined as the reference eye area. The reference eye area is defined as the area of the horizontal pixels and vertical pixels on the imaging device 130 corresponding to the area where the center of the fully open eye size calculated in step S103 is aligned with the center of the coordinate information of the eye area on the through-image data detected in step S101.

[0037] In step S105, the read condition setting unit 205 determines whether the frame rate F (unit: fps) for reading the reference eye area on the imaging device 130 determined in step S104 is less than or greater than the required frame rate F' (unit: fps). If the frame rate F is less than the required frame rate F', that is, when F < F', the read condition is set so that the frame rate F becomes equal to or higher than the required frame rate F' by performing the decimation read processing on the reference eye area.

[0038] In this embodiment, in order to detect the blinking direction in the step of calculating the shooting timing described later, the initial value of the required frame rate F' is set so that at least 3 frames can be acquired within the time it takes for the eye to go from a fully open state to a fully closed state due to the blinking motion (the motion of closing and then opening the eye). For example, if the time taken for the blinking motion (the motion of closing and then opening the eye) is 100 msec and the time it takes for the eye to go from a fully open state to a fully closed state due to the blinking motion is 50 msec, then the required frame rate F' is approximately 60 fps. By setting the frame rate F to 60 fps or higher, it becomes possible to acquire 3 frames within the time it takes for the eye to go from a fully open state to a fully closed state due to the blinking motion. Since it is possible to acquire 3 frames within the time it takes for the eye to go from a fully open state to a fully closed state due to the blinking motion, it becomes possible to acquire 2 frames during the movement of the eyelid in the direction of closing or opening the eye due to the blinking motion, regardless of the blinking motion period and the acquisition period of the through image data, and the blinking direction can be calculated from the amount of change in eye size in the 2 acquired frames. By setting the frame rate F as high as possible, the number of frames that can be acquired during blinking can be increased, which improves the accuracy of estimating the shooting timing in the shooting timing determination step described later.

[0039] The settings for the decimation readout process include, for example, speeding up the readout of the reference area on the image sensor 130 by decimating one pixel at a time in the horizontal and vertical directions for each pixel, or by decimating one row and one column at a time in the horizontal and vertical directions for each pixel, so that the frame rate F is equal to or greater than the required frame rate F'. Note that the decimation interval for each pixel does not need to be constant.

[0040] In step S106, the user notification unit 206 displays the face region and reference eye region detected and calculated in steps S101 to S104 on the through image data displayed on the image display unit 190, overlaying each region with a frame indicating its position.

[0041] In step S107, the shooting condition memory unit 207 stores the fully open eye size, reference eye area, and readout conditions calculated in steps S103 to S105 in the RAM 162. Once storage is complete, the process proceeds to step S002 in Figure 2.

[0042] In step S002, the release determination unit 208 determines whether or not the release button, which is one of the user interface 180, was pressed by the user before reaching step S002.

[0043] If the release button, which is one of the user interface 180, has not been pressed by the user before reaching step S002, the process returns to step S001 and the flowchart is executed.

[0044] If the release button, which is one of the user interface 180, is pressed by the user before reaching step S002, the process proceeds to step S003.

[0045] In step S003, the shooting condition setting unit 301 obtains the fully open eye size calculated by the eye fully open size calculation unit 203 stored in the RAM 162 in step S001, the reference eye region on the image sensor 130 determined by the reference eye region determination unit 204, the setting information for the decimation readout process which is the readout condition set by the readout condition setting unit 205, and the frame rate F for reading the reference eye region on the image sensor 130.

[0046] In step S004, the shooting condition determination unit 302 determines whether the eye size, blinking direction, and gaze direction from the through image data described later have been acquired by the RAM 162. If they have not been acquired, the process proceeds to step S005; if they have been acquired, the process proceeds to step S006.

[0047] In step S005, the signal processing unit 141, which obtains the reference eye region on the image sensor 130, outputs through image data, and each processing unit in the CPU 150 that is involved in calculating the shooting timing acquires the state of the eye region on the through image data necessary for determining the shooting timing in step S006.

[0048] Here, using Figure 4, the processing steps of the eye region reading unit 303, eye size detection unit 304, blink time calculation unit 305, frame rate determination unit 306, frame rate change unit 307, blink direction detection unit 308, gaze detection unit 309, and condition saving unit 310 within the CPU 150 in the calculation of the shooting timing in step S005 will be explained.

[0049] In step S501, the eye region readout unit 303 reads out the reference eye region on the image sensor 130 acquired in step S003 at frame rate F. Once the eye region readout unit 303 reads out the reference eye region on the image sensor 130 at frame rate F, the signal processing unit 141 outputs the through image data of the reference eye region on the image sensor 130 at frame rate F.

[0050] In step S502, the eye size detection unit 304 detects the coordinates of the eye region on the through image data sequentially output at frame rate F in step S501, and detects the size of the eye region and the position information of the pupil within the eye region. The size of the eye region is detected by taking the coordinates of the boundary portion of the rectangular area surrounding the eye region on the through image data and determining the internal area in terms of horizontal and vertical pixel sizes. The position information of the pupil is detected as coordinate data of the boundary between the white of the eye region and the pupil region on the through image data. If the decimation readout setting for the reference eye region on the image sensor 130 is enabled, the size of the eye region is calculated by adding the pixel size on the through image data corresponding to the decimated pixels.

[0051] In step S503, the blink time calculation unit 305 detects the change in size of the eye region on the current through-image data detected by the eye size detection unit 304 in step S502 and the change rate between the size of the eye region on the through-image data detected by the eye size detection unit 304 in the previous frame stored in RAM 162, and calculates the blink time. If the size of the eye region on the through-image data detected by the eye size detection unit 304 in the previous frame is not stored in RAM 162, the process proceeds to step S504 without calculating the blink time. The blink time calculation method involves calculating the time taken for the actual blinking action (the action of closing and opening the eyes) from the fully open eye size calculated by the eye fully open size calculation unit 203, the change in the size of the eye region on the through-image data output sequentially at the frame rate F detected by the blink time calculation unit 305, and the inter-frame time, and then calculating the time it takes for the eyes to go from a fully open state to a fully closed state due to the blinking action. For example, if the vertical pixel size of the fully open eye is 100px, the vertical pixel size of the eye area in the current frame's through image data is 50px, the vertical pixel size of the eye area in the previous frame's through image data is 10px, and the frame rate F is 60fps, then the change is 40px during the 16msec interval between frames. Therefore, the time taken for blinking (the action of closing and then opening the eye) is 80msec, and the time taken for the eye to go from fully open to fully closed due to blinking is 40msec.

[0052] In step S504, the frame rate determination unit 306 determines whether at least 3 frames can be acquired at a frame rate F that reads the reference eye area set by the read condition setting unit 205 within the time it takes for the eye to go from fully open to fully closed due to the blinking motion calculated in step S503. If acquisition is possible or if the blinking time was not calculated in step S503, the process proceeds to step S506. If acquisition is not possible, the process proceeds to step S505.

[0053] In step S105, the frame rate F is set to 60fps or higher, because the required frame rate F' is approximately 60fps, assuming that the time taken for blinking (closing and opening the eyes) is 100msec and the time it takes for the eyes to go from fully open to fully closed due to blinking is 50msec. In step S503, if it is calculated that the time it takes for the eyes to go from fully open to fully closed due to blinking is 50msec or less, the required frame rate F' will be greater than 60fps, and it will be impossible to acquire 3 frames at frame rate F.

[0054] In step S505, the frame rate change unit 307 calculates the required frame rate F' from the blinking time calculated in step S503, and changes the frame rate F set by the read condition setting unit 204 stored in RAM 162 so that it is equal to or greater than the required frame rate F'.

[0055] Since the time it takes to blink varies from person to person, the required frame rate F' is recalculated from the actual time it takes to blink, and the frame rate F is changed so that it is equal to or greater than the recalculated required frame rate F'. This makes it possible to acquire 3 frames even if the time it takes for the eye to go from fully open to fully closed due to blinking differs from the conditions set at the initial value. For example, in step S503, when the time it takes to blink (the action of closing and then opening the eye) is calculated to be 80 msec, the required frame rate F' is approximately 75 fps, so the frame rate F is set to 75 fps or more by performing a decimation readout process on the reference eye area on the image sensor 130.

[0056] In step S506, the blink direction detection unit 308 detects the blink direction from the change in the size of the eye region on the through-image data detected in step S502. If the blink time was not calculated in step S503, the process proceeds to step S507 without detecting the blink direction. The blink direction is detected by the sign of the change in the size of the eye region on the through-image data between frames or the sign of the rate of change in the size of the eye region on the through-image data between frames. The frame rate F is greater than or equal to the required frame rate F', that is, a frame rate that allows at least 3 frames to be acquired within the time it takes for the eyes to go from fully open to fully closed due to the blinking motion. Therefore, regardless of the blinking period and the acquisition period of the through-image data, it is possible to acquire 2 frames during the movement of the eyelids in the direction of closing or opening the eyes due to the blinking motion, and the blink direction can be detected by the sign of the change in the size of the eye region on the through-image data between frames or the sign of the rate of change in the size of the eye region on the through-image data between frames. For example, if the vertical pixel size of the fully open eye calculated by the fully open eye size calculation unit 203 is 100px, and the vertical pixel size of the eye area on the through image data was 64px in the current frame and 32px in the previous frame, then the change is increasing by 32, which indicates that the eye is moving in the direction of opening due to the blinking motion. For the sake of explanation, the direction of eye opening will be represented as + (plus) and the direction of eye closing as - (minus).

[0057] In step S507, the gaze detection unit 309 detects the gaze direction from the position information of the pupil within the eye region on the through image data detected in step S502. Specific methods for detecting the gaze direction in this invention can be known, such as determining the gaze direction from whether the position information of the pupil within the eye region is biased in any direction (up, down, left, or right) from the center of the eye region's coordinate information.

[0058] In step S508, the condition storage unit 310 stores in the RAM 162 the size of the eye region on the through image data detected in step S502, the rate of change in eye size between the current frame and the previous frame calculated in step S503, the blinking direction detected in step S506, and the gaze direction detected in step S507. Once storage is complete, the process proceeds to step S004 in Figure 2.

[0059] Each step in calculating the shooting timing in step S005 may be incorporated into a flowchart that includes the process up to the point in step S002 when the release determination unit 208 determines whether or not the release button, which is one of the user interfaces 180, was pressed by the user. By incorporating each step in calculating the shooting timing in step S005 before the release determination, it becomes possible to detect the blink time before the user presses the release button with the intention of taking a picture, and to set the frame rate F appropriately.

[0060] In step S006, the timing determination unit 311 determines whether the size of the eye region, blinking direction, and gaze direction on the through image data stored in RAM 162 in step S508 are optimal for the next frame. If it determines that they are optimal, the process proceeds to step S007. If it does not determine that they are optimal, the process returns to step S005 and the shooting timing calculation is performed again.

[0061] The timing determination unit 311 determines whether the size of the eye region on the through-image data stored in RAM 162, the blinking direction, and the gaze direction are optimal in the next frame as follows: Regarding the eye size, it determines whether the timing in the next frame is when the eye size calculated by the eye full-open size calculation unit 203 becomes the eye full-open size when the size of the eye region on the through-image data changes according to the rate of change of eye size between frames stored in RAM 162. Regarding the blinking direction, it determines whether the sign of the blinking direction is + (plus). Regarding the gaze direction, it determines whether the timing in the next frame is when the eyes are looking at the camera based on the pupil position information detected by gaze detection. In the present invention, known methods can be used to determine whether the gaze direction is optimal, such as predicting the timing when the gaze is directed towards the imaging device 100 from the gaze period.

[0062] In step S007, the shooting instruction unit 312 issues an interrupt instruction to the buffer memory inside the image sensor 130, setting a timing flag to read all pixels of the image sensor 130, and proceeds to step S008.

[0063] In step S008, the shooting instruction unit 312 instructs the buffer memory inside the image sensor 130 to change the exposure time for reading all pixels of the image sensor 130 to the shutter speed set by the user. If the shutter speed is set to auto (automatic setting), it instructs the imaging device 100 to change to an appropriate shutter speed calculated by the imaging device 100. Once the change is complete, the process proceeds to step S009.

[0064] In step S009, the image sensor 130 performs a full pixel readout and saves the image data to the recording medium 171 via the recording medium interface 170.

[0065] As explained above, the imaging method described in the present invention makes it possible to take a photograph in which the eyes are not blinking and the size of the eyes is optimal with the appropriate shutter timing. [Explanation of symbols]

[0066] 100 Imaging device 110 Camera body 120 Lens Optics 130 Image Sensors 141 Signal Processing Unit 142 Image Processing Unit 150 CPU 161 ROM 162 RAM 170 Recording medium interface 171 Recording media 180 User Interfaces 190 Image display section

Claims

1. A photographing method for an imaging device equipped with an imaging element that photoelectrically converts a subject image formed through an optical system and sequentially outputs continuous image data, comprising: a readout condition setting step for setting readout conditions for the reference eye area; a photographing condition storing step for storing the full-open size of the eye, the reference eye area, and the readout conditions; a photographing timing calculation step for acquiring a state of the eye area on image data that is sequentially output and obtained by reading out the reference eye area; a photographing timing determination step for determining whether the size of the eye area on image data that is sequentially output and obtained by reading out the reference eye area will satisfy the full-open eye size within a certain period of time, based on the state of the eye area on image data that is sequentially output and obtained by reading out the reference eye area acquired in the photographing timing calculation step; and a photographing instruction step for instructing the image sensor to read all pixels when it is determined in the photographing timing determination step that the size of the eye area on image data that is sequentially output and obtained by reading out the reference eye area will satisfy the full-open eye size within the certain period of time.

2. The imaging method of the imaging device described in claim 1, characterized in that in the read condition setting step, when the frame rate for reading out the reference eye region is less than a predetermined frame rate, a thinning readout process for the reference eye region is set so that the frame rate for reading out the reference eye region is equal to or higher than the predetermined frame rate.

3. 3. The imaging method of claim 1, wherein in the photographing timing calculation step, the amount and rate of change in the size of the eye region on the image data that is sequentially output and obtained by reading out the reference eye region are detected, and the blink duration and blink direction are calculated from the amount and rate of change in the size of the eye region on the image data that is sequentially output and obtained by reading out the reference eye region.

4. 4. The imaging method of claim 3, wherein in the photographing timing determination step, the rate of change in the size of the eye area on the image data output sequentially in the next frame obtained by reading out the reference eye area and the blink direction are used to determine whether the size of the eye area on the image data output sequentially in the next frame satisfies the fully opened eye size.

Images