Display device, imaging device, control method for display device, program, and recording medium

Abstract

To reduce power consumption and prevent a user from feeling flickering of LV display.SOLUTION: A display unit has: image acquisition means that acquires time-series images; image processing means that processes the images acquired by the image acquisition means; display means that displays processed images processed by the image processing means; detection means that detects a closed state of the eyelid of a user; and control means that, in response to the detection of the closed state of the eyelid performed by the detection means, starts power-saving processing of reducing power consumption of the image processing means or the display means, and at timing when a predetermined time elapses from the detection of the closed state of the eyelid, instructs resumption of the processing performed by the image processing means or the display means. The predetermined time is based on time obtained by subtracting the time required from cancellation of the power-saving processing until the resumption of the processing performed by the image processing means or the display means from a predetermined blinking time of the user.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a display device, an imaging device, a control method for a display device, a program, and a recording medium.

Background Art

[0002] In recent years, in imaging devices such as digital cameras, shooting is performed by confirming a subject through live view (hereinafter referred to as LV) display on a display unit such as a rear monitor or an EVF (Electronic View Finder). In LV shooting, there is a problem that power consumption increases because the display is performed on the rear monitor or EVF. In Patent Document 1, a power saving processing method is proposed in which a blink is detected by detecting the temperature at the time of eye opening and closing with an infrared sensor, and the screen display is turned off during the blink to reduce power consumption.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, the screen display is turned on at the timing when it is detected that the blink has ended (the eyelids have opened). However, since there is a time lag until the end of the blink is detected and the process is reflected, a black image may be displayed when the user opens their eyes, and the screen may appear to flicker.

[0005] The present invention has been made in view of the above problems, and an object thereof is to reduce the flicker of a display image and reduce power consumption during blinks.

Means for Solving the Problems

[0006] ​

[0006] The first aspect of the present invention is image acquisition means for acquiring time-series images, image processing means for processing the images acquired by the image acquisition means, display means for displaying the processed images processed by the image processing means, detection means for detecting the closed state of the user's eyelids, in response to the detection of the closed state of the eyelids by the detection means, start power-saving processing for reducing the power consumption of the image processing means or the display means, and at the timing when a predetermined time has elapsed since the closed state of the eyelids was detected, the image processing means or the display means Before the power saving process processing of control means for instructing resumption, a blink time measurement means for measuring the blink time of the user; having The blink time measurement means determines the type of blink based on the presence or absence of a sudden change in brightness or sound in a frame immediately before or in the vicinity of the detected closed state of the eyelid, measures the blink time of the user for each type of blink, The blink time of the user used for calculating the predetermined time is based on the blink time corresponding to the type of blink measured by the blink time measurement means. A display device characterized by the above.

[0007] The second aspect of the present invention is A control method for a display device, comprising: an image acquisition step of acquiring time-series images, an image processing step of processing the images acquired in the image acquisition step, a display step of displaying the processed images processed in the image processing step, a detection step of detecting the closed state of the user's eyelids, in response to the detection of the closed state of the eyelids in the detection step, start power-saving processing for reducing the power consumption by the image processing step or the display step, and at the timing when a predetermined time has elapsed since the closed state of the eyelids was detected, instruct resumption of the processing before the power-saving processing in the image processing step or the display step, a blink time measurement step of measuring the blink time of the user, including In the blink time measurement step, the type of blink is determined based on whether a rapid change in brightness or sound is detected in a frame immediately before or near the frame in which the closed state of the eyelid is detected, and the blink time of the user is measured for each type of blink. The blink time of the user used for calculating the predetermined time is based on the blink time corresponding to the type of blink measured in Step the above blink time measurement. This is a control method for a display device.

[0008] A third aspect of the present invention is a program for causing a computer to execute each step of the control method for the display device described above.

[0009] A fourth aspect of the present invention is a computer-readable recording medium storing a program for causing a computer to execute each step of the control method for the display device described above.

[0010] A fifth aspect of the present invention is imaging means for acquiring an image of a subject, image processing means for processing the image acquired by the Imaging above means, display means for displaying the processed image processed by the image processing means, detection means for detecting the closed state of the user's eyelid, in response to the detection of the closed state of the eyelid by the detection means, power saving processing for reducing the power consumption of the image processing means or the display means is started, and at the timing when a predetermined time has elapsed since the closed state of the eyelid was detected, the image processing means or the display means Before the power saving process processing of resumption is instructed by control means. a blink time measurement means for measuring the blink time of the user; It has The blink time measurement means determines the type of blink based on the presence or absence of a sudden change in brightness or sound in a frame immediately before or in the vicinity of the detected closed state of the eyelid, measures the blink time of the user for each type of blink, The blink time of the user used for calculating the predetermined time is based on the blink time corresponding to the type of blink measured by the blink time measurement means. This is an imaging device.

Advantages of the Invention

[0011] According to the present invention, it is possible to reduce power consumption and at the same time reduce the flicker of the displayed image.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Modes for Carrying Out the Invention

[0013] <Embodiment 1> Hereinafter, a preferred embodiment of the present invention will be described by taking a digital still camera as an example with reference to the accompanying drawings. However, this embodiment is not limited to a digital still camera, and can also be applied to, for example, a digital video camera, a head-mounted display, a glasses-type display, etc.

[0014] <Description of the Configuration> Figure 1 is Figure 2 (A)It is a cross-sectional view of the camera housing cut along the YZ plane formed by the Y-axis and the Z-axis shown in the figure, and is an explanatory diagram showing the outline of the configuration of the digital still camera 1 in the present invention. In FIGS. 1 and 2, corresponding parts are denoted by the same numbers.

[0015] Inside the photographing lens unit 1A, two lenses 101, 102, a diaphragm 111, a diaphragm driving unit 112, a lens driving motor 113, a lens driving member 114, a photocoupler 115, a pulse plate 116, a mount contact 117, a focus adjustment circuit 118, etc. are included. In this embodiment, two lenses 101, 102 are shown for simplicity, but actually more than two lenses are included in the photographing lens unit 1A. The lens driving member 114 is composed of a driving gear etc., and the photocoupler 115 detects the rotation of the pulse plate 116 interlocked with the lens driving member 114 and transmits it to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens driving motor 113 based on the information from the photocoupler 115 and the information from the camera housing 1B (information on the lens driving amount), and moves the lens 101 to change the focusing position. The mount contact 117 is an interface between the photographing lens unit 1A and the camera housing 1B.

[0016] Inside the camera housing 1B, an imaging element 2, a CPU 3, a memory unit 4, a display element 10 (display means), a display element driving circuit 11, etc. are included. The imaging element 2 (image acquisition means; imaging means) is arranged on the planned imaging plane of the photographing lens unit 1A. The CPU 3 is a central processing unit of a microcomputer and controls the entire digital still camera 1. The memory unit 4 stores time-series images etc. captured by the imaging element 2. The display element 10 is composed of liquid crystal etc. and displays the captured image (subject image) etc. on the display surface of the display element 10. The display element driving circuit 11 drives the display element 10. The user can view the display surface of the display element 10 through the eyepiece 12.

[0017] Inside the camera housing 1B, there are also included a light sources 13a, 13b, a beam splitter 15, a light receiving lens 16, an imaging device 17 for the eyeball, etc. The light sources 13a, 13b are light sources that have been conventionally used in single-lens reflex cameras and the like for detecting the line-of-sight direction based on the relationship between the reflected image (corneal reflection image; Purkinje image) by the corneal reflection of light and the pupil, and are light sources for illuminating the user's eyeball 14. Specifically, the light sources 13a, 13b are infrared light-emitting diodes or the like that emit infrared light insensitive to the user, and are arranged around the eyepiece lens 12. The optical image of the illuminated eyeball 14 (eyeball image; an image formed by the reflected light emitted from the light sources 13a, 13b and reflected by the eyeball 14) passes through the eyepiece lens 12 and is reflected by the beam splitter 15. Then, the eyeball image is formed on the imaging device 17 for the eyeball in which a two-dimensional array of photoelectric elements such as CMOS is arranged by the light receiving lens 16. The light receiving lens 16 positions the pupil of the eyeball 14 and the imaging device 17 for the eyeball in a conjugate imaging relationship. Based on a predetermined algorithm described later, the line-of-sight direction of the eyeball 14 (the viewpoint (viewpoint position; line-of-sight position) on the display surface of the display element 10) is detected from the position of the corneal reflection image in the eyeball image formed on the imaging device 17 for the eyeball.

[0018] Figure 2 shows the appearance of the digital still camera 1 according to the present invention. Figure 2(A) is a front perspective view, and Figure 2(B) is a rear perspective view. In this embodiment, the digital still camera 1 is composed of a photographing lens Unit 1A and a camera housing 1B as shown in the front perspective view of Figure 2(A). A release button 5, which is an operation member for receiving an imaging operation from the user (photographer), is arranged on the camera housing 1B. Also, as shown in the rear perspective view of Figure 2(B), an eyepiece lens 12 (eyepiece optical system) for the user to look into the display element 10 (display panel) included in the camera housing 1B is arranged on the back of the camera housing 1B. Note that the eyepiece optical system may include a plurality of lenses. Operation members 41 to 43 for receiving various operations from the user are also arranged on the back of the camera housing 1B. For example, the operation member 41 is a touch panel that receives touch operations. Also, the operation member 41 includes a display panel such as a liquid crystal panel and has a function of displaying an image on the display panel. The operation member 42 is as shown in the figure2 (C) As shown, it is an operation lever that can be pushed down in each direction. The operation member 43 is a four-way key (button-type cross key) that can be pushed in each of the four directions.

[0019] FIG. 3 is a block diagram showing the electrical configuration incorporated in the digital still camera having the above-described configuration, and those identical to FIG. 1 are given the same reference numerals. Connected to the CPU 3 are a line-of-sight detection circuit 201, a photometry circuit 202, an autofocus detection circuit 203, a signal input circuit 204, a display element drive circuit 11, an illumination light source drive circuit 205, a tracking circuit 207, a recognition circuit 208, an image processing circuit 209, and an audio input unit 210. Further, the CPU 3 transmits signals to a focus adjustment circuit 118 arranged in the photographic lens unit 1A and a diaphragm control circuit 206 included in the diaphragm drive unit 112 in the photographic lens unit 1A via the mount contact 117. The memory unit 4 associated with the CPU 3 has a function of recording imaging signals from the imaging element 2 and the eyeball imaging element 17 and a function of recording line-of-sight correction data for correcting individual differences in line of sight, which will be described later.

[0020] The line-of-sight detection circuit 201 performs A / D conversion on the output resulting from the formation of an eyeball image from the eyeball imaging element 17 (CCD - EYE) and transmits this image information to the CPU 3. The CPU 3 (line-of-sight detection means) extracts each feature point of the eyeball image necessary for line-of-sight detection according to a predetermined algorithm described later, and further detects the user's viewpoint position (line of sight) from the positions of the respective feature points.

[0021] The photometry circuit 202 amplifies, logarithmically compresses, and performs A / D conversion on the signal obtained from the imaging element 2 that also serves as a photometry sensor, and sends it to the CPU 3 as scene brightness information corresponding to the brightness of the scene.

[0022] The autofocus detection circuit 203 A / D-converts the signal voltages from a plurality of pixels used for phase difference detection, which are included in the CCD in the imaging device 2, and sends them to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals of the plurality of pixels. This is a well-known technique known as imaging plane phase difference AF. In this embodiment, as an example, it is assumed that there are 180 focus detection points at positions on the imaging plane corresponding to the locations shown in the finder internal field image of FIG. 4.

[0023] Connected to the signal input circuit 204 is a switch SW1 that turns on at the first stroke of the release button 5 and starts operations such as photometry, distance measurement, and line-of-sight detection of the digital still camera 1. Also connected to the signal input circuit 204 is a switch SW2 that turns on at the second stroke of the release button 5 and starts the shooting operation. The ON signals from the switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the CPU 3. The operation signals of the above-described operation members 41 to 43 are configured to be transmitted to the CPU 3, and movement operation control of the estimated gaze point frame position described later is performed according to the operation signals of the operation members 41 to 43.

[0024] The tracking circuit 207 is a circuit that inputs an image and tracks the subject under the control of the CPU 3, and transmits the information of the tracking frame of the image information to the CPU 3. Note that the tracking process is performed, for example, by SAD (Sum Of Absolute Difference) to obtain the similarity between two images for tracking. Also, tracking processing other than SAD may be used for the tracking circuit 207.

[0025] The recognition circuit 208 is a circuit that recognizes a subject in the input image, and performs, for example, face detection of a person or detection of an animal.

[0026] The image processing circuit 209 (image processing means) is composed of various image processing units and buffer memories, etc., and appropriately performs processes such as magnification chromatic aberration correction, development processing, noise reduction processing, geometric distortion, and resizing such as scaling on the acquired image data. In addition, the image processing circuit 209 also includes a correction unit and the like that appropriately perform pixel correction, black level correction, shading correction, defect correction, etc. on the image data. The processed image processed by the image processing circuit 209 is displayed by the display element 10.

[0027] The voice input unit 210 acquires voice from a built-in microphone or a voice input device connected via a voice input terminal, and sends the acquired voice to the CPU 3. The CPU 3 selectively performs analog-to-digital conversion on the input voice signal as necessary, and performs level optimization processing and specific frequency reduction processing to generate a voice signal.

[0028] FIG. 4 is a diagram showing the inside of the finder view, and shows the state in which the display element 10 is operating. In FIG. 4, 300 is a view mask, 400 is a focus detection area, and 4001 to 4180 are 180 ranging point visual marks displayed superimposed on the through image shown on the display element 10 at positions corresponding to a plurality of focus detection points on the imaging surface. Among those indicators, an indicator corresponding to the current estimated fixation point position is displayed with a frame appearing like the estimated fixation point A in the figure.

[0029] <Explanation of the gaze detection operation> The gaze detection method and blink detection method according to the first embodiment will be described with reference to FIGS. 5, 6(A), 6(B), and 7.

[0030] FIG. 5 is a diagram for explaining the principle of the gaze detection method, and corresponds to a schematic diagram of the optical system for performing the gaze detection of FIG. 1 described above. In FIG. 5, 13a and 13b are light sources such as light-emitting diodes that emit infrared light insensitive to the user, and each light source is arranged substantially symmetrically with respect to the optical axis of the light-receiving lens 16 to illuminate the user's eyeball 14. A part of the illumination light reflected by the eyeball 14 is condensed by the light-receiving lens 16 onto the imaging element 17 for the eyeball.

[0031] FIG. 6(A) is a schematic diagram of an eye image projected onto the eye imaging device 17 (the eye image captured by the eye imaging device 17), and FIG. 6(B) is an output intensity diagram of the CCD in the eye imaging device 17.

[0032] FIG. 7 is a schematic flowchart of the gaze detection operation according to the first embodiment. In FIG. 7, the gaze detection method and the discrimination method for detecting blinks will be described.

[0033] In step S001, the light sources 13a and 13b emit infrared light toward the user's eyeball 14. The user's eye image illuminated by the infrared light forms an image on the eye imaging device 17 through the light receiving lens 16 and is photoelectrically converted by the eye imaging device 17. As a result, the eye image can be processed as an electrical signal.

[0034] In step S002, the eye image (eye image signal; electrical signal of the eye image) obtained from the eye imaging device 17 is sent to the CPU 3.

[0035] In step S003, the CPU 3 determines whether the corneal reflection images Pd and Pe of the light sources 13a and 13b shown in FIG. 5 are detected. If the corneal reflection images Pd and Pe are detected, the process proceeds to step S004; if not, the process proceeds to step S010. That is, the CPU 3 is also a gaze detection means for detecting the user's viewpoint position with respect to the display means (display element 10), and is also a detection means for detecting the closed state of the user's eyelids. When the viewpoint position is not detected (the corneal reflection images Pd and Pe are not detected), the CPU 3 detects the closed state of the eyelids (blink). Note that the method for detecting blinks is not limited to this, and other methods may be used. For example, blinks may be detected based on changes in the surface temperature of the eyeball, or blinks may be detected by image recognition.

[0036] In step S004, the CPU 3 obtains the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b and the pupil center c shown in FIG. 5 from the eye image obtained in S002. The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the user's eyeball 14. At this time, the corneal reflection images Pd and Pe formed by a part of the infrared light reflected on the surface of the cornea 142 are condensed by the light receiving lens 16 and form images on the imaging element 17 for the eyeball (points Pd' and Pe' shown in the figure). Similarly, the light fluxes from the ends a and b of the pupil 141 also form images on the imaging element 17 for the eyeball.

[0037] FIG. 6(B) shows the luminance information (luminance distribution) of the region α in the eye image of FIG. 6(A) obtained from the imaging element 17 for the eyeball. In FIG. 6(B), the horizontal direction of the eye image is the X-axis direction, the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. At this time, let the coordinates in the X-axis direction (horizontal direction) of the images Pd' and Pe' where the corneal reflection images of the light sources 13a and 13b are formed be Xd and Xe. Also, let the coordinates in the X-axis direction of the images a' and b' where the light fluxes from the ends a and b of the pupil 14b are formed be Xa and Xb. In the example of the luminance information in FIG. 6(B), at the positions Xd and Xe corresponding to the images Pd' and Pe' where the corneal reflection images of the light sources 13a and 13b are formed, extremely high levels of luminance are obtained. In the region between the coordinates Xa and Xb corresponding to the region of the pupil 141, extremely low levels of luminance are obtained except for the positions of Xd and Xe. In contrast, for the pupil 141 In the luminance information example of FIG. 6(B), extremely high levels of luminance are obtained at the positions Xd and Xe corresponding to the images Pd' and Pe' where the corneal reflection images of the light sources 13a and 13b are formed. In the region between the coordinates Xa and Xb corresponding to the region of the pupil 141, extremely low levels of luminance are obtained except for the positions of Xd and Xe. In the region between the coordinates Xa and Xb corresponding to the region of the pupil 141, extremely low levels of luminance are obtained except for the positions of Xd and Xe. In the regions with X - coordinate values lower than Xa and higher than Xb, which correspond to the region of the external brilliance 143, the intermediate value between the two luminance levels is obtained. From the variation information of the luminance level with respect to the X - coordinate position, the X - coordinates Xd, Xe of the images Pd’, Pe’ where the corneal reflection images of the light sources 13a, 13b are formed, and the X - coordinates Xa, Xb of the images a’, b’ of the pupil edges can be obtained. Also, when the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light - receiving lens 16 is small, the coordinate Xc of the location (denoted as c’) corresponding to the pupil center c formed on the imaging element 17 for the eyeball can be expressed as Xc≒(Xa + Xb) / 2. From the above, the X - coordinate of c’ corresponding to the pupil center formed on the imaging element 17 for the eyeball and the coordinates of the corneal reflection images Pd’, Pe’ of the light sources 13a, 13b can be estimated.

[0038] In step S005, the CPU 3 calculates the imaging magnification β of the eyeball image. β is a magnification determined by the position of the eyeball 14 with respect to the light - receiving lens 16, and can be obtained substantially as a function of the interval (Xd - Xe) between the corneal reflection images Pd‘, Pe’.

[0039] In step S006, the CPU 3 calculates the rotation angle of the optical axis of the eyeball 14 with respect to the optical axis of the light - receiving lens 16. The X - coordinate of the mid - point of the corneal reflection images Pd and Pe substantially coincides with the X - coordinate of the curvature center O of the cornea 142. Therefore, assuming the standard distance from the curvature center O of the cornea 142 to the center c of the pupil 141 is Oc, the rotation angle θx of the optical axis of the eyeball 14 in the Z - X plane can be calculated by the following formula 1. The rotation angle θy when the user's eyeball rotates in a plane perpendicular to the X - axis can also be calculated in the same way as the rotation angle θx. β×Oc×SINθx≒{(Xd + Xe) / 2}-Xc ···(Formula 1)

[0040] In step S008, using the rotation angles θx, θy calculated in step S007, the viewpoint (the position where the line of sight is directed; the position where the user is looking) of the user on the display element 10 is obtained. Assuming that the coordinates (Hx, Hy) of the viewpoint correspond to the coordinates (Hx, Hy) of the center c of the pupil 141 on the display element 10, the coordinates (Hx, Hy) of the viewpoint can be calculated by the following formulas 2 and 3. Hx = m × (Ax × θx + Bx) ··· (Equation 2) Hy = m × (Ay × θy + By) ··· (Equation 3)

[0041] The coefficient m in Equations 2 and 3 is a constant determined by the configuration of the viewfinder optical system of the camera, and is a conversion coefficient for converting the rotation angles θx and θy into the position coordinates corresponding to the center c of the pupil 141 on the display element 10. It is assumed that the coefficient m is determined in advance and recorded in the memory unit 4, and is read from the memory unit 4 in step S007. Also, Ax, Bx, Ay, and By are gaze correction coefficients for correcting individual differences in the user's gaze, are obtained by performing a calibration operation described later, and are assumed to be recorded in the memory unit 4 before the gaze detection routine starts.

[0042] In step S009, the CPU 3 records the coordinates (Hx, Hy) of the viewpoint in the memory unit 4. In this way, the gaze detection routine when no blink is detected is terminated.

[0043] On the other hand, if a blink is detected in step S003, the process proceeds to step S010 to determine the type of blink. Here, there are three types of the user's blinking behavior: reflex blinking due to external stimuli such as sound and light, spontaneous blinking where the eyelids are unconsciously closed without external stimuli, and voluntary blinking where the eyelids are consciously closed. The blink time of each type tends to be longer for spontaneous blinking than for reflex blinking, and longer for voluntary blinking than for spontaneous blinking. Therefore, by discriminating the type of blink and switching the power saving period according to the type of blink, it is possible to perform power saving processing while suppressing the user's discomfort. Also, since voluntary blinking tends to have a longer blink time than spontaneous blinking, it is considered that the user's discomfort is small even if the same power saving period as that of spontaneous blinking is used in the case of voluntary blinking. Therefore, in the first embodiment, the CPU 3 discriminates the type of blink into reflex blinking and spontaneous blinking, and records the type of blink in the memory unit 4.

[0044] In step S010, the CPU 3 determines whether the blink (eyelid closed state) detected in step S003 is a reflex blink or a spontaneous blink. Therefore, the CPU 3 may determine the type of blink based on whether a sudden brightness change or a loud sound was detected in the frame immediately before the blink was detected in step S003. Note that the brightness change or sound detection target may be the frame immediately before the blink was detected, or a neighboring frame. The neighboring frames may be, for example, several frames (the number is arbitrary) before and after the frame in which the CPU 3 detected the blink. The brightness change may be determined, for example, using the difference in luminance values between the frame in which the blink was detected and the immediately preceding or neighboring frame. Note that the difference in luminance values may be compared by averaging the luminance values of the entire image, comparing specific regions (e.g., the center of the image), or using other methods. The loud sound may be determined, for example, based on whether the voice input unit 210 acquired a volume equal to or greater than a predetermined level in the immediately preceding or neighboring frame. The methods for determining the brightness change and the loud sound are examples, and any other arbitrary methods can be used. In the first embodiment, if at least one of the brightness change and the loud sound is detected, the process proceeds to step S011, and if neither the brightness change nor the loud sound is detected, the process proceeds to step S012.

[0045] In step S011, the CPU 3 records the blink detected in step S003 as a reflex blink in the memory unit 4 and ends the line-of-sight detection routine.

[0046] In step S012, the CPU 3 records the blink detected in step S003 as a spontaneous blink in the memory unit 4 and ends the line-of-sight detection routine.

[0047] The above shows a method for acquiring viewpoint coordinates on the display element 10 using the corneal reflection images of the light sources 13a and 13b. However, the method for acquiring viewpoint coordinates is not limited to this, and the present invention is applicable to any method for acquiring the eyeball rotation angle from the captured eye image.

[0048] <Calibration Work> In the above-described line-of-sight detection routine, the rotation angles θx and θy of the eyeball are obtained from the eye image, and an operation is performed to perform coordinate conversion of the pupil center position to the corresponding position on the display element 10 to estimate the viewpoint position. However, due to factors such as individual differences in the shape of the human eyeball, it may not be possible to accurately estimate the viewpoint. Specifically, if the values of the line-of-sight correction coefficients Ax, Ay, Bx, and By are not adjusted to appropriate values by the user, as shown in FIG. 4(B), a deviation will occur between the actual viewpoint B and the estimated viewpoint C of the user. In FIG. 4(B), although the user is gazing at the person at position B, the camera side erroneously estimates that the background is being gazed at, and it has fallen into a state where appropriate focus detection and adjustment cannot be performed. Therefore, it is necessary to perform a calibration operation before imaging with the camera, obtain the values of the correction coefficients suitable for the user, and store them in the digital still camera 1.

[0049] Conventionally, the calibration operation has been performed by highlighting a plurality of indicators at different positions as shown in FIG. 4(C) within the viewfinder field before imaging and having the observer look at the indicators. A technique for obtaining a line-of-sight correction coefficient suitable for the user from the calculated plurality of viewpoints (estimated positions) and the coordinates of each indicator when performing the line-of-sight detection routine at the time of each visual target fixation is known as a known technique. Note that the method of displaying the indicator is not particularly limited as long as the position that the user should look at is suggested, and a graphic that is the indicator may be displayed, or the indicator may be displayed by changing the brightness or color of an image (such as a captured image).

[0050] Here, the CPU 3 may measure the blink time of the user in the line-of-sight detection routine during calibration. That is, the CPU 3 is also a blink time measuring means. For example, the spontaneous blink time may be measured using the time from when the CPU 3 detects a blink (the corneal reflection image is not detected) in step S003 and proceeds to the process in step S010 until step S004 is processed in the subsequent line-of-sight detection routine. Also, for example, the reflex blink time is determined by changing the brightness of the image displayed on the display element 10 from bright to dark (or Dark from Bright)It may be rapidly changed to that, and the measurement may be performed using the blink time detected at that time. In this way, the CPU 3 discriminates the types of the user's blinks and measures the user's blink time for each type of blink. In the first embodiment, the CPU 3 measures the blink time during calibration, but it may be performed at any time during normal shooting or in response to a user's request. When the blink time is measured multiple times, for example, the average value may be used as the user's blink time, or the blink time measured immediately before may be used as the user's blink time.

[0051] Using FIGS. 8, 9, and 10, the power saving process when the CPU 3 detects a blink will be described.

[0052] <Blink power saving process flow> FIG. 8 is a flowchart showing a process flow repeatedly executed in the digital still camera 1. In FIG. 8, as an example of a power saving process for reducing power consumption, the CPU 3 (control means) stops the processing of the image processing circuit 209 and describes the flow in the case of switching the power saving period for performing the power saving process according to the type of blink. Here, an example of performing the power saving process on the image processing circuit 209 is shown, but the CPU 3 may control to perform the power saving process on the display element 10, or may control to perform the power saving process on both the image processing circuit 209 and the display element 10.

[0053] In step S101, the CPU 3 reads the blink detection result of step S011 or step S012 of the line-of-sight detection routine from the memory unit 4.

[0054] In step S102, the CPU 3 determines the presence or absence of blink detection from the read result of step S101. If there is a "blink", it proceeds to step S103, and if there is "no blink", it returns to step S101.

[0055] In step S103, the CPU 3 writes "no blink" into the blink detection result in the memory unit 4. This is to prevent a determination of "blink" from being made in the next iteration process. The CPU 3 issues an instruction to stop the processing of the image processing circuit 209, initializes the timer for determining the blink end timing, and starts the timer. That is, the CPU 3 instructs the start of power saving processing for the image processing circuit 209 in response to the detection of the closed state of the eyelids. Also, in step S103, the CPU 3 determines which of the blink times for reflexive blinking and spontaneous blinking to adopt among the predetermined blink times based on the type of blink stored in the memory unit 4 in step S011 or step S012. Here, the blink time for reflexive blinking (when the CPU 3 detects a sudden change in brightness or sound) is shorter than the blink time for spontaneous blinking. Therefore, the power saving period in the case of reflexive blinking may be made shorter than that in the case of spontaneous blinking. Thus, in the first embodiment, the power saving period is determined in consideration of the blink time according to the type of blink. Thereby, power saving processing can be performed within a time corresponding to the type of blink, and the discomfort of the user can be suppressed. Note that as the blink time of the user used for calculating the power saving period, it is preferable to use the blink time (measurement value) for each user measured during calibration or the like, but a general blink time (fixed value) recorded in advance in the memory unit 4 or the like may also be used.

[0056] In step S104, in response to the instruction to start power saving processing issued in step S103, the processing of the image processing circuit 209 stops. Note that as described above, the content of the power saving processing is not limited to this.

[0057] In step S105, the CPU 3 determines whether the power saving period (predetermined time) of the image processing circuit 209 has elapsed since the CPU 3 detected a blink, based on the value of the timer for determining the blink end timing. If the power saving period has elapsed, the CPU 3 instructs the image processing circuit 209 to resume processing and proceeds to S106. If the power saving period has not elapsed, the process returns to S105, that is, it waits until the power saving period elapses. Here, the power saving period may be based on the time obtained by subtracting the time required from the release of the power saving process to the resumption of processing of the image processing circuit 209 from the predetermined blink time of the user. That is, after the start of the power saving process of the image processing circuit 209, the CPU 3 may instruct the image processing circuit 209 to resume processing so that the image processed after the resumption of processing of the image processing circuit 209 is displayed on the display element 10 by the time the user's blink ends. By doing so, a new image is displayed on the display element 10 at the timing when the user opens the eyelids, and it is possible to suppress the sense of discomfort caused by the fact that the power saving process is still being performed (the image is not being displayed) at the end of the blink.

[0058] In step S106, in response to the instruction to resume processing issued in step S105, the image processing circuit 209 resumes the image processing for display on the display element 10. In FIG. 8, the power saving process flow of the image processing circuit 209 has been described, but the power saving process of the display element 10 can also be applied in the same manner. However, it is advisable to determine the power saving period in consideration of the time required to resume processing of the display element 10. That is, after the start of the power saving process of the display element 10, the CPU 3 may instruct the display element 10 to resume processing so that the image processed by the image processing circuit 209 is displayed on the display element 10 by the time the user's blink ends.

[0059] <Display example during blink power saving process> FIG. 9 assumes the time of taking a running photo and shows the LV images and the user's eye images at time series times T1 to T7 for each frame. FIG. 9(A) shows the LV image displayed on the display element 10, FIG. 9(B) shows the user's eye image formed on the imaging element 17 for the eyeball, and FIG. 9(C) shows the state of the user's eyelids and the processing content of the digital still camera 1 at each time. At times T1 and T7, it shows that the user's eyelids are open, and at times T2 to T6, it shows the closed state of the eyelids, that is, the user is blinking. Note that there are a plurality of frames between T4 and T5.

[0060] Next, with reference to FIG. 9, the power saving process when the CPU 3 detects a blink will be described. In FIG. 9, as an example of the power saving process, the process when the image processing circuit 209 and the display element 10 are stopped will be described. Note that the content of the power saving process is merely an example and is not limited thereto. In FIG. 9(C), the processing of the image processing circuit 209 and the display element 10 is denoted as "power saving ON" for the power saving process and "power saving OFF" for the process when not in the power saving process (normal time). Also, the blink time will use the measured value for each user.

[0061] At time T1, since the CPU 3 has not detected a blink, image processing is performed as the process in the power saving OFF state, and the LV image is displayed on the display element 10.

[0062] At time T2, the closed state of the eyelids is imaged on the imaging element 17 for the eyeball, but the CPU 3 has not yet detected a blink. Therefore, at time T2, since the CPU 3 has not yet issued an instruction for the power saving process, image processing and LV display are being performed.

[0063] At time T3, the CPU 3 detects a blink using the eye image at time T2 and issues an instruction to start the power saving process (power saving ON) for the image processing circuit 209 and the display element 10 (corresponding to step S103 in FIG. 8). Since time T3 is the timing before the instruction for the power saving process is reflected, image processing and LV display are being performed. That is, T3 to T4 shows the time lag from when the CPU 3 detects a blink until the power saving process starts.

[0064] At time T4, the power-saving process of the image processing circuit 209 and the display element 10 starts. That is, the image processing circuit 209 stops image processing (corresponding to step S104 in FIG. 8), and the display element 10 stops and displays nothing.

[0065] Here, in FIG. 9, the CPU 3 resumes the LV display when the user's blink ends (time T7). Therefore, the CPU 3 may cancel the power-saving process (power-saving OFF) and issue an instruction to resume the normal process in consideration of the time lag required for the image processing circuit 209 and the display element 10 to resume processing so as to make it in time for time T7. The time lag of the image processing circuit 209 may be considered as the time from when the image processing circuit 209 resumes processing until the processed image is displayed on the display element 10. Also, the time lag of the display element 10 may be considered as the time required for the display element 10 to resume processing and display an image.

[0066] Time T5 is the timing at which the time obtained by subtracting the time lag of the image processing circuit 209 from the user's blink time has elapsed since the CPU 3 detected the blink (step S105 “YES” in FIG. 8). Therefore, the CPU 3 cancels the power-saving process for the image processing circuit 209 and issues an instruction to resume image processing.

[0067] Time T6 is the timing at which the time obtained by subtracting the time lag of the display element 10 from the user's blink time has elapsed since the CPU 3 detected the blink. Therefore, the CPU 3 cancels the power-saving process for the display element 10 and issues an instruction to resume display. Also, at time T6, the image processing circuit 209 resumes image processing in response to the instruction to resume processing issued at time T5 (corresponding to step S106 in FIG. 8). Note that from time T4 to time T6, since the display element 10 is in the power-saving process, the display element 10 stops and displays nothing.

[0068] At time T7, the user opens their eyes, and the display element 10 resumes normal processing. At time T7, the power-saving processing of the image processing circuit 209 and the display element 10 is canceled, and the image processed by the image processing circuit 209 is LV-displayed. In FIG. 9, an example of resuming LV display so as to match the timing when the user opens their eyelids has been described. However, the CPU 3 issues an instruction to resume processing so that LV display resumes with a margin several frames before the user opens their eyelids. This may be acceptable. In this case, the power-saving period may be, for example, the time obtained by subtracting the time lag required for resuming processing and several frames (for allowing a margin) from the blink time, or the time obtained by subtracting the time lag required for resuming processing from the blink time set shorter in advance. Although it is assumed that the actual blink time of the user is longer than the average blink time, by setting the power-saving period in this way, the resumption of LV display before the user opens their eyelids can be realized more reliably. In order to more reliably suppress the discomfort that occurs in such a case, for example, as shown in FIG. 10 (details will be described later), the image processing circuit 209 may adopt a power-saving process in which it stops while the display element 10 does not stop.

[0069] As described above, the power-saving period (the period in which the processing is actually reflected) may be set to different times in consideration of the processing times of each part. For example, in FIG. 9, the image processing circuit 209 receives an instruction to start power-saving processing at time T3 and an instruction to cancel power-saving processing at time T5, and the power-saving period is from time T4 to T5. The display element 10 (display element driving circuit 11) receives an instruction to start power-saving processing at time T4 and an instruction to cancel power-saving processing at time T6, and the power-saving period is from time T4 to T6. In this way, the image processing circuit 209 cancels the power-saving processing earlier because the power-saving period of the image processing circuit 209 also takes into account the time required until the processed image is displayed by the display element 10. Also, since the frame timings of the imaging element 2 and the display element 10 are not always the same, it is assumed that the power-saving periods of each part are different. When the time required from the cancellation of the power-saving processing of each part to the resumption of processing varies depending on the frame rate of the imaging element 2 or the display element 10, the CPU 3 may issue an instruction to resume processing according to the frame rate so as to be in time for resuming the LV display.

[0070] <Example of power saving processing> Note that the content of the power saving processing is not limited to this. For example, in the power saving processing of the image processing circuit 209, the CPU 3 may stop the image processing circuit 209 or control it to reduce the frame rate from T4 to T6. Further, when the image processing circuit 209 includes a plurality of processes, the CPU 3 may control to perform power saving processing on the image processing related to display among the image processing circuits 209.

[0071] Also, for the power saving processing of the display element 10, a power saving processing different from the above description in FIG. 9 may be adopted. For example, the CPU 3 may stop the driving of the display element 10 itself, reduce the frame rate of the display element 10, or control to display a black image. Also, for example, the CPU 3 may control the frame rate of the display element 10 to also decrease while the process of decreasing the frame rate is being performed as the power saving processing for the image processing circuit 209.

[0072] As the power saving processing for the image processing circuit 209, while the image processing circuit is stopped, the display element 10 may display a black image or may display the frame image at the time when a blink is detected (the last frame image processed before power saving starts). For example, when the blink time of the user is shorter than the time measured during calibration, if a black image is displayed as the power saving processing of the display element 10, a black image will be displayed at the timing when the eyelids are opened, and it may look flickering. In such a case, for example, by displaying the frame image at the time when a blink is detected, it is possible to prevent the occurrence of flickering even when the blink time is shorter than expected. Note that the image to be displayed may be the image at the time when the CPU 3 detects a blink, or the image immediately before detecting a blink, or the image at the point in time assumed to be the image the user was looking at immediately before the user's blink considering the time lag until blink detection.

[0073] FIG. 10 illustrates the LV images displayed on the display element 10 at times T3 to T6 in FIG. 9(A). In FIG. 10, power-saving processing for stopping the image processing circuit 209 is performed (at times T4 to T5), and while the image to be displayed on the display element 10 after resuming the processing is being processed (at time T6), the CPU 3 displays the frame image at the time T3 when a blink is detected. By doing so, although the power-saving effect is inferior to the case where the driving of the display element 10 is also stopped (FIG. 9), the occurrence of flicker can be more reliably suppressed. Note that the CPU 3 may be controlled to change the display method of the display element 10 at the boundary of a threshold time while the power-saving processing for stopping the image processing circuit 209 is being performed. For example, the CPU 3 may control to display a black image or stop the display element 10 from the stop of the image processing circuit 209 (immediately after detecting a blink) until the threshold time, and display the frame image at the time when a blink is detected after the threshold time has elapsed. Thereby, a higher power-saving effect can be obtained than in the case of always displaying the image immediately before the blink during a blink.

[0074] The content of the power-saving processing may be switched as appropriate. For example, the CPU 3 may switch the content of the power-saving processing according to the display mode (frame rate) of the EVF. For example, as an example of the display mode of the EVF, there are three modes with different frame rates, that is, a smooth display mode (120 fps), an image quality priority display mode (60 fps), and a power-saving display mode (30 fps). The power-saving processing executed in each mode may be such that the power-saving effect becomes greater in the display mode that prioritizes the power-saving effect, that is, when the frame rate is low. For example, no power-saving processing may be performed in the smooth display mode, frames for which image processing is performed may be decimated in the image quality priority display mode, and the image processing and the display element 10 may be stopped in the power-saving display mode. By doing so, power-saving processing can be performed according to the user's usage. Note that the names of the respective display modes, the values of the frame rates, and the content of the power-saving processing to be executed are examples and are not limited thereto.

[0075] Also, the control targets during the power-saving period are not limited to the image processing circuit 209 and the display element 10. For example, the CPU 3 may control the imaging element 2, the photometry circuit 202, the recognition circuit 208, the tracking circuit 207, the autofocus detection circuit 203, etc. so as to reduce power consumption. By performing power-saving processing on these elements and circuits, a higher power-saving effect can be obtained. In any case, it is advisable to set the power-saving period so that when the user opens their eyelids, the image after resuming processing is displayed on the display element 10. Also, the CPU 3 may switch the content of the power-saving processing according to the operation mode of the camera. Here, the operation modes of the camera are a non-shooting state, a pre-shooting preparation state, and a shooting state. The non-shooting state is a state in which the LV image is displayed on the display element 10 and the release button 5 is not pressed. The pre-shooting preparation state is a state in which AF is being adjusted before shooting, and the release button 5 is Half pressed. The shooting state is a state in which a still image is being shot, and the release button 5 is pressed. During the shooting state, that is, during continuous shooting, it is necessary to shoot a still image even if the user blinks. Therefore, the blink power-saving process is preferably applied to the non-shooting state and the pre-shooting preparation state. For example, in the non-shooting state, there is a non-AF-during-non-shooting mode in which autofocus (AF) is not performed during non-shooting. In the non-AF-during-non-shooting mode, the imaging element 2, the photometry circuit 202, the recognition circuit 208, the tracking circuit 207, and the autofocus detection circuit 203 may be controlled to stop or be thinned out.

[0076] In this way, by switching the power-saving process according to the display mode and the operation mode, it is possible to perform a power-saving process suitable for the user's usage method. Note that the power-saving process for each of the above modes is not limited to this, and the content of the power-saving process may be changed or combined.

[0077] As described above, in this embodiment, blinks are detected, and the process is resumed in consideration of the blink time and the process resumption time of each circuit. As a result, it is possible to reduce power consumption and at the same time reduce the flicker of the displayed image.

[0078] <Modification Example> In Example 1, an example was described in which the blinking time of a user was measured and the power saving period was calculated using the blinking time for each user. However, for example, the average blinking time may be recorded in advance in the memory unit 4 or the like, and the power saving period may be calculated using the pre-recorded blinking time. In this case, considering the time lag from the occurrence of the user's blink until the blink is detected by the CPU 3, it is advisable to set the power saving period. Also, in Example 1, an example was described in which the type of blink was discriminated and the power saving period was used appropriately for each type of blink. However, the same blinking time may be used regardless of the type of blink. When the same blinking time is used regardless of the type of blink, the discrimination process for the type of blink (steps S010 to 012) is unnecessary. As a result, power saving processing can be performed with a simpler process. From the occurrence of the user's blink until the blink is detected by the CPU 3, it is advisable to set the power saving period considering the time lag. Also, in Example 1, an example was described in which the type of blink was discriminated and the power saving period was used appropriately for each type of blink. However, the same blinking time may be used regardless of the type of blink. When the same blinking time is used regardless of the type of blink, the discrimination process for the type of blink (steps S010 to 012) is unnecessary. As a result, power saving processing can be performed with a simpler process.

[0079] <Other Embodiments> As described above, the present invention has been described in detail based on its preferred embodiments. However, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of this invention are also included in the present invention. Some of the above-described embodiments may be appropriately combined.

[0080] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a recording medium, and having one or more processors in the computer of the system or device read and execute the program. Also, it can be realized by a circuit (for example, ASIC) that realizes one or more functions.

Explanation of Reference Numerals

[0081] 3: CPU 4: Memory unit 10: Display element 17: Image pickup element for the eyeball 209: Image processing circuit

Claims

1. an image acquisition means for acquiring time-series images; an image processing means for processing the image acquired by the image acquisition means; a display means for displaying the processed image processed by the image processing means; a detection means for detecting the closed state of the user's eyelids; a control means for starting power saving processing for reducing the power consumption of the image processing means or the display means in response to the detection of the closed state of the eyelids by the detection means, and at the timing when a predetermined time has elapsed since the closed state of the eyelids was detected, instructing the resumption of the processing before the power saving processing of the image processing means or the display means; a blink time measurement means for measuring the blink time of the user; having The blink time measurement means discriminates the type of blink based on the detection of a sudden change in brightness or sound in a frame immediately before or in the vicinity of the detection of the closed state of the eyelids, measures the blink time of the user for each type of blink, The blink time of the user used for calculating the predetermined time is based on the blink time corresponding to the type of blink measured by the blink time measurement means. A display device characterized by the above.

2. The predetermined time is based on the blink time of the user determined in advance. The display device according to claim 1.

3. The predetermined time is based on the time obtained by subtracting the time required from the release of the power saving processing to the resumption of the processing of the image processing means or the display means from the blink time of the user determined in advance. The display device according to claim 1 or 2.

4. In the power saving processing, the control means stops the display means or reduces the frame rate of the display means. The display device according to any one of claims 1 to 3.

5. In the power saving process, the control means stops the image processing means or reduces the frame rate of the image processing means. The display device according to any one of claims 1 to 4.

6. While the image processing means is stopped as the power saving process for the image processing means, the display means displays a black image or a frame image at the time when the closed state of the eyelid is detected. The display device according to claim 5.

7. While the image processing means is stopped as the power saving process for the image processing means, the display means displays a black image from the stop of the image processing means until the threshold time, and displays a frame image at the time when the closed state of the eyelid is detected after the elapse of the threshold time. The display device according to any one of claims 5 or 6.

8. The control means switches the content of the power saving process according to the frame rate of the display means. The display device according to any one of claims 1 to 7.

9. It has a gaze detection means for detecting the viewpoint position of the user with respect to the display means. When the viewpoint position is not detected by the gaze detection means, the detection means detects the closed state of the eyelid. The display device according to any one of claims 1 to 8.

10. A gaze detection means for detecting the viewpoint position of the user with respect to the display means; It has a calibration means for calibrating the gaze detection means. The blink time measurement means measures the blink time of the user during the calibration by the calibration means. The display device according to any one of claims 1 to 9.

11. When a sudden change in brightness or sound is detected by the blink time measuring means, the blink time of the user used for calculating the predetermined time is shorter than when it is not so. The display device according to any one of claims 1 to 10.

12. The image acquisition means includes imaging means for acquiring an image of a subject. The display device according to any one of claims 1 to 11.

13. A control method for a display device, comprising: an image acquisition step of acquiring time-series images; an image processing step of processing the image acquired in the image acquisition step; a display step of displaying a processed image processed in the image processing step; a detection step of detecting a closed state of the user's eyelids; in response to the detection of the closed state of the eyelids in the detection step, starting power saving processing for reducing power consumption by the image processing step or the display step, and at the timing when a predetermined time has elapsed since the closed state of the eyelids was detected, instructing resumption of the processing before the power saving processing of the image processing step or the display step; a blink time measurement step of measuring the blink time of the user; including In the blink time measurement step, the type of blink is determined based on whether or not a sudden change in brightness or sound is detected in a frame immediately before or in the vicinity of the detection of the closed state of the eyelids, and the blink time of the user is measured for each type of blink. The blink time of the user used for calculating the predetermined time is based on the blink time corresponding to the type of blink measured by the blink time measurement step. A control method for a display device, characterized in that.

14. A program for causing a computer to execute each step of the control method for a display device according to claim 13.

15. A computer-readable recording medium storing a program for causing a computer to execute each step of the method for controlling a display device according to claim 13.

16. An imaging means for acquiring an image of a subject, An image processing means for processing the image acquired by the imaging means, A display means for displaying the processed image processed by the image processing means, A detection means for detecting the closed state of the user's eyelids, A control means for starting a power saving process for reducing the power consumption of the image processing means or the display means in response to the detection of the closed state of the eyelids by the detection means, and instructing the resumption of the processing before the power saving process of the image processing means or the display means at a timing when a predetermined time has elapsed since the closed state of the eyelids was detected, A blink time measuring means for measuring the blink time of the user, having The blink time measuring means discriminates the type of blink based on the detection of a sudden change in brightness or sound in a frame immediately before or in the vicinity of the detection of the closed state of the eyelids, measures the blink time of the user for each type of blink, The blink time of the user used for calculating the predetermined time is based on the blink time corresponding to the type of blink measured by the blink time measuring means, An imaging device characterized by the above.

Images