Eye condition detection device and its operating method

Abstract

The present invention provides an eye condition detection device that provides accurate eye condition detection results, and an operating method used for the eye condition detection device. [Solution] The present invention provides an eye state detection device and an operating method for use with the eye state detection device. The eye state detection device includes an image capture element. The operating method includes capturing a face image of the current frame with the image capture element, obtaining a current image including the eye region from the face image, obtaining at least one past image including the eye region from at least one frame prior to the current frame, performing a weighting calculation on the current image and the at least one past image to generate a processed image, and determining the eye state within the eye region based on the processed image.

Description

Technical Field

[0001] The present invention relates to a detection device and an operation method used for the detection device, and particularly relates to an eye state detection device and an operation method used for the eye state detection device.

Background Art

[0002] Currently, eye state detection technology is becoming increasingly popular. Users can interact with devices using eye states and viewing angles. Also, eye states can be used to determine the mental state of users. Therefore, how to provide accurate eye state detection technology has become an important research topic for those skilled in the art.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The present invention provides an eye state detection device that provides accurate eye state detection results and an operation method used for the eye state detection device.

Means for Solving the Problems

[0004] In one embodiment of the present invention, the operation method is used for an eye state detection device. The eye state detection device includes an image capture element. The eye state detection device executes commands of the operation method. The operation method includes capturing a face image of the current frame by the image capture element, obtaining a current image including an eye region from the face image, obtaining at least one past image including an eye region of at least one frame before the current frame, performing a weighting operation on the current image and the at least one past image to generate a processed image, and determining an eye state within the eye region based on the processed image.

[0005] In one embodiment of the present invention, the eye state detection device includes an image capture element and a determination circuit. The image capture element captures a face image of the current frame. The determination circuit is connected to the image capture element. The determination circuit obtains a current image including the eye region from the face image and obtains at least one past image including the eye region from at least one frame prior to the current frame. The determination circuit performs a weighting calculation on the current image and the at least one past image to generate a processed image. The determination circuit determines the eye state within the eye region based on the processed image. [Effects of the Invention]

[0006] Based on the above, the eye state detection device acquires a current image including the eye region from a face image, acquires a past image, performs weighting calculations on the current and past images, and generates a processed image. The eye state detection device determines the eye state within the eye region based on the processed image. The processed image can clearly show the details of the eye and its movement. As a result, the eye state detection device can provide accurate eye state detection results. [Brief explanation of the drawing]

[0007] [Figure 1] This is a schematic diagram of an eye condition detection device according to one embodiment of the present invention. [Figure 2] This is a flowchart illustrating the operation method of one embodiment of the present invention. [Figure 3] This is a schematic diagram of an eye condition detection device according to one embodiment of the present invention. [Figure 4] This is a flowchart illustrating the operation method of one embodiment of the present invention. [Figure 5] This is a schematic diagram of a facial image relating to one embodiment of the present invention. [Figure 6] This is a schematic diagram of the current image relating to one embodiment of the present invention. [Figure 7] This is a flowchart illustrating the operation method of one embodiment of the present invention. [Modes for carrying out the invention]

[0008] Referring to Figure 1, Figure 1 is a schematic diagram of an eye state detection device according to one embodiment of the present invention. In this embodiment, the eye state detection device 100 includes an image capture element 110 and a determination circuit 120. The image capture element 110 captures the face image FIMG of the current frame. The determination circuit 120 is connected to the image capture element 110. The determination circuit 120 receives the face image FIMG. The determination circuit 120 obtains the current image EIMG(0) including the eye region from the face image FIMG, and past images EIMG(-1) and EIMG(-2) including the eye region from at least one frame prior to the current frame.

[0009] In this embodiment, past image EIMG(-1) corresponds to the face image of the frame immediately preceding the current frame. Past image EIMG(-2) corresponds to the face image of the frame two frames prior to the current frame.

[0010] In this embodiment, the determination circuit 120 performs weighting calculations on the current image EIMG(0) and past images EIMG(-1) and EIMG(-2) to generate a processed image EIMG'. Based on the processed image EIMG', the determination circuit 120 determines the eye state within the eye region.

[0011] It should be noted that the determination circuit 120 performs weighting calculations on the current image EIMG(0) and past images EIMG(-1) and EIMG(-2) to generate a processed image EIMG', and then determines the eye state within the eye region based on the processed image EIMG'. The processed image EIMG' can clearly show the details and movement of the eye. As a result, the determination circuit 120 can provide accurate eye state detection results.

[0012] In this embodiment, the determination circuit 120 can generate a processed image EIMG' by performing a weighting operation on the current image EIMG(0) and past images EIMG(-1) and EIMG(-2) using a plurality of weights. For example, the determination circuit 120 can provide weights W1, W2, and W3, but the present invention is not limited thereto. The determination circuit 120 generates a weighted image EW1 by multiplying the current image EIMG(0) by weight W1. The determination circuit 120 generates a weighted image EW2 by multiplying the past image EIMG(-1) by weight W2. The determination circuit 120 generates a weighted image EW3 by multiplying the past image EIMG(-2) by weight W3. Next, the determination circuit 120 generates a processed image EIMG' by superimposing the weighted images EW1, EW2, and EW3.

[0013] Furthermore, the values ​​of weights W1, W2, and W3 may be adjusted according to the importance between the current image EIMG(0) and the past images EIMG(-1) and EIMG(-2). For example, the importance of the current image EIMG(0) is higher than the importance of the past images EIMG(-1) and EIMG(-2). Therefore, weight W1 is higher than weights W2 and W3. For example, the importance of the current image EIMG(0) is higher than the importance of the past image EIMG(-1). The importance of the past image EIMG(-1) is higher than the importance of the past image EIMG(-2). Therefore, weight W1 is higher than weight W2, and weight W2 is higher than weight W3. For example, weight W1 is, for example, ''0.8'', weight W2 is, for example, ''0.5'', and weight W3 is, for example, ''0.3''. The present invention does not limit the values ​​of weights W1, W2, and W3.

[0014] Furthermore, the determination circuit 120 may estimate the eye state from the processed image EIMG' using an analysis model. For example, the eye state may be the line of sight of the eyeball, the angle of sight, or the state of eye closure. The analysis model may be, for example, a convolutional neural network (CNN) model or another model.

[0015] In some embodiments, the determination circuit 120 performs a weighting calculation on only the current image EIMG(0) and the past image EIMG(-1) to generate the processed image EIMG'. In other words, the determination circuit 120 of the present invention performs a weighting calculation on the current image EIMG(0) and one or more past images to generate the processed image EIMG'.

[0016] In this embodiment, the image capture element 110 is implemented, for example, by any type of dynamic vision sensor (DVS). The face image FIMG is a dynamic vision image. Therefore, the face image FIMG, the current image EIMG(0), and the past images EIMG(-1) and EIMG(-2) are images generated by local pixel brightness changes. Compared to a general image, the face image FIMG, the current image EIMG(0), and the past images EIMG(-1) and EIMG(-2) have less data. Therefore, the determination circuit 120 has a faster image processing speed.

[0017] Furthermore, the determination circuit 120 performs weighting calculations on the current image EIMG(0) and past images EIMG(-1) and EIMG(-2) to generate the processed image EIMG'. Therefore, the processed image EIMG' shows multiple changes in image brightness across multiple adjacent frames. Consequently, the processed image EIMG' has clearer contour features. Other features of the processed image EIMG' (skin color, facial wrinkles, etc.) are unclear. The determination circuit 120 analyzes the contour features of the processed image EIMG'. Therefore, user privacy is protected.

[0018] In this embodiment, the determination circuit 120 is, for example, a central processing unit (CPU), or other programmable general-purpose or dedicated microprocessor, digital signal processor (DSP), programmable controller, application specific integrated circuit (ASIC), programmable logic device (PLD) or a similar device or a combination of these devices.

[0019] Referring to FIGS. 1 and 2, FIG. 2 is a flowchart of an operation method according to an embodiment of the present invention. In this embodiment, the operation method S100 can be applied to the eye state detection device 100. The eye state detection device 100 can execute the commands of the operation method S100. The operation method S100 includes steps S110 to S150. In step S110, the image capture element 110 captures the face image FIMG of the current frame. In step S120, the determination circuit 120 obtains the current image EIMG(0) including the eye region from the face image FIMG. In step S130, the determination circuit 120 obtains the past images EIMG(-1), EIMG(-2) including the eye region of at least one frame before the current frame. In step S140, the determination circuit 120 performs a weighting operation on the current image EIMG(0) and the past images EIMG(-1), EIMG(-2) to generate a processed image EIMG'. In step S150, the determination circuit 120 determines the eye state within the eye region based on the processed image EIMG'. The determination circuit 120 can clearly determine the details of the eyes and the movement of the eyes based on the processed image EIMG'.

[0020] The details of the implementation of steps S110 to S150 are clearly described in the embodiment of FIG. 1, so they will not be repeated here.

[0021] Referring to FIG. 3, FIG. 3 is a schematic diagram of an eye state detection device according to an embodiment of the present invention. In this embodiment, the eye state detection device 200 includes an image capture element 110, a determination circuit 120, an infrared light source 230, and a filter element 240. The infrared light source 230 is connected to the image capture element 110. The image capture element 110 can control the infrared light source 230 to provide infrared light LIR. In some embodiments, the determination circuit 120 can control the infrared light source 230 to supply infrared light LIR. Since the operations of the image capture element 110 and the determination circuit 120 are clearly described in the embodiment of FIG. 1, they will not be repeated here.

[0022] In this embodiment, the image capture element 110 is realized by a DVS. The infrared light source 230 provides infrared light LIR. The filter element 240 filters light other than the infrared light LIR. Therefore, the face image FIMG captured by the image capture element 110 through the filter element 240 is an infrared light image. Therefore, the current image EIMG(0) and the past images EIMG(-1), EIMG(-2) are also infrared light images.

[0023] Generally, existing eye state detection methods detect the eye state using visible light images. However, existing eye state detection methods are affected by changes in ambient light, differences in the pupil color of the user, and glasses worn by the user (such as glasses with different lens colors and reflections of the lenses), making it difficult to discriminate the eye state. When applying existing eye state detection techniques to the detection of the driver's eye state, the interference caused by changes in ambient light, differences in the pupil color of the user, and glasses worn by the user becomes more prominent.

[0024] It should be noted here that the eye state detection device 200 utilizes infrared light to assist in detecting the eye state. Since infrared light penetrates eyeglass lenses, interference from ambient light and pupil color can be eliminated. In other words, the eye state detection device 200 can provide accurate eye state detection results without being affected by lenses, ambient light, or pupil color. Furthermore, the determination circuit 120 performs weighting calculations on the current image EIMG(0) and past images EIMG(-1) and EIMG(-2) to generate the processed image EIMG'.

[0025] It goes without saying that the operation method S100 shown in Figure 2 can be applied to the eye condition detection device 200.

[0026] Referring to Figures 3 and 4, Figure 4 is a flowchart of an operation method according to one embodiment of the present invention. In this embodiment, the operation method S200 can be applied to an eye state detection device 200. The eye state detection device 200 can execute the commands of the operation method S200. The operation method S200 includes steps S210 to S250. In step S210, the determination circuit 120 receives the face image FIMG of the current frame. In step S220, the determination circuit 120 detects a plurality of eye keypoints from the face image FIMG. In step S230, the determination circuit 120 acquires a region including the eye region based on the plurality of eye keypoints. In step S240, the determination circuit 120 expands the eye region to generate the current image EIMG(0).

[0027] Furthermore, referring to Figures 3, 4, 5, and 6, Figure 5 is a schematic diagram of a face image according to one embodiment of the present invention. Figure 6 is a schematic diagram of a current image according to one embodiment of the present invention. In this embodiment, Figure 5 shows the face image FIMG captured by the image capture element 110 in step S210. The image capture element 110 also operates in cooperation with the infrared light source 230 and the filter element 240, so that the captured face image FIMG is also an infrared light image. Infrared light can pass through the lenses of eyeglasses, eliminating interference from ambient light and pupil color. Therefore, the features of the eye outline in the face image FIMG become clearer.

[0028] In step S220, the determination circuit 120 detects eye keypoints P01 to P12 from the face image FIMG. For example, the determination circuit 120 can identify left eye keypoints P0 to P6 and left eye keypoints P7 to P12 from the features of the face image FIMG. In step S230, the determination circuit 120 obtains the eye region from the face image FIMGK based on the eye keypoints P01 to P12.

[0029] In step S240, the determination circuit 120 expands the eye region to generate the current image EIMG(0). In this embodiment, the range of the current image EIMG(0) is slightly larger than the eye region defined by the eye keypoints P01 to P12. In this embodiment, the current image EIMG(0) may also be an infrared eye image after filtering the eyeglass frame image.

[0030] In step S250, the determination circuit 120 acquires past images EIMG(-1) and EIMG(-2) containing the eye region from at least one frame prior to the current frame, performs a weighting calculation on the current image EIMG(0) and the past images EIMG(-1) and EIMG(-2), and generates the processed image EIMG'. The weighting calculation in step S250 is clearly explained in the embodiment of Figure 1 and will not be repeated here. Also, the method for generating the past images EIMG(-1) and EIMG(-2) may be the same as the method for generating the current image EIMG(0).

[0031] Referring to Figures 3 and 7, Figure 7 is a flowchart of an operation method according to one embodiment of the present invention. In this embodiment, operation method S300 can be applied to an eye state detection device 200. The eye state detection device 200 can execute the commands of operation method S300. Operation method S300 includes S301 to S310. In this embodiment, the eye state detection device 200 may be applied to monitor the eye state of a user while driving. The eye state detection device 200 may be installed on the steering wheel, rearview mirror, or dashboard to detect the eye state of a user while driving.

[0032] In step S301, the determination circuit 120 acquires the face image FIMG and the processed image EIMG'. Step S301 can be completed, for example, by the operation method S200. In step S302, the determination circuit 120 acquires the line of sight angle GA based on the face image FIMG and the processed image EIMG'. Based on the face image FIMG and the processed image EIMG', the determination circuit 120 can determine the angle of the user's face and the position of the pupil, and estimate the line of sight angle GA.

[0033] In step S303, the determination circuit 120 determines whether the gaze angle GA is greater than the set angle. If the gaze angle GA is less than or equal to the set angle (for example, 30°, but the present invention is not limited thereto), this indicates that the user is facing forward. Therefore, the operation method S300 returns to step S301 and acquires the face image and processed image corresponding to the next frame. If the gaze angle GA is greater than the set angle, this indicates that the user is not facing forward. Therefore, in step S304, the determination circuit 120 measures the time length during which the gaze angle GA is greater than the set angle and generates a first time length.

[0034] In step S305, the determination circuit 120 determines whether the first time length is longer than a first set time length (for example, 3 seconds, but the present invention is not limited thereto). If the first time length is less than or equal to the first set time length, this indicates that the user's gaze angle GA has returned to the set angle within the first set time length. Therefore, the operation method S300 returns to step S301 to acquire the face image and processed image corresponding to the next frame. If the first time length is longer than the first set time length, in step S306, the determination circuit 120 controls the eye state detection device 200 to provide a first warning signal. The eye state detection device 200 can use the first warning signal to prompt the user to look forward.

[0035] Furthermore, in step S307, the determination circuit 120 determines whether the eyes are closed based on the processed image EIMG'. If the determination circuit 120 determines that the eyes are open (i.e., ''no''), the operation method S300 returns to step S301 and acquires the face image and processed image corresponding to the next frame. If the determination circuit 120 determines that the eyes are closed (i.e., ''yes''), the determination circuit 120 returns to step S308, measures the duration for which the eyes were closed, and generates a second time length. In step S309, the determination circuit 120 determines whether the second time length is longer than a second set time length (for example, 3 seconds, but the present invention is not limited thereto). If the second time length is less than or equal to the second set time length, this indicates that the user opened their eyes within the second set time length. Therefore, the operation method S300 returns to step S301 and acquires the face image and processed image corresponding to the next frame. If the second time length is longer than the second set time length, the determination circuit 120 controls the eye state detection device 200 in step S310 to provide a second warning signal. The eye state detection device 200 can use the second warning signal to prompt the user to open their eyes.

[0036] In some embodiments, the eye state detection device 200 provides a first warning signal to the vehicle host. Based on the first warning signal, the vehicle host prompts the user to look ahead. In some embodiments, the eye state detection device 200 provides a second warning signal to the vehicle host. Based on the second warning signal, the vehicle host prompts the user to open their eyes.

[0037] In some embodiments, the operation method S300 does not need to include steps S302 to S306. In some embodiments, the operation method S300 does not need to include steps S307 to S310.

[0038] In summary, the eye state detection device performs weighting calculations on the current and past images to generate a processed image, and determines the eye state within the eye region based on the processed image. The processed image can clearly show the details and movement of the eye. This allows the determination circuit to provide accurate eye state detection results. In some embodiments, the image capture element is implemented by DVS. Therefore, since the face image, current image, and past image are each generated by local pixel brightness changes, the amount of data is small. Consequently, the determination circuit has a faster image processing speed. Other features of the processed image (skin color, facial wrinkles, etc.) are obscured. Therefore, user privacy is protected.

[0039] The present invention is disclosed with reference to the embodiments described above, but these do not limit the invention. Those skilled in the art can make some changes and modifications without departing from the spirit and scope of the invention. Accordingly, the scope of protection of the present invention is determined by the scope of the appended patent application. [Industrial applicability]

[0040] The eye condition detection device of the present invention can provide accurate eye condition detection results. [Explanation of symbols]

[0041] 100 Eye condition detection device 110 image capture elements 120 Judgment circuit 200-inch state detection device 230 Infrared Light Source 240 filter elements EIMG(0) Current image EIMG(-1), EIMG(-2) Past Images EIMG processed image EW1, EW2, EW3 weighted images FIMG facial image GA line of sight angle LIR infrared light Pages 1-12: Key Points How S100, S200, S300 works S110~S150, S210~S250, S301~S310 process W1, W2, W3 weights

Claims

1. An operating method used in an eye condition detection device, The eye state detection device includes an image capture element, and the eye state detection device executes the operation method. The aforementioned operation method is, The image capture element captures the face image of the current frame, Obtaining a current image including the eye region from the aforementioned facial image, To obtain at least one past image including the eye region of at least one frame prior to the current frame, A weighted calculation is performed on the current image and at least one past image to generate a processed image. The eye condition within the eye region is determined based on the processed image, The method of operation, including the operation method.

2. The image capture element is a motion vision sensor, and the face image is a motion vision sensing image. The operating method described in claim 1.

3. Obtaining the current image including the eye region from the aforementioned facial image is, The process involves detecting multiple eye keypoints from the aforementioned facial image and obtaining the eye region based on the multiple eye keypoints, Acquiring the current image based on the aforementioned eye region, The operating method according to claim 1, including the method described in claim 1.

4. Acquiring the current image based on the aforementioned eye region means that This includes expanding the eye region to generate the current image, wherein the range of the current image is greater than the range of the eye region defined by the plurality of eye keypoints. The operating method according to claim 3.

5. The at least one past image includes a first past image corresponding to the frame immediately preceding the current frame and a second past image corresponding to the frame two frames preceding the current frame, and the weighting calculation is performed on the current image and the at least one past image to generate the processed image. The current image is multiplied by a first weight to generate a first weighted image, The first past image is multiplied by a second weight to generate a second weighted image, The process involves multiplying the aforementioned second past image by a third weight to generate a third weighted image, The first weighted image, the second weighted image, and the third weighted image are superimposed to generate the processed image, The operating method according to claim 1, including the method described in claim 1.

6. The first weight is greater than the second weight and the third weight. The operating method according to claim 5.

7. Determining the line of sight angle of the eyes based on the aforementioned face image and the processed image, If the aforementioned viewing angle is greater than the set angle, time is measured to generate the first time length. If the first time length is longer than the first set time length, a first warning signal is provided. The operating method according to claim 1, further comprising:

8. Determining the eye state within the eye region based on the processed image is: Determining whether the eyes are closed based on the processed image, If it is determined that the eyes are closed, the duration for which the eyes are closed is measured, and a second duration is generated. If the second time length is longer than the second set time length, a second warning signal is provided. The operating method according to claim 1, including the method described in claim 1.

9. The eye condition detection device further includes an infrared light source and a filter element, The aforementioned operation method is, To provide infrared light using the aforementioned infrared light source, The filter element filters out light other than infrared light, It further includes, The aforementioned face image, the current image, and the at least one past image are infrared light images. The operating method according to claim 8.

10. An image capture element positioned to capture the face image of the current frame, A determination circuit connected to the image capture element, Includes, The aforementioned determination circuit is A current image including the eye region is obtained from the aforementioned facial image. Obtain at least one past image including the eye region from at least one frame prior to the current frame. A weighting operation is performed on the current image and the at least one past image to generate a processed image. Based on the processed image, the eye condition within the eye region is determined. It is configured in such a way. Eye condition detection device.

11. The image capture element is a motion vision sensor, and the face image is a motion vision sensing image. The eye condition detection device according to claim 10.

12. The determination circuit detects multiple eye keypoints from the face image, acquires a region including the eye region based on the multiple eye keypoints, and acquires the current image based on the eye region. The eye condition detection device according to claim 10.

13. The determination circuit generates the current image by expanding the eye region such that the range of the current image is greater than the range of the eye region defined by the plurality of eye keypoints. The eye condition detection device according to claim 12.

14. The at least one past image includes a first past image corresponding to the frame immediately preceding the current frame and a second past image corresponding to the frame two frames prior to the current frame. The determination circuit generates a first weighted image by multiplying the current image by a first weight, generates a second weighted image by multiplying the first past image by a second weight, generates a third weighted image by multiplying the second past image by a third weight, and generates the processed image by superimposing the first weighted image, the second weighted image, and the third weighted image. The eye condition detection device according to claim 10.

15. The first weight is greater than the second weight and the third weight. The eye condition detection device according to claim 14.

16. The determination circuit determines the line of sight angle of the eyes based on the face image and the processed image, The determination circuit, if the viewing angle is greater than the set angle, measures the time and generates a first time length. The determination circuit controls the eye state detection device and provides a first warning signal if the first time length is longer than the first set time length. The eye condition detection device according to claim 10.

17. The determination circuit determines whether the eye is closed based on the processed image, If the determination circuit determines that the eye is closed, the determination circuit measures the duration for which the eye is closed and generates a second duration. The determination circuit, when the second time length is longer than the second set time length, controls the eye state detection device to provide a second warning signal. The eye condition detection device according to claim 10.

18. The aforementioned eye state detection device is An infrared light source arranged to provide infrared light, A filter element arranged to filter out light other than the aforementioned infrared light, It further includes, The image capture element captures the face image via the filter element, The aforementioned face image, the aforementioned current image, and the aforementioned at least one past image are each infrared light images. The eye condition detection device according to claim 10.

Images