Electronic device and method for measuring closing strength of eyelid

Abstract

The invention provides an electronic device and method for measuring eyelid closing strength. The method includes: acquiring an image of an eye; detecting at least one wrinkle in the image; determining eyelid closing strength according to the area of the at least one wrinkle; and outputting the eyelid closing intensity.

Description

Technical Field

[0001] This invention relates to image processing, and more particularly to an electronic device and method for measuring eyelid closure strength. Background Technology

[0002] In recent years, extended reality (XR) technologies, including eye-tracking, have been widely applied across various fields. For example, XR devices can track a user's eye movements to enable interaction within an XR environment. Additionally, image extraction devices can extract a user's facial expressions (e.g., open or closed eyes) and synchronously map these expressions onto the user's virtual body. However, current eye-tracking technologies used in XR devices have several drawbacks. For instance, image processing techniques can identify whether a user's eyes are closed based on the shape of their eyebrows. However, if the field of view (FoV) of the eye-tracking device is too small, below-screen type eye-tracking devices may face challenges in extracting images of the user's eyebrows. Determining the degree to which a user's eyes are closed is a significant challenge in the field of eye-tracking technology. Summary of the Invention

[0003] This invention relates to an electronic device and method for measuring eyelid closure strength.

[0004] This invention relates to an electronic device for measuring eyelid closure strength. The electronic device includes a transceiver and a processor. The processor is coupled to the transceiver, wherein the processor is configured to: acquire an image of the eye via the transceiver; detect at least one wrinkle in the image; determine the eyelid closure strength based on the area of ​​the at least one wrinkle; and output the eyelid closure strength via the transceiver.

[0005] This invention relates to a method for measuring eyelid closure strength. The method includes: acquiring an image of the eye; detecting at least one wrinkle in the image; determining the eyelid closure strength based on the area of ​​the at least one wrinkle; and outputting the eyelid closure strength.

[0006] Based on the above, the present invention can measure the eyelid closure strength of a user based on the image of the user's eyes.

[0007] To make the above content clearer and easier to understand, several embodiments will be described in detail below with reference to the accompanying drawings. Attached Figure Description

[0008] The accompanying drawings are included to provide a further understanding of this disclosure and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0009] Figure 1 A block diagram of an electronic device for measuring eyelid closure strength according to an embodiment of the present invention is shown.

[0010] Figure 2 A schematic diagram of different types of eyelid closure according to an embodiment of the present invention is shown.

[0011] Figure 3 A flowchart for determining eyelid closure strength according to an embodiment of the present invention is shown.

[0012] Figure 4 A schematic diagram illustrating the training and use of a machine learning model according to an embodiment of the present invention is shown.

[0013] Figure 5 A flowchart of a method for measuring eyelid closure strength according to an embodiment of the present invention is shown. Detailed Implementation

[0014] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same element references are used in the drawings and description to denote the same or similar parts.

[0015] Figure 1 A block diagram of an electronic device 100 for measuring eyelid closure strength according to an embodiment of the present invention is shown. The electronic device 100 may include a processor 110, a storage medium 120, and a transceiver 130. The electronic device 100 may be implemented in, for example, an XR system (e.g., a virtual reality (VR) system, an augmented reality (AR) system, or a mixed reality (MR) system).

[0016] Processor 110 may be, for example, a general-purpose or special-purpose central processing unit (CPU) or other programmable microcontroller unit (MCU), microprocessor, digital signal processor (DSP), programmable controller, application-specific integrated circuit (ASIC), graphics processing unit (GPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field-programmable gate array (FPGA), or other similar device or combination thereof. Processor 110 may be coupled to storage medium 120 and transceiver 130.

[0017] Storage medium 120 may be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or similar elements or combinations thereof, configured to record multiple modules or various applications executable by processor 110. In one embodiment, storage medium 120 may store machine learning model 121.

[0018] Transceiver 130 can be configured to transmit or receive wired or wireless signals. Transceiver 130 can also perform low-noise amplification, impedance matching, mixing, up or down frequency conversion, filtering, amplification, and other operations.

[0019] Figure 2 Schematic diagrams of different types of eyelid closure according to an embodiment of the present invention are shown. Image 210 shows a person closing their eyes normally. That is, the person closes their eyes without applying force to their eyelids. Accordingly, no wrinkles appear around the person's eyes. On the other hand, image 220 shows a person with their eyes tightly closed. That is, when the person closes their eyes, they apply a great deal of force to their eyelids. Therefore, one or more wrinkles 20 appear around the person's eyes. In summary, the appearance of wrinkles around a person's eyes can be related to the strength of eyelid closure. The present invention provides a method for measuring eyelid closure strength based on wrinkles around the human eyes.

[0020] Figure 3 A flowchart illustrating the determination of eyelid closure strength according to an embodiment of the present invention is shown, wherein the steps of the flowchart can be determined by… Figure 1 The electronic device 100 shown is used to implement this.

[0021] In step S301, the processor 110 can acquire an image of the eye via the transceiver 130. The processor 110 can detect one or more wrinkles in the image.

[0022] In one embodiment, processor 110 may acquire a raw image containing the user's face. Processor 110 may then perform object detection on the raw image to identify the user's eyes and extract images of the eyes accordingly from the raw image.

[0023] In one embodiment, processor 110 may perform edge detection on the image to detect wrinkles in the image. For example, processor 110 may perform Canny edge detection on the image to detect wrinkles in the image. Specifically, processor 110 may detect the image (e.g., by applying object detection) to obtain a region of interest (ROI) corresponding to the user's eye. Processor 110 may perform Gaussian blur on the ROI to obtain a blurred image. After obtaining the blurred image, processor 110 may calculate the gradient of each pixel of the blurred image according to the Sobel operator, as shown in equations (1) to (3), where A represents the blurred image and G represents the pixel gradient in the blurred image. After obtaining the gradient of each pixel of the blurred image, processor 110 may perform non-maximum suppression on each pixel gradient of the blurred image to obtain one or more edges. Processor 110 may set one or more edges as one or more wrinkles.

[0024]

[0025] In one embodiment, processor 110 can detect wrinkles in an image by using machine learning (ML) model 121. Figure 4A schematic diagram illustrating the training and use of a machine learning model 121 according to an embodiment of the present invention is shown. The processor 110 can train the ML model 121 based on a supervised learning algorithm using a set of training images, where each training image can be labeled with one or more actual wrinkles. The loss function of the supervised learning algorithm can be binary cross entropy (BCE), as shown in equation (4), where Loss represents BCE, T(x, y) represents the pixel at coordinate (x, y) in the image labeled with actual wrinkles, and P(x, y) represents the pixel at coordinate (x, y) in the image labeled with estimated wrinkles (i.e., the image output by the ML model 121). After training the ML model 121, the processor 110 can input an image of the eye into the ML model 121 to output one or more estimated wrinkles in the image of the eye.

[0026]

[0027] return Figure 3 In step S302, the processor 110 can determine whether the eyes in the image are closed. If the eyes in the image are closed, then step S304 is executed. If the eyes in the image are not closed, then step S303 is executed. In one embodiment, the processor 110 can determine whether the eyes in the image are closed by performing image recognition on the image.

[0028] In step S303, the processor 110 can determine that the user's eyelids are open.

[0029] In step S304, processor 110 may compare the area of ​​one or more wrinkles with a reference value. Processor 110 may determine whether the area of ​​one or more wrinkles is greater than the reference value, wherein the reference value may be correlated with the area of ​​wrinkles when a person normally closes their eyes. If the area of ​​one or more wrinkles is greater than the reference value, step S306 is executed. If the area of ​​one or more wrinkles is less than or equal to the reference value, step S305 is executed. In one embodiment, processor 110 may calculate the area of ​​one or more wrinkles based on the number of pixels in one or more wrinkles.

[0030] In step S305, processor 110 can determine that the user has closed their eyes normally. That is, the user closes their eyes without applying force to their eyelids. Processor 110 can determine the eyelid closure strength based on a default value, where the default value indicates that no significant force is applied to the eyelids in the image. Processor 110 can output the eyelid closure strength via transceiver 130. For example, processor 110 can output the eyelid closure strength to a virtual avatar system, and the virtual avatar system can update the virtual avatar's expression (e.g., facial features or emotions) based on the eyelid closure strength.

[0031] In step S306, processor 110 may determine eyelid closure strength based on the difference between the area of ​​one or more wrinkles and a reference value, as shown in equation (5), where D represents eyelid closure strength, A1 represents the area of ​​one or more wrinkles, and A2 is the reference value. The reference value may be correlated with the area of ​​wrinkles that appear when a person normally closes their eyes. Processor 110 may output eyelid closure strength via transceiver 130.

[0032] D = A1 - A2…(5)

[0033] Figure 5 A flowchart of a method for measuring eyelid closure strength according to an embodiment of the present invention is shown, wherein the method can be performed by... Figure 1 The electronic device 100 shown is implemented. In step S501, an image of the eye is acquired. In step S502, at least one wrinkle in the image is detected. In step S503, the eyelid closure strength is determined based on the area of ​​the at least one wrinkle. In step S504, the eyelid closure strength is output.

[0034] In summary, this invention provides a method for measuring the strength of a user's eyelid closure, where the eyelid closure strength represents the force with which the user closes their eyes. The electronic device of this invention can determine the eyelid closure strength based solely on an image of the eyes. Since the image of the eyes does not necessarily include other parts of the user's face (e.g., eyebrows), the FoV of the camera extracting the image does not need to be large. For example, the method of this invention can be applied to images extracted by under-screen eye-tracking devices. Furthermore, the image processing of this invention requires fewer computational resources. The output of this invention can be applied to various fields, such as XR technology or virtual avatar technology.

[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An electronic device for measuring eyelid closure strength, characterized in that, include: transceiver; as well as A processor, coupled to the transceiver, wherein the processor is configured to: The transceiver acquires an image of the eye; Detect at least one wrinkle in the image; The eyelid closure strength is determined based on the area of ​​the at least one wrinkle; and The eyelid closure strength is output through the transceiver.

2. The electronic device of claim 1, wherein the processor is further configured to: The area of ​​the at least one wrinkle is compared with a reference value to determine the eyelid closure strength.

3. The electronic device of claim 2, wherein the processor is further configured to: In response to the area being greater than the reference value, the eyelid closure strength is determined based on the difference between the area and the reference value.

4. The electronic device of claim 2, wherein the processor is further configured to: In response to the area being less than or equal to the reference value, the eyelid closure strength is determined according to a default value.

5. The electronic device of claim 1, wherein the processor is further configured to: Edge detection is performed on the image to detect the at least one wrinkle.

6. The electronic device of claim 5, wherein the processor is further configured to: The image is detected to obtain the region of interest; Gaussian blur is applied to the region of interest to obtain a blurred image; Calculate the gradient of each pixel in the blurred image; Non-maximum suppression is performed on the gradient of each pixel to obtain at least one edge; and The at least one edge is defined as the at least one wrinkle.

7. The electronic device of claim 1, further comprising: Storage medium, coupled to the processor and storing machine learning models, wherein the processor is further configured to: The image is input into the machine learning model to output the at least one wrinkle in the image.

8. The electronic device of claim 7, wherein the processor is further configured to: The machine learning model is trained based on training images labeled with actual wrinkles using a supervised learning algorithm.

9. The electronic device of claim 1, wherein the processor is further configured to: The area of ​​the at least one wrinkle is calculated based on the number of pixels in the at least one wrinkle.

10. A method for measuring eyelid closure strength, characterized in that, include: Obtain an image of the eye; Detect at least one wrinkle in the image; The eyelid closure strength is determined based on the area of ​​at least one wrinkle; as well as Output the eyelid closure strength.

Images