Eyelid distance calculation method based on eye image and related device

Abstract

The present application relates to artificial intelligence and digital medicine, and provides an eyelid distance calculation method based on eye images and related equipment, wherein the method trains an eye segmentation model based on obtained multiple human eye videos; then uses the eye segmentation model to segment the eye region in the human eye image to be measured, to obtain a binary human eye image to be measured; then performs horizontal correction on the human eye image to be measured, to obtain a corrected human eye image; and finally, the distance of the eyelid in the human eye image to be measured can be obtained based on the corrected human eye image, thereby improving the accuracy of the eyelid distance calculation. The present application can be applied to the medical field and the automatic driving field.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method, apparatus, electronic device, and storage medium for calculating eyelid distance based on eye images. Background Technology

[0002] Due to the widespread use and rapid updates of electronic products, the incidence of dry eye syndrome remains high. Dry eye examinations rely on specialized equipment and require medical diagnosis, resulting in high time costs. While blinking frequency and incomplete blinking frequency can be used for preliminary analysis of dry eye risk and can be captured using ordinary cameras, manual analysis is difficult due to the extremely short duration of each blink. Therefore, an automated eyelid closure analysis method is needed to quickly and accurately reflect blinking status.

[0003] In the process of developing this invention, the inventors discovered that in the prior art, eye image classification technology is used to determine whether a video frame is in an open or closed state, thereby calculating the blink frequency. However, image classification technology is prone to confusing video frames with incompletely closed eyes with those with closed eyes, and it cannot detect incomplete blinking processes related to dry eye. Alternatively, morphological features of eye images can be used to measure the degree of eyelid closure using traditional image algorithms, but this method has poor accuracy and versatility. Summary of the Invention

[0004] In view of the above, it is necessary to propose a method, device, electronic device and storage medium for calculating eyelid distance based on eye images, which can improve the accuracy of eyelid distance calculation based on eye images.

[0005] A first aspect of the present invention provides a method for calculating eyelid distance based on eye images, the method comprising:

[0006] Acquire multiple human eye videos and train an eye segmentation model based on the multiple human eye videos;

[0007] In response to a command to obtain the distance of the eyelid in the image of the human eye to be tested, the eye segmentation model is used to segment the eye region in the image of the human eye to be tested to obtain a binarized image of the human eye to be tested.

[0008] The image of the eye to be tested is horizontally corrected to obtain a corrected eye image;

[0009] The distance between the eyelids in the image of the subject's eye is obtained based on the corrected eye image.

[0010] According to an optional embodiment of the present invention, the step of horizontally correcting the image of the eye to be tested to obtain a corrected image of the eye includes:

[0011] Obtain multiple connected regions in the eye image to be tested;

[0012] Calculate the area of ​​each of the connected regions;

[0013] The target connectivity area is the largest available connectivity area.

[0014] Obtain the minimum and maximum x-coordinates of the pixels in the target connected region;

[0015] Calculate the angle between the line connecting the minimum abscissa point and the maximum abscissa point and the horizontal line;

[0016] The corrected eye image is obtained by rotating the image of the eye to be tested counterclockwise according to the included angle.

[0017] According to an optional embodiment of the present invention, obtaining the distance between the eyelids in the image of the subject eye based on the corrected eye image includes:

[0018] Obtain the minimum and maximum ordinate points of the pixels in the target connected region;

[0019] The distance between the eyelids in the image of the human eye under test is calculated based on the minimum and maximum ordinate points.

[0020] According to an optional embodiment of the present invention, the eye segmentation model obtained by training based on the plurality of human eye videos includes:

[0021] Obtain the original sampling frame rate for each of the aforementioned human eye videos;

[0022] The original sampling frame rate is compared with multiple preset sampling frame rate thresholds to obtain the comparison results;

[0023] Based on the comparison results, the original sampling frame rate is updated to the target sampling frame rate;

[0024] The target sampling frequency is used to sample the corresponding human eye video to obtain multiple human eye video frames;

[0025] The eye region in each of the aforementioned human eye video frames is labeled;

[0026] An eye segmentation model was obtained by training multiple video frames of human eyes labeled with eye regions.

[0027] According to an optional embodiment of the present invention, the plurality of preset sampling frame rate thresholds include a preset first sampling frame rate threshold and a preset second sampling frame rate threshold, wherein the preset first sampling frame rate threshold is less than the preset second sampling frame rate threshold; updating the original sampling frame rate to the target sampling frame rate according to the comparison result includes:

[0028] When the original sampling frame rate is less than the preset first sampling frame rate threshold, calculate a first ratio of the preset first sampling frame rate threshold to the original sampling frame rate, and calculate a second ratio of the preset second sampling frame rate threshold to the original sampling frame rate;

[0029] A first update factor is generated based on the first ratio and the second ratio;

[0030] The original sampling frame rate is updated to the target sampling frame rate using the first update factor.

[0031] According to an optional embodiment of the present invention, updating the original sampling frame rate to the target sampling frame rate based on the comparison result further includes:

[0032] When the original sampling frame rate is greater than the preset second sampling frame rate threshold, calculate the third ratio of the preset first sampling frame rate threshold to the original sampling frame rate, and calculate the fourth ratio of the preset second sampling frame rate threshold to the original sampling frame rate.

[0033] A second update factor is generated based on the third ratio and the fourth ratio;

[0034] The original sampling frame rate is updated to the target sampling frame rate using the second update factor.

[0035] According to an optional embodiment of the present invention, the eye segmentation model obtained by training based on multiple human eye video frames labeled with eye regions includes:

[0036] An encoder is constructed using an overlapping block merging module with downsampling capabilities and an efficient self-attention transformer module capable of global modeling representation;

[0037] A decoder is constructed using a block expansion module with linear upsampling capability, the efficient self-attention transformer module, and a jump connection module capable of fusing high- and low-level features;

[0038] An eye segmentation model is constructed using the encoder and the decoder;

[0039] A loss function is constructed, and the eye segmentation model is iteratively trained using the multiple human eye video frames labeled with eye regions.

[0040] A second aspect of the present invention provides an eyelid distance calculation device based on an eye image, the device comprising:

[0041] The training module is used to acquire multiple human eye videos and train an eye segmentation model based on the multiple human eye videos;

[0042] The segmentation module is used to segment the eye region in the image of the human eye under test using the eye segmentation model when responding to the instruction to obtain the distance of the eyelid in the image of the human eye under test, so as to obtain a binarized image of the human eye under test.

[0043] The correction module is used to perform horizontal correction on the eye image to be tested, so as to obtain a corrected eye image;

[0044] The calculation module is used to obtain the distance between the eyelids in the image of the subject's eye based on the corrected eye image.

[0045] A third aspect of the present invention provides an electronic device comprising a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the described eyelid distance calculation method based on eye images.

[0046] A fourth aspect of the present invention is a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the eyelid distance calculation method based on eye images.

[0047] This invention is based on a deep learning eye segmentation model, which has higher accuracy and wider applicability. By performing horizontal correction on the obtained eye image, it can avoid calculation errors caused by shooting angle, shooting distance, etc., and can accurately locate the maximum distance between the upper and lower eyelids. Attached Figure Description

[0048] Figure 1 This is a flowchart of the eyelid distance calculation method based on eye images provided in Embodiment 1 of the present invention.

[0049] Figure 2 This is a schematic diagram of the framework of the eyelid distance calculation method based on eye images provided in an embodiment of the present invention.

[0050] Figure 3 This is a structural diagram of the eyelid distance calculation device based on eye images provided in Embodiment 2 of the present invention.

[0051] Figure 4 This is a schematic diagram of the structure of the electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0052] To better understand the above-mentioned objects, features, and advantages of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing an embodiment in one alternative implementation and is not intended to be limiting of the invention.

[0054] The eyelid distance calculation method based on eye images provided in this embodiment of the invention is executed by an electronic device, and correspondingly, the eyelid distance calculation device based on eye images runs in the electronic device.

[0055] This invention provides a standardized approach to symptom processing based on artificial intelligence (AI) technology. AI is the theory, method, technology, and application system that uses digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0056] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0057] Example 1

[0058] Figure 1 This is a flowchart of an eyelid distance calculation method based on eye images provided in Embodiment 1 of the present invention. The eyelid distance calculation method based on eye images specifically includes the following steps. Depending on different needs, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0059] S11, acquire multiple human eye videos, and train an eye segmentation model based on the multiple human eye videos.

[0060] Electronic devices can capture videos of multiple objects including their eye movements. For ease of description, videos including eye movements are referred to as human eye videos. One or more human eye videos of each object can be captured. Specifically, human eye videos can be captured using the same or different camera devices (e.g., head-mounted cameras) and then transmitted to the electronic device.

[0061] When this invention is applied to a medical setting, the human eye video can be obtained from a digital medical database. This digital medical database can be a digital repository storing patient medical records in a hospital, or it can be a networked database of multiple hospitals; this invention does not impose any limitations.

[0062] After acquiring multiple human eye videos, electronic devices can train on these videos to obtain an eye segmentation model, which can then be used to segment the input human eye image.

[0063] In an optional implementation, the eye segmentation model trained based on the plurality of human eye videos includes:

[0064] Obtain the original sampling frame rate for each of the aforementioned human eye videos;

[0065] The original sampling frame rate is compared with multiple preset sampling frame rate thresholds to obtain the comparison results;

[0066] Based on the comparison results, the original sampling frame rate is updated to the target sampling frame rate;

[0067] The target sampling frequency is used to sample the corresponding human eye video to obtain multiple human eye video frames;

[0068] The eye region in each of the aforementioned human eye video frames is labeled;

[0069] An eye segmentation model was obtained by training multiple video frames of human eyes labeled with eye regions.

[0070] The original sampling frame rate is the number of images captured by the camera device that captures human eye video per unit time.

[0071] The multiple preset sampling frame rate thresholds are pre-set critical values ​​used to determine whether the original sampling frame rate of the human eye video meets the requirements. These preset sampling frame rate thresholds may include a preset first sampling frame rate threshold and a preset second sampling frame rate threshold, wherein the preset first sampling frame rate threshold is less than the preset second sampling frame rate threshold. For example, the preset first sampling frame rate threshold may be set to 10 fps, and the preset second sampling frame rate threshold may be set to 20 fps.

[0072] The electronic device obtains a comparison result by acquiring the raw sampling frame rate of a human eye video and comparing it with multiple preset sampling frame rate thresholds. The comparison result may include: the raw sampling frame rate of the human eye video is less than the preset first sampling frame rate threshold; the raw sampling frame rate of the human eye video is greater than the preset first sampling frame rate threshold but less than the preset second sampling frame rate threshold; or the raw sampling frame rate of the human eye video is greater than the preset second sampling frame rate threshold.

[0073] Since a blink takes 0.2-0.4 seconds, if the original sampling frame rate of the human eye video is less than the preset first sampling frame rate threshold, then using an original sampling frame rate lower than the preset first sampling frame rate threshold to sample the human eye video may result in multiple human eye video frames being extracted from the human eye video missing the closed eye state, meaning the actual distance of the eyelids cannot be effectively detected. Therefore, it is necessary to update the original sampling frame rate (which is lower than the preset first sampling frame rate threshold) and use the updated sampling frame rate to sample the human eye video, thereby solving the problem of the extracted human eye video frames missing the closed eye state.

[0074] If the original sampling frame rate of the human eye video is greater than the preset second sampling frame rate threshold, although there is no problem of missing the closed-eye state in the extracted human eye video frames, using a larger original sampling frame rate to sample the human eye video results in a large number of human eye video frames. Calculating eyelid distance on a large number of human eye video frames increases the number of calculations and slows down the analysis speed of the human eye video. Therefore, it is necessary to update the original sampling frame rate that is greater than the preset second sampling frame rate threshold, and then use the updated sampling frame rate to sample the human eye video, thereby solving the problem of slow computation efficiency due to the large number of extracted human eye video frames.

[0075] When the original sampling frame rate of the human eye video is greater than the preset first sampling frame rate threshold but less than the preset second sampling frame rate threshold, it indicates that sampling the human eye video using the original sampling frame rate will result in a number of human eye video frames that are neither too many nor will it miss the closed-eye state. For the sake of consistency, when the original sampling frame rate of the human eye video is greater than the preset first sampling frame rate threshold but less than the preset second sampling frame rate threshold, the original sampling frame rate is taken as the target sampling frame rate.

[0076] In an optional implementation, updating the original sampling frame rate to the target sampling frame rate based on the comparison result includes:

[0077] When the original sampling frame rate is less than the preset first sampling frame rate threshold, calculate a first ratio of the preset first sampling frame rate threshold to the original sampling frame rate, and calculate a second ratio of the preset second sampling frame rate threshold to the original sampling frame rate;

[0078] A first update factor is generated based on the first ratio and the second ratio;

[0079] The original sampling frame rate is updated to the target sampling frame rate using the first update factor.

[0080] For example, if the original sampling frame rate is f01, the preset first sampling frame rate threshold is f11, and the preset second sampling frame rate threshold is f21, and f01 < f11 < f21, then the calculated first ratio is f11 / f01, and the calculated second ratio is f21 / f01. Obviously, f21 / f01 > f11 / f01 > 1. The electronic device can use a random function to generate a random number between the first ratio f11 / f01 and the second ratio f21 / f01. The generated random number is used as the first update factor. The product of the first update factor and the original sampling frame rate is calculated as the target sampling frame rate.

[0081] In the above optional implementation, when the original sampling frame rate is less than the preset first sampling frame rate threshold, a first update factor is generated based on the first ratio of the preset first sampling frame rate threshold to the original sampling frame rate and a second ratio of the preset second sampling frame rate threshold to the original sampling frame rate. The original sampling frame rate is then updated using the first update factor. Since the first update factor is greater than 1, updating the original sampling frame rate using the first update factor is equivalent to increasing the original sampling frame rate. Therefore, the updated target sampling frame rate is between the preset first sampling frame rate threshold and the preset second sampling frame rate threshold. Thus, when sampling human eye video using the target sampling frame rate, the number of multiple human eye video frames obtained is appropriate, and the closed-eye state is not missed.

[0082] In an optional implementation, updating the original sampling frame rate to the target sampling frame rate based on the comparison result further includes:

[0083] When the original sampling frame rate is greater than the preset second sampling frame rate threshold, calculate the third ratio of the preset first sampling frame rate threshold to the original sampling frame rate, and calculate the fourth ratio of the preset second sampling frame rate threshold to the original sampling frame rate.

[0084] A second update factor is generated based on the third ratio and the fourth ratio;

[0085] The original sampling frame rate is updated to the target sampling frame rate using the second update factor.

[0086] For example, if the original sampling frame rate is f02, the preset first sampling frame rate threshold is f12, and the preset second sampling frame rate threshold is f22, and f02 > f22 > f12, then the calculated first ratio is f12 / f02, and the calculated second ratio is f22 / f02. Obviously, f12 / f02 < f22 / f02 < 1. The electronic device can use a random function to generate a random number between the first ratio f12 / f02 and the second ratio f22 / f02. The generated random number is used as the second update factor. The product of the second update factor and the original sampling frame rate is calculated as the target frame rate.

[0087] In the above optional implementation, when the original sampling frame rate is greater than the preset first sampling frame rate threshold, a second update factor is generated based on the third ratio of the preset first sampling frame rate threshold to the original sampling frame rate and the fourth ratio of the preset second sampling frame rate threshold to the original sampling frame rate. The original sampling frame rate is then updated using the second update factor. Since the second update factor is less than 1, updating the original sampling frame rate using the second update factor is equivalent to reducing the original sampling frame rate. Therefore, the updated target sampling frame rate is between the preset first sampling frame rate threshold and the preset second sampling frame rate threshold. Thus, when sampling the human eye video using the target sampling frame rate, the number of multiple human eye video frames obtained is appropriate, and the closed-eye state is not missed.

[0088] In an optional implementation, the eye segmentation model obtained by training multiple human eye video frames labeled with eye regions includes:

[0089] An encoder is constructed using an overlapping block merging module with downsampling capabilities and an efficient self-attention transformer module capable of global modeling representation;

[0090] A decoder is constructed using a block expansion module with linear upsampling capability, the efficient self-attention transformer module, and a jump connection module capable of fusing high- and low-level features;

[0091] An eye segmentation model is constructed using the encoder and the decoder;

[0092] A loss function is constructed, and the eye segmentation model is iteratively trained using the multiple human eye video frames labeled with eye regions.

[0093] After obtaining multiple human eye video frames, electronic devices can use annotation tools to annotate the eye region between the upper and lower eyelid edges in each human eye video frame. The human eye video frames with annotated eye regions can then be used as training samples to train the eye segmentation model.

[0094] Electronic devices can use SegFormer-B0 (Simple and Efficient Design for Semantic Segmentation with Transformers) as the network architecture to train an eye segmentation model. SegFormer is an efficient and powerful semantic segmentation framework composed of Transformer and a lightweight multilayer perceptron (MLP). It employs the Transformer architecture to achieve multi-scale self-attention feature extraction, containing rich semantic information to ensure a high-accuracy eye segmentation model. The eye segmentation model training uses a similarity metric as the loss function, and the Adam optimizer iteratively updates the network weights with a learning rate of 0.0001.

[0095] In one optional implementation, before inputting the training samples into the eye segmentation model, the training data can be augmented by processes such as flipping, rotating, and brightness transformation. The augmented training data is then input into the eye segmentation model for training.

[0096] For a training sample with an input height of H and a width of W, an encoder composite function is constructed. Each feature map corresponds to a first sub-function. Each first sub-function of the encoder consists of one overlapping block merging module and two efficient Transformer modules, which are called sub-coding blocks. Each sub-coding block can halve the spatial dimension of the input feature map. Thus, the four sub-coding blocks form four consecutive stages of the encoder, which can obtain four feature maps of different scales to provide features of different abstraction levels.

[0097] Within each sub-encoding block, the overlapping block merging module performs matrix multiplication on the input training samples using an overlapping sliding window approach, thereby achieving linear embedding of the feature maps and downsampling the spatial resolution dimension of the input data by a factor of 2. The efficient Transformer module includes a self-attention sub-module and a feedforward neural network sub-module, used to learn global and fused features between different locations in the training samples.

[0098] For the multi-stage encoded features, a decoder function is constructed. The decoding process adopts a symmetrical approach to the encoding process. The decoder function includes four second sub-functions, which together constitute the four stages of the decoder. Each second sub-function is constructed using a block expansion module based on fully connected layer upsampling and two efficient Transformer modules, referred to as a sub-decoding block. The block expansion module trades channel dimension for spatial dimension. It doubles the channel dimension of the data through a fully connected layer before performing a dimension reshaping operation. Specifically, it splits the channels into four parts, and interleaves every two parts along the spatial dimension, thereby doubling the spatial dimension while halving the channel dimension.

[0099] Meanwhile, in the corresponding stages of the encoder and decoder, a "skip connection" structure is used to connect and splice the features of the encoder and decoder, so that the decoder can receive features from the encoder in the same stage for fusion, in order to fuse different semantic features at shallow coarse-grained and high-level fine-grained levels.

[0100] Pixel-level cross-entropy loss (CE) is chosen as the primary optimization objective during model training. To allow the model to examine region relevance, a similarity metric (Dice) loss is used as an auxiliary method. For example, the loss function is: Loss = 0.7 * CE + 0.3 * Dice.

[0101] By calculating the gradient of the loss function, the network parameters of the eye segmentation model are updated along the negative gradient direction to achieve a continuous decrease in the loss function.

[0102] The above implementation method can combine local attention information and global attention information to render features, making feature extraction more efficient and thus improving the accuracy of the eye segmentation model.

[0103] S12, in response to the instruction to obtain the distance of the eyelid in the image of the human eye to be tested, the eye region in the image of the human eye to be tested is segmented using the eye segmentation model to obtain a binarized image of the human eye to be tested.

[0104] The image of the subject's eye refers to the image of the eye from which the distance between the upper and lower eyelids needs to be calculated. This image can be either the left or right eye of the subject.

[0105] When an electronic device receives an image of the subject's eye, it triggers a command to acquire the distance between the eyelids in the image. In response to this command, the image is input into an eye segmentation model. The model identifies the eye region within the image, obtaining the area between the upper and lower eyelids. This area is then cropped from the original image to obtain the final eye image. The area between the upper and lower eyelids is the eyeball region or the open eye region.

[0106] It should be noted that the image of the eye being tested is a binarized image.

[0107] S13, perform horizontal correction on the eye image to be tested to obtain a corrected eye image.

[0108] Because the subject may be tilted or have droopy eyelids due to old age when the camera device acquires the image of the subject's eye, the eyelids will be tilted to varying degrees in the binarized image of the subject's eye output by the eye segmentation model. Therefore, the electronic device needs to perform horizontal correction on the image of the subject's eye so that the line connecting the two corners of the eye is parallel to the coordinate axis of the preset image coordinate system.

[0109] In an optional implementation, the step of horizontally correcting the image of the eye to be tested to obtain a corrected image of the eye includes:

[0110] Obtain multiple connected regions in the eye image to be tested;

[0111] Calculate the area of ​​each of the connected regions;

[0112] The target connectivity area is the largest available connectivity area.

[0113] Obtain the minimum and maximum x-coordinates of the pixels in the target connected region;

[0114] Calculate the angle between the line connecting the minimum abscissa point and the maximum abscissa point and the horizontal line;

[0115] The corrected eye image is obtained by rotating the image of the eye to be tested counterclockwise according to the included angle.

[0116] The eye image under test contains multiple connected regions. The electronic device can calculate the area of ​​the connected region by counting the number of pixels in the connected region, and determine the connected region with the largest area as the target connected region.

[0117] After obtaining the target connected region, it is retained while the remaining connected regions are deleted, thus achieving horizontal correction of the eye image under test based on the target connected region. The minimum abscissa of the pixels in the target connected region is the left corner of the eye image under test, and the maximum abscissa of the pixels in the target connected region is the right corner of the eye image under test. The angle θ between the line connecting the left and right corners of the eye and the horizontal line is calculated. The eye image under test is rotated counterclockwise by θ to complete the horizontal correction.

[0118] S14, Based on the corrected eye image, obtain the distance between the eyelids in the image of the subject's eye.

[0119] In an optional implementation, obtaining the distance between the eyelids in the test eye image based on the corrected eye image includes:

[0120] Obtain the minimum and maximum ordinate points of the pixels in the target connected region;

[0121] The distance between the eyelids in the image of the human eye under test is calculated based on the minimum and maximum ordinate points.

[0122] The electronic device acquires the minimum and maximum ordinates of pixels in the target connected region of the corrected eye image. The minimum ordinate represents the highest point of the upper eyelid, and the maximum ordinate represents the lowest point of the lower eyelid. The vertical distance between the horizontal line containing the highest point of the upper eyelid and the horizontal line containing the lowest point of the lower eyelid is calculated to obtain the eyelid distance in the image of the subject's eye. Here, the eyelid distance represents the opening degree between the upper and lower eyelids.

[0123] Due to limitations in image recognition accuracy, the identified upper and lower eyelids may not be smooth, for example, with the appearance of an incorrect point, resulting in abnormal curvature of the upper and lower eyelids, which is clearly unrealistic. In an alternative implementation, the electronic device can smooth the upper and lower eyelids in the image of the subject's eye to obtain images that more closely resemble reality, thereby improving the accuracy of subsequent measurements.

[0124] The method described in this invention can be applied to the field of digital medical technology, promoting the development of smart cities. For example, by plotting a waveform image of the distance between the eyelids (i.e., the opening and closing degree between the upper and lower eyelids) over time, the waveform change characteristics can be used to indirectly assess ocular surface diseases, such as dry eye syndrome and keratoconjunctivitis, thereby aiding in the diagnosis and treatment of ocular surface diseases. The following is a related description... Figure 2 The process is described as follows: Acquire eye videos of the patient over a period of time. Frames are extracted according to ①, resulting in multiple single-frame images. Each single-frame image is then input into a trained eye segmentation model according to ②. The model segments the eye region within the input single-frame images to obtain a prediction image. This prediction image is a binary eye image containing two connected regions. Then, according to ③, the image corresponding to the largest connected region is selected for horizontal correction to obtain the corrected eye image. Finally, according to ④, the minimum ordinate of the pixels in the target connected region is used as the highest point of the upper eyelid, and the maximum ordinate is used as the lowest point of the lower eyelid. The vertical distance between the horizontal line containing the highest point of the upper eyelid and the horizontal line containing the lowest point of the lower eyelid is calculated to obtain the distance of the patient's eyelids. The eyelid distances corresponding to each single-frame image are then plotted as waveform images of the eyelids in chronological order.

[0125] The method described in this invention, in addition to being used to calculate the distance between the upper and lower eyelids to assess eye risks, can also be applied to fatigue detection in the field of autonomous driving.

[0126] For example, an electronic device acquires video of the driver's eyes within a preset time period. Based on any frame rate between a preset first sampling frame rate threshold and a preset second sampling frame rate threshold, it samples the video of the driver's eyes within the preset time period, obtaining multiple video frames of the driver's eyes to be tested. The method described in this invention is used to obtain the distance between the upper and lower eyelids in each video frame of the driver's eyes to be tested. Further, the electronic device acquires target distances among the distances between the upper and lower eyelids in the video frames of the driver's eyes to be tested that are less than a preset distance threshold. A first number of target distances is calculated, and a second number of these target distances is used to determine the driving state of the driver under test based on this ratio.

[0127] Specifically, if the ratio is greater than a preset ratio threshold, the driver under test is determined to be in a fatigued state; if the ratio is not greater than the preset ratio threshold, the driver under test is determined to be in a non-fatigued state.

[0128] If the driver under test is determined to be in a state of fatigue, an alarm message is generated and sent.

[0129] The opening and closing state of the human eye, or eye condition, can, to some extent, characterize a person's level of fatigue. This state can be identified by the distance between the upper and lower eyelids. Generally, when a person is fatigued, the distance between the upper and lower eyelids is relatively small, while when a person is not fatigued, the distance is relatively large. In this embodiment of the invention, the driving state of the driver under test can be determined as fatigued or not based on the opening and closing length.

[0130] Compared with previous blink studies based on image binary classification, this invention realizes the function of calculating the distance between the upper and lower eyelids. It can not only be used to calculate the blink frequency of the subject, but also to calculate the frequency of incomplete blinks and quantitatively measure the degree of eyelid closure during incomplete blinks. Compared with measuring eyelid closure based on traditional image algorithms, this invention has higher accuracy and wider applicability based on the deep learning eye segmentation model of the Transformer structure. Furthermore, horizontal correction of the obtained eye images can avoid calculation errors caused by shooting angle, shooting distance, etc., and can accurately locate the maximum distance between the upper and lower eyelids.

[0131] Example 2

[0132] Figure 3 This is a structural diagram of the eyelid distance calculation device based on eye images provided in Embodiment 2 of the present invention.

[0133] In some embodiments, the eyelid distance calculation device 20 based on eye images may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the eyelid distance calculation device 20 may be stored in the memory of an electronic device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality for calculating eyelid distance based on eye images.

[0134] In this embodiment, the eyelid distance calculation device 20 based on eye images can be divided into multiple functional modules according to its functions. These functional modules may include: a training module 201, a segmentation module 202, a correction module 203, and a calculation module 204. The term "module" in this invention refers to a series of computer program segments that can be executed by at least one processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module will be detailed in subsequent embodiments.

[0135] The training module 201 is used to acquire multiple human eye videos and train an eye segmentation model based on the multiple human eye videos.

[0136] Electronic devices can capture videos of multiple objects including their eye movements. For ease of description, videos including eye movements are referred to as human eye videos. One or more human eye videos of each object can be captured. Specifically, human eye videos can be captured using the same or different camera devices (e.g., head-mounted cameras) and then transmitted to the electronic device.

[0137] When this invention is applied to a medical setting, the human eye video can be obtained from a digital medical database. This digital medical database can be a digital repository storing patient medical records in a hospital, or it can be a networked database of multiple hospitals; this invention does not impose any limitations.

[0138] After acquiring multiple human eye videos, electronic devices can train on these videos to obtain an eye segmentation model, which can then be used to segment the input human eye image.

[0139] In an optional implementation, the eye segmentation model trained based on the plurality of human eye videos includes:

[0140] Obtain the original sampling frame rate for each of the aforementioned human eye videos;

[0141] The original sampling frame rate is compared with multiple preset sampling frame rate thresholds to obtain the comparison results;

[0142] Based on the comparison results, the original sampling frame rate is updated to the target sampling frame rate;

[0143] The target sampling frequency is used to sample the corresponding human eye video to obtain multiple human eye video frames;

[0144] The eye region in each of the aforementioned human eye video frames is labeled;

[0145] An eye segmentation model was obtained by training multiple video frames of human eyes labeled with eye regions.

[0146] The original sampling frame rate is the number of images captured by the camera device that captures human eye video per unit time.

[0147] The multiple preset sampling frame rate thresholds are pre-set critical values ​​used to determine whether the original sampling frame rate of the human eye video meets the requirements. These preset sampling frame rate thresholds may include a preset first sampling frame rate threshold and a preset second sampling frame rate threshold, wherein the preset first sampling frame rate threshold is less than the preset second sampling frame rate threshold. For example, the preset first sampling frame rate threshold may be set to 10 fps, and the preset second sampling frame rate threshold may be set to 20 fps.

[0148] The electronic device obtains a comparison result by acquiring the raw sampling frame rate of a human eye video and comparing it with multiple preset sampling frame rate thresholds. The comparison result may include: the raw sampling frame rate of the human eye video is less than the preset first sampling frame rate threshold; the raw sampling frame rate of the human eye video is greater than the preset first sampling frame rate threshold but less than the preset second sampling frame rate threshold; or the raw sampling frame rate of the human eye video is greater than the preset second sampling frame rate threshold.

[0149] Since a blink takes 0.2-0.4 seconds, if the original sampling frame rate of the human eye video is less than the preset first sampling frame rate threshold, then using an original sampling frame rate lower than the preset first sampling frame rate threshold to sample the human eye video may result in multiple human eye video frames being extracted from the human eye video missing the closed eye state, meaning the actual distance of the eyelids cannot be effectively detected. Therefore, it is necessary to update the original sampling frame rate (which is lower than the preset first sampling frame rate threshold) and use the updated sampling frame rate to sample the human eye video, thereby solving the problem of the extracted human eye video frames missing the closed eye state.

[0150] If the original sampling frame rate of the human eye video is greater than the preset second sampling frame rate threshold, although there is no problem of missing the closed-eye state in the extracted human eye video frames, using a larger original sampling frame rate to sample the human eye video results in a large number of human eye video frames. Calculating eyelid distance on a large number of human eye video frames increases the number of calculations and slows down the analysis speed of the human eye video. Therefore, it is necessary to update the original sampling frame rate that is greater than the preset second sampling frame rate threshold, and then use the updated sampling frame rate to sample the human eye video, thereby solving the problem of slow computation efficiency due to the large number of extracted human eye video frames.

[0151] When the original sampling frame rate of the human eye video is greater than the preset first sampling frame rate threshold but less than the preset second sampling frame rate threshold, it indicates that sampling the human eye video using the original sampling frame rate will result in a number of human eye video frames that are neither too many nor will it miss the closed-eye state. For the sake of consistency, when the original sampling frame rate of the human eye video is greater than the preset first sampling frame rate threshold but less than the preset second sampling frame rate threshold, the original sampling frame rate is taken as the target sampling frame rate.

[0152] In an optional implementation, updating the original sampling frame rate to the target sampling frame rate based on the comparison result includes:

[0153] When the original sampling frame rate is less than the preset first sampling frame rate threshold, calculate a first ratio of the preset first sampling frame rate threshold to the original sampling frame rate, and calculate a second ratio of the preset second sampling frame rate threshold to the original sampling frame rate;

[0154] A first update factor is generated based on the first ratio and the second ratio;

[0155] The original sampling frame rate is updated to the target sampling frame rate using the first update factor.

[0156] For example, if the original sampling frame rate is f01, the preset first sampling frame rate threshold is f11, and the preset second sampling frame rate threshold is f21, and f01 < f11 < f21, then the calculated first ratio is f11 / f01, and the calculated second ratio is f21 / f01. Obviously, f21 / f01 > f11 / f01 > 1. The electronic device can use a random function to generate a random number between the first ratio f11 / f01 and the second ratio f21 / f01. The generated random number is used as the first update factor. The product of the first update factor and the original sampling frame rate is calculated as the target sampling frame rate.

[0157] In the above optional implementation, when the original sampling frame rate is less than the preset first sampling frame rate threshold, a first update factor is generated based on the first ratio of the preset first sampling frame rate threshold to the original sampling frame rate and a second ratio of the preset second sampling frame rate threshold to the original sampling frame rate. The original sampling frame rate is then updated using the first update factor. Since the first update factor is greater than 1, updating the original sampling frame rate using the first update factor is equivalent to increasing the original sampling frame rate. Therefore, the updated target sampling frame rate is between the preset first sampling frame rate threshold and the preset second sampling frame rate threshold. Thus, when sampling human eye video using the target sampling frame rate, the number of multiple human eye video frames obtained is appropriate, and the closed-eye state is not missed.

[0158] In an optional implementation, updating the original sampling frame rate to the target sampling frame rate based on the comparison result further includes:

[0159] When the original sampling frame rate is greater than the preset second sampling frame rate threshold, calculate the third ratio of the preset first sampling frame rate threshold to the original sampling frame rate, and calculate the fourth ratio of the preset second sampling frame rate threshold to the original sampling frame rate.

[0160] A second update factor is generated based on the third ratio and the fourth ratio;

[0161] The original sampling frame rate is updated to the target sampling frame rate using the second update factor.

[0162] For example, if the original sampling frame rate is f02, the preset first sampling frame rate threshold is f12, and the preset second sampling frame rate threshold is f22, and f02 > f22 > f12, then the calculated first ratio is f12 / f02, and the calculated second ratio is f22 / f02. Obviously, f12 / f02 < f22 / f02 < 1. The electronic device can use a random function to generate a random number between the first ratio f12 / f02 and the second ratio f22 / f02. The generated random number is used as the second update factor. The product of the second update factor and the original sampling frame rate is calculated as the target frame rate.

[0163] In the above optional implementation, when the original sampling frame rate is greater than the preset first sampling frame rate threshold, a second update factor is generated based on the third ratio of the preset first sampling frame rate threshold to the original sampling frame rate and the fourth ratio of the preset second sampling frame rate threshold to the original sampling frame rate. The original sampling frame rate is then updated using the second update factor. Since the second update factor is less than 1, updating the original sampling frame rate using the second update factor is equivalent to reducing the original sampling frame rate. Therefore, the updated target sampling frame rate is between the preset first sampling frame rate threshold and the preset second sampling frame rate threshold. Thus, when sampling the human eye video using the target sampling frame rate, the number of multiple human eye video frames obtained is appropriate, and the closed-eye state is not missed.

[0164] In an optional implementation, the eye segmentation model obtained by training multiple human eye video frames labeled with eye regions includes:

[0165] An encoder is constructed using an overlapping block merging module with downsampling capabilities and an efficient self-attention transformer module capable of global modeling representation;

[0166] A decoder is constructed using a block expansion module with linear upsampling capability, the efficient self-attention transformer module, and a jump connection module capable of fusing high- and low-level features;

[0167] An eye segmentation model is constructed using the encoder and the decoder;

[0168] A loss function is constructed, and the eye segmentation model is iteratively trained using the multiple human eye video frames labeled with eye regions.

[0169] After obtaining multiple human eye video frames, electronic devices can use annotation tools to annotate the eye region between the upper and lower eyelid edges in each human eye video frame. The human eye video frames with annotated eye regions can then be used as training samples to train the eye segmentation model.

[0170] Electronic devices can use SegFormer-B0 (Simple and Efficient Design for Semantic Segmentation with Transformers) as the network architecture to train an eye segmentation model. SegFormer is an efficient and powerful semantic segmentation framework composed of Transformer and a lightweight multilayer perceptron (MLP). It employs the Transformer architecture to achieve multi-scale self-attention feature extraction, containing rich semantic information to ensure a high-accuracy eye segmentation model. The eye segmentation model training uses a similarity metric as the loss function, and the Adam optimizer iteratively updates the network weights with a learning rate of 0.0001.

[0171] In one optional implementation, before inputting the training samples into the eye segmentation model, the training data can be augmented by processes such as flipping, rotating, and brightness transformation. The augmented training data is then input into the eye segmentation model for training.

[0172] For a training sample with an input height of H and a width of W, an encoder composite function is constructed. Each feature map corresponds to a first sub-function. Each first sub-function of the encoder consists of one overlapping block merging module and two efficient Transformer modules, which are called sub-coding blocks. Each sub-coding block can halve the spatial dimension of the input feature map. Thus, the four sub-coding blocks form four consecutive stages of the encoder, which can obtain four feature maps of different scales to provide features of different abstraction levels.

[0173] Within each sub-encoding block, the overlapping block merging module performs matrix multiplication on the input training samples using an overlapping sliding window approach, thereby achieving linear embedding of the feature maps and downsampling the spatial resolution dimension of the input data by a factor of 2. The efficient Transformer module includes a self-attention sub-module and a feedforward neural network sub-module, used to learn global and fused features between different locations in the training samples.

[0174] For the multi-stage encoded features, a decoder function is constructed. The decoding process adopts a symmetrical approach to the encoding process. The decoder function includes four second sub-functions, which together constitute the four stages of the decoder. Each second sub-function is constructed using a block expansion module based on fully connected layer upsampling and two efficient Transformer modules, referred to as a sub-decoding block. The block expansion module trades channel dimension for spatial dimension. It doubles the channel dimension of the data through a fully connected layer before performing a dimension reshaping operation. Specifically, it splits the channels into four parts, and interleaves every two parts along the spatial dimension, thereby doubling the spatial dimension while halving the channel dimension.

[0175] Meanwhile, in the corresponding stages of the encoder and decoder, a "skip connection" structure is used to connect and splice the features of the encoder and decoder, so that the decoder can receive features from the encoder in the same stage for fusion, in order to fuse different semantic features at shallow coarse-grained and high-level fine-grained levels.

[0176] Pixel-level cross-entropy loss (CE) is chosen as the primary optimization objective during model training. To allow the model to examine region relevance, a similarity metric (Dice) loss is used as an auxiliary method. For example, the loss function is: Loss = 0.7 * CE + 0.3 * Dice.

[0177] By calculating the gradient of the loss function, the network parameters of the eye segmentation model are updated along the negative gradient direction to achieve a continuous decrease in the loss function.

[0178] The above implementation method can combine local attention information and global attention information to render features, making feature extraction more efficient and thus improving the accuracy of the eye segmentation model.

[0179] The segmentation module 202 is used to segment the eye region in the image of the human eye under test using the eye segmentation model in response to an instruction to obtain the distance of the eyelid in the image of the human eye under test, so as to obtain a binarized image of the human eye under test.

[0180] The image of the subject's eye refers to the image of the eye from which the distance between the upper and lower eyelids needs to be calculated. This image can be either the left or right eye of the subject.

[0181] When an electronic device receives an image of the subject's eye, it triggers a command to acquire the distance between the eyelids in the image. In response to this command, the image is input into an eye segmentation model. The model identifies the eye region within the image, obtaining the area between the upper and lower eyelids. This area is then cropped from the original image to obtain the final eye image. The area between the upper and lower eyelids is the eyeball region or the open eye region.

[0182] It should be noted that the image of the eye being tested is a binarized image.

[0183] The correction module 203 is used to perform horizontal correction on the eye image to be tested to obtain a corrected eye image.

[0184] Because the subject may be tilted or have droopy eyelids due to old age when the camera device acquires the image of the subject's eye, the eyelids will be tilted to varying degrees in the binarized image of the subject's eye output by the eye segmentation model. Therefore, the electronic device needs to perform horizontal correction on the image of the subject's eye so that the line connecting the two corners of the eye is parallel to the coordinate axis of the preset image coordinate system.

[0185] In an optional implementation, the step of horizontally correcting the image of the eye to be tested to obtain a corrected image of the eye includes:

[0186] Obtain multiple connected regions in the eye image to be tested;

[0187] Calculate the area of ​​each of the connected regions;

[0188] The target connectivity area is the largest available connectivity area.

[0189] Obtain the minimum and maximum x-coordinates of the pixels in the target connected region;

[0190] Calculate the angle between the line connecting the minimum abscissa point and the maximum abscissa point and the horizontal line;

[0191] The corrected eye image is obtained by rotating the image of the eye to be tested counterclockwise according to the included angle.

[0192] The eye image under test contains multiple connected regions. The electronic device can calculate the area of ​​the connected region by counting the number of pixels in the connected region, and determine the connected region with the largest area as the target connected region.

[0193] After obtaining the target connected region, it is retained while the remaining connected regions are deleted, thus achieving horizontal correction of the eye image under test based on the target connected region. The minimum abscissa of the pixels in the target connected region is the left corner of the eye image under test, and the maximum abscissa of the pixels in the target connected region is the right corner of the eye image under test. The angle θ between the line connecting the left and right corners of the eye and the horizontal line is calculated. The eye image under test is rotated counterclockwise by θ to complete the horizontal correction.

[0194] The calculation module 204 is used to obtain the distance between the eyelids in the image of the subject eye based on the corrected eye image.

[0195] In an optional implementation, obtaining the distance between the eyelids in the test eye image based on the corrected eye image includes:

[0196] Obtain the minimum and maximum ordinate points of the pixels in the target connected region;

[0197] The distance between the eyelids in the image of the human eye under test is calculated based on the minimum and maximum ordinate points.

[0198] The electronic device acquires the minimum and maximum ordinates of pixels in the target connected region of the corrected eye image. The minimum ordinate represents the highest point of the upper eyelid, and the maximum ordinate represents the lowest point of the lower eyelid. The vertical distance between the horizontal line containing the highest point of the upper eyelid and the horizontal line containing the lowest point of the lower eyelid is calculated to obtain the eyelid distance in the image of the subject's eye. Here, the eyelid distance represents the opening degree between the upper and lower eyelids.

[0199] Due to limitations in image recognition accuracy, the identified upper and lower eyelids may not be smooth, for example, with the appearance of an incorrect point, resulting in abnormal curvature of the upper and lower eyelids, which is clearly unrealistic. In an alternative implementation, the electronic device can smooth the upper and lower eyelids in the image of the subject's eye to obtain images that more closely resemble reality, thereby improving the accuracy of subsequent measurements.

[0200] The device described in this invention can be applied to the field of digital medical technology, promoting the development of smart cities. For example, by plotting a waveform image of the distance between the eyelids (i.e., the opening and closing degree between the upper and lower eyelids) over time, the waveform change characteristics can be used to indirectly assess ocular surface diseases, such as dry eye syndrome and keratoconjunctivitis, thereby aiding in the diagnosis and treatment of ocular surface diseases. The following is a related description... Figure 2 The process is described as follows: Acquire eye videos of the patient over a period of time. Frames are extracted according to ①, resulting in multiple single-frame images. Each single-frame image is then input into a trained eye segmentation model according to ②. The model segments the eye region within the input single-frame images to obtain a prediction image. This prediction image is a binary eye image containing two connected regions. Then, according to ③, the image corresponding to the largest connected region is selected for horizontal correction to obtain the corrected eye image. Finally, according to ④, the minimum ordinate of the pixels in the target connected region is used as the highest point of the upper eyelid, and the maximum ordinate is used as the lowest point of the lower eyelid. The vertical distance between the horizontal line containing the highest point of the upper eyelid and the horizontal line containing the lowest point of the lower eyelid is calculated to obtain the distance of the patient's eyelids. The eyelid distances corresponding to each single-frame image are then plotted as waveform images of the eyelids in chronological order.

[0201] The device described in this invention, in addition to being used to calculate the distance between the upper and lower eyelids to assess eye risks, can also be applied to fatigue detection in the field of autonomous driving.

[0202] For example, an electronic device acquires video of the driver's eyes within a preset time period. Based on any frame rate between a preset first sampling frame rate threshold and a preset second sampling frame rate threshold, it samples the video of the driver's eyes within the preset time period, obtaining multiple video frames of the driver's eyes to be tested. The method described in this invention is used to obtain the distance between the upper and lower eyelids in each video frame of the driver's eyes to be tested. Further, the electronic device acquires target distances among the distances between the upper and lower eyelids in the video frames of the driver's eyes to be tested that are less than a preset distance threshold. A first number of target distances is calculated, and a second number of these target distances is used to determine the driving state of the driver under test based on this ratio.

[0203] Specifically, if the ratio is greater than a preset ratio threshold, the driver under test is determined to be in a fatigued state; if the ratio is not greater than the preset ratio threshold, the driver under test is determined to be in a non-fatigued state.

[0204] If the driver under test is determined to be in a state of fatigue, an alarm message is generated and sent.

[0205] The opening and closing state of the human eye, or eye condition, can, to some extent, characterize a person's level of fatigue. This state can be identified by the distance between the upper and lower eyelids. Generally, when a person is fatigued, the distance between the upper and lower eyelids is relatively small, while when a person is not fatigued, the distance is relatively large. In this embodiment of the invention, the driving state of the driver under test can be determined as fatigued or not based on the opening and closing length.

[0206] Compared with previous blink studies based on image binary classification, this invention realizes the function of calculating the distance between the upper and lower eyelids. It can not only be used to calculate the blink frequency of the subject, but also to calculate the frequency of incomplete blinks and quantitatively measure the degree of eyelid closure during incomplete blinks. Compared with measuring eyelid closure based on traditional image algorithms, this invention has higher accuracy and wider applicability based on the deep learning eye segmentation model of the Transformer structure. Furthermore, horizontal correction of the obtained eye images can avoid calculation errors caused by shooting angle, shooting distance, etc., and can accurately locate the maximum distance between the upper and lower eyelids.

[0207] Example 3

[0208] This embodiment provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps in the above-described embodiment of the eyelid distance calculation method based on eye images. For example... Figure 1 S11-S14 as shown:

[0209] S11, acquire multiple human eye videos, and train an eye segmentation model based on the multiple human eye videos;

[0210] S12, in response to the instruction to obtain the distance of the eyelid in the image of the human eye to be tested, the eye region in the image of the human eye to be tested is segmented using the eye segmentation model to obtain a binarized image of the human eye to be tested;

[0211] S13, perform horizontal correction on the eye image to be tested to obtain a corrected eye image;

[0212] S14, Based on the corrected eye image, obtain the distance between the eyelids in the image of the subject's eye.

[0213] Alternatively, when the computer program is executed by the processor, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 3 Modules 201-204 in the middle:

[0214] The training module 201 is used to acquire multiple human eye videos and train an eye segmentation model based on the multiple human eye videos.

[0215] The segmentation module 202 is used to segment the eye region in the eye image of the subject eye using the eye segmentation model when responding to the instruction to obtain the distance of the eyelid in the eye image of the subject eye, so as to obtain a binarized eye image of the subject eye.

[0216] The correction module 203 is used to perform horizontal correction on the eye image to be tested to obtain a corrected eye image;

[0217] The calculation module 204 is used to obtain the distance between the eyelids in the image of the subject's eye based on the corrected eye image.

[0218] Example 4

[0219] See Figure 4 The diagram shown is a structural schematic of an electronic device provided in Embodiment 3 of the present invention. In a preferred embodiment of the present invention, the electronic device 3 includes a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.

[0220] Those skilled in the art should understand that Figure 4 The structure of the electronic device shown does not constitute a limitation of the embodiments of the present invention. It can be a bus structure or a star structure. The electronic device 3 may also include more or fewer other hardware or software than shown, or different component arrangements.

[0221] In some embodiments, the electronic device 3 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital processors, and embedded devices. The electronic device 3 may also include client devices, including, but not limited to, any electronic product capable of human-computer interaction with a client via a keyboard, mouse, remote control, touchpad, or voice control device, such as personal computers, tablets, smartphones, and digital cameras.

[0222] The electronic device 3 described herein is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0223] In some embodiments, the memory 31 stores a computer program that, when executed by the at least one processor 32, implements all or part of the steps in the eyelid distance calculation method based on eye images as described above. The memory 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0224] Furthermore, the computer-readable storage medium may primarily include a program storage area and a data storage area, wherein the program storage area may store the operating system, at least one application required for a function, etc.; and the data storage area may store data created based on the use of blockchain nodes, etc.

[0225] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0226] In some embodiments, the at least one processor 32 is the control unit of the electronic device 3, connecting various components of the electronic device 3 via various interfaces and lines. It executes programs or modules stored in the memory 31 and calls data stored in the memory 31 to perform various functions and process data. For example, when the at least one processor 32 executes a computer program stored in the memory, it implements all or part of the steps of the eyelid distance calculation method based on eye images described in this embodiment of the invention; or it implements all or part of the functions of the eyelid distance calculation device based on eye images. The at least one processor 32 may be composed of integrated circuits, such as a single-packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0227] In some embodiments, the at least one communication bus 33 is configured to enable communication between the memory 31 and the at least one processor 32, etc.

[0228] Although not shown, the electronic device 3 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 32 via a power management device, thereby enabling functions such as charging, discharging, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 3 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0229] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) or processor to execute portions of the methods described in the various embodiments of the present invention.

[0230] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0231] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0232] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0233] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements, and the singular does not exclude the plural. Multiple elements or devices recited in the specification may also be implemented by a single element or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0234] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for calculating eyelid distance based on eye images, characterized in that, The method includes: Acquiring multiple human eye videos and training an eye segmentation model based on the multiple human eye videos includes: acquiring the original sampling frame rate of each human eye video; comparing the original sampling frame rate with multiple preset sampling frame rate thresholds, the multiple preset sampling frame rate thresholds including a preset first sampling frame rate threshold and a preset second sampling frame rate threshold; updating the original sampling frame rate that is less than the preset first sampling frame rate threshold or greater than the preset second sampling frame rate threshold to a target sampling frame rate; sampling the human eye videos using the target sampling frame rate to obtain multiple human eye video frames; annotating the eye regions in each human eye video frame; and training the eye segmentation model based on the multiple human eye video frames annotated with eye regions. In response to a command to obtain the distance of the eyelid in the image of the human eye to be tested, the eye segmentation model is used to segment the eye region in the image of the human eye to be tested to obtain a binarized image of the human eye to be tested. The image of the eye to be tested is horizontally corrected to obtain a corrected eye image; The distance between the eyelids in the image of the subject's eye is obtained based on the corrected eye image.

2. The eyelid distance calculation method based on eye images as described in claim 1, characterized in that, The step of horizontally correcting the image of the eye to be tested to obtain a corrected image of the eye includes: Obtain multiple connected regions in the eye image to be tested; Calculate the area of ​​each of the connected regions; The target connectivity area is the largest available connectivity area. Obtain the minimum and maximum x-coordinates of the pixels in the target connected region; Calculate the angle between the line connecting the minimum abscissa point and the maximum abscissa point and the horizontal line; The corrected eye image is obtained by rotating the image of the eye to be tested counterclockwise according to the included angle.

3. The eyelid distance calculation method based on eye images as described in claim 2, characterized in that, The step of obtaining the distance between the eyelids in the test eye image based on the corrected eye image includes: Obtain the minimum and maximum ordinate points of the pixels in the target connected region; The distance between the eyelids in the image of the human eye under test is calculated based on the minimum and maximum ordinate points.

4. The eyelid distance calculation method based on eye images as described in claim 1, characterized in that, The preset first sampling frame rate threshold is less than the preset second sampling frame rate threshold; The step of updating the original sampling frame rate, which is less than the preset first sampling frame rate threshold or greater than the preset second sampling frame rate threshold, to the target sampling frame rate includes: When the original sampling frame rate is less than the preset first sampling frame rate threshold, calculate a first ratio of the preset first sampling frame rate threshold to the original sampling frame rate, and calculate a second ratio of the preset second sampling frame rate threshold to the original sampling frame rate; A first update factor is generated based on the first ratio and the second ratio; The original sampling frame rate is updated to the target sampling frame rate using the first update factor.

5. The eyelid distance calculation method based on eye images as described in claim 4, characterized in that, The step of updating the original sampling frame rate, which is less than the preset first sampling frame rate threshold or greater than the preset second sampling frame rate threshold, to the target sampling frame rate further includes: When the original sampling frame rate is greater than the preset second sampling frame rate threshold, calculate the third ratio of the preset first sampling frame rate threshold to the original sampling frame rate, and calculate the fourth ratio of the preset second sampling frame rate threshold to the original sampling frame rate. A second update factor is generated based on the third ratio and the fourth ratio; The original sampling frame rate is updated to the target sampling frame rate using the second update factor.

6. The eyelid distance calculation method based on eye images as described in claim 5, characterized in that, The eye segmentation model, trained based on multiple human eye video frames labeled with eye regions, includes: An encoder is constructed using an overlapping block merging module with downsampling capabilities and an efficient self-attention transformer module capable of global modeling representation; A decoder is constructed using a block expansion module with linear upsampling capability, the efficient self-attention transformer module, and a jump connection module capable of fusing high- and low-level features; An eye segmentation model is constructed using the encoder and the decoder; A loss function is constructed, and the eye segmentation model is iteratively trained using the multiple human eye video frames labeled with eye regions.

7. An eyelid distance calculation device based on eye images, characterized in that, The apparatus is used to implement the eyelid distance calculation method based on eye images as described in any one of claims 1 to 6, and the apparatus comprises: The training module is used to acquire multiple human eye videos and train an eye segmentation model based on the multiple human eye videos; The segmentation module is used to segment the eye region in the image of the human eye under test using the eye segmentation model when responding to the instruction to obtain the distance of the eyelid in the image of the human eye under test, so as to obtain a binarized image of the human eye under test. The correction module is used to perform horizontal correction on the eye image to be tested, so as to obtain a corrected eye image; The calculation module is used to obtain the distance between the eyelids in the image of the subject's eye based on the corrected eye image.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the eyelid distance calculation method based on eye images as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the eyelid distance calculation method based on eye images as described in any one of claims 1 to 6.

Images