Eye fatigue detection method, device and equipment and computer readable storage medium

Abstract

The invention provides an eye fatigue detection method, device and equipment and a computer readable storage medium, and the method comprises the steps: obtaining an eye reference image of a user who currently wears head-mounted equipment, and obtaining continuous frames of eye images in a preset time period after the eye reference image is obtained; according to the eye reference image and the continuous frames of eye images, through an eye fatigue detection model, respectively determining the eye closing degree of the user in the normal eye state and the eye closing degree of the user in the continuous frames of eye states; and determining whether the eyes of the user are in a fatigue state or not according to the eye closing degree of the user in the normal eye state and the eye closing degree of the user in the continuous frame eye state. According to the method provided by the invention, the problem that the personalized demand of the user cannot be met due to the fact that personalized and accurate fatigue detection is provided for the user in the prior art can be solved.

Description

Technical Field

[0001] This application relates to the field of virtual reality, and in particular, to an eye fatigue detection method, device, equipment, and computer-readable storage medium. Background Art

[0002] Virtual Reality (VR) is a combination of multiple technologies, including real-time three-dimensional computer graphics technology, wide-angle (wide field of view) stereoscopic display technology, tracking technology for the observer's head, eyes, and hands, as well as haptic / force feedback, stereo sound, network transmission, voice input / output technology, etc. With the development of virtual reality, immersive experiences are becoming increasingly popular among a wide range of users.

[0003] However, due to the long time that users wear the head-mounted device during immersive experiences, eye fatigue may occur. Currently, in order to relieve users' eye fatigue, measures such as reminding users to remove the device after wearing it for more than a certain time are usually taken. However, at the same time, it brings many inconveniences to users.

[0004] Therefore, the existing technology cannot provide users with personalized and accurate fatigue detection, and thus cannot meet the personalized needs of users. Summary of the Invention

[0005] This application provides an eye fatigue detection method, device, equipment, and computer-readable storage medium to solve the problem that the existing technology cannot provide users with personalized and accurate fatigue detection, and thus cannot meet the personalized needs of users.

[0006] In a first aspect, an embodiment of this application provides an eye fatigue detection method, and the method includes:

[0007] Obtain a reference eye image of the user currently wearing the head-mounted device, and after obtaining the reference eye image, obtain consecutive frame eye images within a preset time period; the reference eye image is used to represent the eye image of the user in a normal eye state.

[0008] According to the reference eye image and the consecutive frame eye images, respectively determine the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive frame eye states through an eye fatigue detection model; wherein, the eye fatigue detection model is obtained by training a deep learning network model.

[0009] Determine whether the user's eyes are in a fatigued state according to the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive frame eye states.

[0010] In a possible design, determining whether the user's eyes are in a fatigued state based on the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the eye states of consecutive frames includes:

[0011] Based on the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the eye states of consecutive frames, determine whether the user's eyes are in a fatigued state through a blinking judgment strategy and / or an eye closure degree judgment strategy.

[0012] In a possible design, the degree of eye closure is used to represent the eye aspect ratio; based on the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the eye states of consecutive frames, determine whether the user's eyes are in a fatigued state through a blinking judgment strategy and / or an eye closure degree judgment strategy, including:

[0013] Based on the degree of eye closure of the user in the eye states of consecutive frames, determine the number of blinks within the preset time period, and based on the degree of eye closure of the user in the normal eye state, determine the proportion of the first time when the eye state returns to the normal eye state of the user during the blinking process; if the number of blinks is greater than the first preset number threshold, and / or, if the first time proportion is greater than the first preset time proportion threshold, then determine that the user's eyes are in a fatigued state; and / or,

[0014] Multiply the degree of eye closure of the user in the normal eye state by the corresponding weight to obtain an eye aspect ratio threshold; based on the degree of eye closure of the user in the eye states of consecutive frames, determine the proportion of the second time within the preset time period when the degree of eye closure of the user in the eye states of consecutive frames is less than the eye aspect ratio threshold; if the second time proportion is greater than the first preset time proportion threshold, then determine that the user's eyes are in a fatigued state.

[0015] In a possible design, based on the degree of eye closure of the user in the eye states of consecutive frames, determining the number of blinks within the preset time period, and based on the degree of eye closure of the user in the normal eye state, determining the proportion of the first time when the eye state returns to the normal eye state of the user during the blinking process, includes:

[0016] Take the degree of eye closure of the user in the normal eye state and the degree of eye closure of 0 as the reference lines respectively, and draw the eye aspect ratio curve corresponding to the user based on the degree of eye closure of the user in the eye states of consecutive frames and the degree of eye closure of the user in the normal eye state; the horizontal axis of the eye aspect ratio curve is time, and the vertical axis is the eye aspect ratio.

[0017] Determine the number of blinks, the first time ratio, and the second time ratio respectively according to the eye aspect ratio curve.

[0018] In a possible design, the method further includes:

[0019] If it is determined that the user's eyes are in a fatigued state, determine the fatigue level according to at least one of the number of blinks, the first time ratio, and the second time ratio;

[0020] Adjust the perceived distance between the currently displayed virtual image in the head-mounted device and the eyes according to the correlation between the fatigue level and the perceived distance between the virtual image displayed in the head-mounted device and the eyes;

[0021] Wherein, the correlation is determined by the minimum distance and the maximum distance supported by the head-mounted device and the degree of eye closure of the user in a normal eye state.

[0022] In a possible design, the head-mounted device is used to support the user to customize and turn on the fatigue detection function; before obtaining the eye reference image of the user currently wearing the head-mounted device, the method further includes:

[0023] Respond to the instruction operation of the user currently wearing the head-mounted device to turn on the fatigue detection function;

[0024] Prompt the user to look straight ahead to collect the eye reference image of the user.

[0025] In a possible design, the determining the degree of eye closure of the user in a normal eye state and the degree of eye closure of the user in the continuous frame eye state respectively through the eye fatigue detection model according to the eye reference image and the eye images of the continuous frames includes:

[0026] Input the eye reference image and the eye images of the continuous frames into the eye fatigue detection model respectively to obtain the degree of eye closure of the user in a normal eye state and the degree of eye closure of the user in the continuous frame eye state; or,

[0027] Input the eye reference image and the eye images of the consecutive frames into the eye fatigue detection model respectively, to obtain the positions of the feature points of the user in the normal eye state and the positions of the feature points of the user in the eye state of each frame in the consecutive frames; according to the positions of the feature points of the user in the normal eye state and the positions of the feature points of the user in the eye state of each frame in the consecutive frames, through the eye closure degree model, obtain the eye closure degree of the user in the normal eye state and the eye closure degree of the user in the eye state of the consecutive frames respectively; wherein, the eye closure degree model is determined by the positions of the feature points.

[0028] In a second aspect, an embodiment of the present application provides an eye fatigue detection device, which is applied to a head-mounted device or an electronic device; the device includes:

[0029] An image acquisition module, configured to acquire an eye reference image of a user currently wearing the head-mounted device, and after acquiring the eye reference image, acquire eye images of consecutive frames within a preset time period; the eye reference image is used to represent the eye image of the user in the normal eye state.

[0030] An image processing module, configured to respectively determine the eye closure degree of the user in the normal eye state and the eye closure degree of the user in the eye state of the consecutive frames through an eye fatigue detection model according to the eye reference image and the eye images of the consecutive frames; wherein, the eye fatigue detection model is obtained by training a deep learning network model.

[0031] A fatigue detection module, configured to determine whether the eyes of the user are in a fatigued state according to the eye closure degree of the user in the normal eye state and the eye closure degree of the user in the eye state of the consecutive frames.

[0032] In a third aspect, an embodiment of the present application provides a head-mounted device, including: a data acquisition device and the eye fatigue detection device; wherein, the eye fatigue detection device is configured to execute the method according to any one of the first aspect.

[0033] The data acquisition device includes a camera; the camera is configured to acquire images of the eyes at any moment or in any eye state.

[0034] In a fourth aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory;

[0035] The memory stores computer execution instructions;

[0036] The processor executes the computer execution instructions stored in the memory, so that the processor executes the method according to any one of the first aspect.

[0037] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method according to any one of the first aspect.

[0038] In a sixth aspect, an embodiment of the present application provides a computer program product including a computer program, which, when executed by a processor, implements the method according to any one of the first aspect.

[0039] The eye fatigue detection method, device, equipment and computer-readable storage medium provided in this embodiment first obtain a reference eye image of a user currently wearing the head-mounted device, and after obtaining the reference eye image, obtain consecutive frame eye images within a preset time period; the reference eye image is used to represent the eye image of the user in a normal eye state; further, according to the reference eye image and the consecutive frame eye images, an eye fatigue detection model is used to respectively determine the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive frame eye state; wherein, the eye fatigue detection model is obtained by training a deep learning network model; further, according to the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive frame eye state, it is determined whether the user's eyes are in a fatigued state. Therefore, the present application uses the user's own eye image in the normal eye state as a reference image, performs logical processing with the eye images continuously collected within a subsequent preset time period, and respectively determines the degree of eye closure of the user in different eye states through the eye fatigue detection model, realizing the judgment of eye fatigue based on the user's personalized degree of eye closure, rather than calculating the corresponding degree of eye closure of different users according to a unified standard or benchmark to determine whether there is eye fatigue, thereby improving the accuracy of fatigue detection and meeting personalized needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0041] Figure 1 It is a schematic diagram of the scenario of the eye fatigue detection method provided in the embodiment of the present application;

[0042] Figure 2 It is a schematic diagram of the VR glasses provided in the embodiment of the present application;

[0043] Figure 3 Schematic flowchart of the eye fatigue detection method provided by an embodiment of the present application;

[0044] Figure 4 Schematic diagram of the scenario of the eye fatigue detection method provided by another embodiment of the present application;

[0045] Figure 5 Schematic diagram of the scenario of the eye fatigue detection method provided by yet another embodiment of the present application;

[0046] Figure 6 Schematic flowchart of the eye fatigue detection method provided by another embodiment of the present application;

[0047] Figure 7 Schematic structural diagram of the eye fatigue detection device provided by an embodiment of the present application;

[0048] Figure 8 Schematic structural diagram of the electronic device provided by an embodiment of the present application. Detailed implementation manners

[0049] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0050] The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of the present application are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that such data may be interchanged under appropriate circumstances so that the embodiments of the present application described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these process, method, product or device.

[0051] It should be noted that the user information involved in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0052] Currently, since users may experience eye fatigue when wearing head-mounted devices for too long during immersive experiences. At present, in order to relieve users' eye fatigue, measures such as reminding users to take off the device after wearing it for more than a certain period are usually taken. However, at the same time, it brings a lot of inconvenience to users. Therefore, the prior art cannot provide personalized and accurate fatigue detection for users, and thus cannot meet the personalized needs of users.

[0053] Therefore, in view of the above problems, the technical concept of this application is to use the eye images of the user in the normal eye state as the reference images, and make logical judgments with the eye images continuously collected within a subsequent preset time period, realizing the judgment of eye fatigue based on the personalized eye closure degree of the user, rather than calculating the corresponding eye closure degrees of different users according to a unified standard to judge whether there is eye fatigue, thereby improving the accuracy of fatigue detection and meeting the personalized needs.

[0054] In practical applications, this eye fatigue detection method can be applied to fields such as VR, and no specific limitation is made here. Exemplarily, see Figure 1 as shown. Figure 1 is a schematic diagram of the scenario of the eye fatigue detection method provided by an embodiment of this application. This scenario includes an eye fatigue detection device, and this eye fatigue detection device can be a head-mounted device (combined with Figure 2 as shown. Figure 2 shows a schematic diagram of a VR glasses), including a data acquisition device and an eye fatigue detection device; among them, a camera in the data acquisition device can be used to acquire the reference eye image and the eye image. For example, an infrared camera, an RGB camera, a black-and-white camera, etc. Taking the acquisition of images by an infrared camera as an example, this data acquisition device can include an infrared camera 1011, a transparent frame 1012, etc. The infrared camera is installed on the transparent frame and can be passed through the left and right infrared cameras inside the VR glasses (see Figure 2The shown infrared camera 1011) acquires infrared data (here referring to an eye infrared image), for example: collects an infrared image of the eye at any moment or in any eye state. The eye fatigue detection device may include a calculation unit, and the calculation unit may include a first calculation unit and / or a second calculation unit. The first calculation unit may be a CPU calculation unit, and the second calculation unit may be a GPU calculation unit; wherein, after the calculation unit acquires a reference eye image through a data acquisition device, it acquires consecutive-frame eye infrared images within a preset time period through the data acquisition device, and then respectively determines the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive-frame eye states based on the reference eye image and the consecutive-frame eye infrared images through an eye fatigue detection model, and then determines whether the user's eyes are in a fatigued state based on the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive-frame eye states.

[0055] It should be noted that the start time of acquiring consecutive-frame eye infrared images within a preset time period may be the next moment after acquiring the reference eye infrared image, or may satisfy a time threshold, and the specific definition of this start time is not provided here.

[0056] Specifically, the process of eye fatigue detection in this scenario includes user system settings, infrared camera collection of eyeball (here referring to the eye) data, data preprocessing, model training, model inference application, and VR response prediction results. Among them, data preprocessing, model training, model inference application, and VR response prediction results may be performed by the first calculation unit, or may be performed by the second calculation unit, or may also be performed by a combination of the first calculation unit and the second calculation unit, and no specific definition is provided here.

[0057] Exemplarily, taking the operation of data preprocessing performed by the first computing unit, and the operations of model training, model inference application, and VR response prediction result performed by the first computing unit / second computing unit as examples, the specific process is as follows: First, the user can configure the user system. For example, log in to the user's account, set when to trigger fatigue detection, or whether to enable the fatigue detection function, etc. If the fatigue detection function is started, during the execution of the fatigue detection function: obtain the eye infrared image or eye RGB image of the user in the normal eye state through the infrared camera or visible light camera in the data acquisition device as the user's eye reference image (the eye reference image here can be the eye infrared image), and continuously collect the eye infrared images within a preset time period through the infrared camera. Then, the first computing unit can preprocess the collected images. Then, the CPU computing unit / GPU computing unit trains a deep learning network model (i.e., a deep learning model) based on the preprocessed data to obtain an eye fatigue detection model. Through the trained eye fatigue detection model, the eye reference image of the user currently wearing the head-mounted device collected by the data acquisition device and the eye infrared images of consecutive frames within a preset time period are respectively input into the eye fatigue detection model to determine the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive frame eye states, and use the CPU computing unit / GPU computing unit to perform VR response to obtain a prediction result, that is, determine whether the user's eyes are in a fatigued state based on the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive frame eye states.

[0058] Therefore, by using the user's own eye infrared image in the normal eye state as the reference image and making a logical judgment with the eye infrared images collected in the subsequent continuous preset time period, the judgment of eye fatigue based on the user's personalized eye closure degree is realized, rather than calculating the corresponding eye closure degrees of different users according to a unified standard to judge whether there is eye fatigue, thereby improving the accuracy of fatigue detection and meeting the personalized needs.

[0059] The technical solution of the present application will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0060] Figure 3 The flowchart of the eye fatigue detection method provided by the embodiment of the present application. This eye fatigue detection method may include:

[0061] S301. Obtain the eye reference image of the user currently wearing the head-mounted device, and after obtaining the eye reference image, obtain the eye images of consecutive frames within a preset time period.

[0062] Among them, the eye reference image is used to represent the eye image of the user in a normal eye state.

[0063] In this embodiment, the execution subject may be an eye fatigue detection device, which is installed in an eye fatigue detection device. The eye fatigue detection device may be an electronic device or a head-mounted device. Optionally, the eye fatigue detection device may also be configured in a server, and the server may be communicatively connected to a data acquisition device. Among them, the data acquisition device may include a camera, and the camera is used to acquire an image of the eye at any moment or in any eye state.

[0064] Exemplarily, taking the head-mounted device as an example, the head-mounted device may acquire the eye reference image of the device through a camera (including an infrared camera, an RGB camera, a black and white camera, etc.). Here, the eye reference image may be an infrared eye image of the user in a normal eye state or an RGB eye image of the user in a normal eye state, etc. If the eye reference image is an RGB eye image of the user in a normal eye state, the RGB eye image of the user in a normal eye state may be converted into an infrared eye image of the user in a normal eye state. The conversion method is not specifically limited herein.

[0065] Taking the infrared camera as an example below, in order to determine whether the user's eyes are in a fatigued state, infrared eye images of consecutive frames are acquired through the infrared camera within a preset time period, and then the infrared eye images of the consecutive frames are subjected to logical judgment processing with the eye reference image of the user, so as to comprehensively analyze whether the user's eyes are in a fatigued state.

[0066] S302. According to the eye reference image and the eye images of the consecutive frames, respectively determine the degree of eye closure of the user in a normal eye state and the degree of eye closure of the user in the eye state of the consecutive frames through an eye fatigue detection model.

[0067] Among them, the eye fatigue detection model is obtained by training a deep learning network model.

[0068] In this embodiment, for the deep learning network model to determine whether it is in an eye fatigued state, for example, input: left and right eye infrared images 240*240*1; output: 12 coordinate values (i.e., the coordinate values of 6 feature points); error: MSE root mean square error; network structure: use a 3*3 convolutional layer with padding = 1 and stride = 2.

[0069] In a possible design, the step of respectively determining the degree of eye closure of the user in a normal eye state and the degree of eye closure of the user in the eye state of the consecutive frames according to the eye reference image and the infrared eye images of the consecutive frames through an eye fatigue detection model includes:

[0070] Input the eye reference image and the eye images of the consecutive frames into the eye fatigue detection model respectively to obtain the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the eye states of the consecutive frames; or,

[0071] Input the eye reference image and the eye images of the consecutive frames into the eye fatigue detection model respectively to obtain the positions of the feature points of the user in the normal eye state and the positions of the feature points of the user in the eye states of each frame in the consecutive frames; according to the positions of the feature points of the user in the normal eye state and the positions of the feature points of the user in the eye states of each frame in the consecutive frames, respectively obtain the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the eye states of the consecutive frames through an eye closure degree model; wherein, the eye closure degree model is determined by the positions of the feature points.

[0072] In this embodiment, the aspect ratio can be directly output by the deep learning model, or the feature points can be output and the aspect ratio can be calculated further. The following takes the deep learning model outputting feature points as an example to illustrate the architecture of the deep learning model:

[0073] Specifically, the deep learning model adopts an architecture of a convolutional layer, a Relu (activation function) layer, a pooling layer, and a fully connected layer.

[0074] 1) It includes 8 convolutional layers with a size of 3*3 and only 1 channel. Each convolutional layer does not change the length and width of the feature map of the previous layer; the number of channels is increased through convolution. In this case, the size of the predefined input infrared image is (batch_size, 1, 240, 240), and then after 2 convolutional layers with a size of 3*3 and 1 channel, the image size becomes (batch_size, 2, 240, 240).

[0075] 2) It includes 3 activation layers and 3 pooling layers, which are distributed after the 1st, 2nd, and 3rd convolutions respectively. Relu is used as the activation layer, which can increase the non-linear expression ability. The role of the pooling layer is to reduce the size of the feature map and improve the anti-interference ability of the network. In this case, the size of the feature map obtained after the 1st convolution is (batch_size, 2, 240, 240). The kernel size of MaxPooling is selected as 2x2, and it moves two steps each time, and the size of the output feature map obtained is (batch_size, 2, 120, 120).

[0076] 3) It includes a fully connected layer. The input and output of the fully connection are (batch_size, features_number). In this embodiment, 12 feature_numbers are required (16 are originally output), and the dropout strategy can be adopted to discard the redundant 4 ones, or only the first 12 values can be selected as the output.

[0077] S303. Determine whether the user's eyes are in a fatigued state according to the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the eye states of consecutive frames.

[0078] In this embodiment, the process of training the eye fatigue detection model is as follows: 1. Data annotation is used as the training ground truth (6 feature points can be selected for each eye, see Figure 4 as shown); 2. Collect the states of the eyes in the normal state (here it refers to collecting the reference images of the user's eyes, such as the reference infrared images of the eyes) and the states of the eyes in other different states as training samples. The deep learning network outputs the positions (x, y) of 6 feature points (such as Figure 4 the p1, p2, p3, p4, p5, p6 shown), and calculates the aspect ratio of the eyes in the normal state to represent the degree of eye closure. By continuously iteratively training the deep learning network model, an eye fatigue detection model is obtained; among them, the formula for calculating the aspect ratio of the eyes is:

[0079] A = (|p2.y - P6.y| + |p3.y - p5.y|) / |p1.x - p4.x|

[0080] Among them, p2.y is used to represent the y value of p2, p3.y is used to represent the y value of p3, p5.y is used to represent the y value of p5, P6.y is used to represent the y value of p6, p1.x is used to represent the x value of p1, and p4.x is used to represent the x value of p4.

[0081] 3. The model application process is as follows: Collect the positions of 6 feature points output by the network for a period of time, and each feature point includes (x, y) coordinates, and then calculate the aspect ratio of the eyes for each frame; 4. Judgment indicators: 1) Count the number of times when the aspect ratio of the eyes approaches 0 and the proportion of the time restored to the normal level within each hour (here it refers to the proportion of the first time); 2) Count the number of times when the aspect ratio of the eyes is less than 0.5A and the proportion of the duration within each hour (here it refers to the proportion of the second time). If the proportion of the duration when the aspect ratio of the eyes is less than 0.5*A within 1 hour is greater than or equal to 50% (30 minutes), and / or, the number of blinks within 1 hour (here it refers to the number of times when the aspect ratio approaches 0) is greater than 60*15 (the frame rate of collecting pictures is 20 / s, and the normal number of blinks per minute is less than 15), it indicates that the user's eyes are in a fatigued state, and then perform the subsequent adjustment operation for the fatigued state.

[0082] The eye fatigue detection method provided by the embodiments of the present application first obtains a reference eye image of a user currently wearing the head-mounted device, and after obtaining the reference eye image, obtains consecutive-frame eye images within a preset time period; the reference eye image is used to represent the eye image of the user in a normal eye state; further, according to the reference eye image and the consecutive-frame eye images, through an eye fatigue detection model, the eye closure degree of the user in the normal eye state and the eye closure degree of the user in the consecutive-frame eye state are respectively determined; wherein, the eye fatigue detection model is obtained by training a deep learning network model; further, according to the eye closure degree of the user in the normal eye state and the eye closure degree of the user in the consecutive-frame eye state, it is determined whether the user's eyes are in a fatigued state. Therefore, the present application uses the user's own eye image in the normal eye state as a reference image, performs logical processing with the eye images continuously collected within the subsequent preset time period, and respectively determines the eye closure degree of the user in different eye states through the eye fatigue detection model, realizing the judgment of eye fatigue based on the user's personalized eye closure degree, rather than calculating the corresponding eye closure degree of different users according to a unified standard or benchmark to judge whether there is eye fatigue, thereby improving the accuracy of fatigue detection and meeting personalized needs.

[0083] In a possible design, the determining whether the user's eyes are in a fatigued state according to the eye closure degree of the user in the normal eye state and the eye closure degree of the user in the consecutive-frame eye state includes:

[0084] According to the eye closure degree of the user in the normal eye state and the eye closure degree of the user in the consecutive-frame eye state, through a blinking judgment strategy and / or an eye closure degree judgment strategy, it is determined whether the user's eyes are in a fatigued state.

[0085] In this embodiment, the eye closure degree of the user in the consecutive-frame eye state is combined with the eye closure degree of the user in the normal eye state, and through at least one dimension of judgment indexes, such as the number of blinks, the proportion of blinking time, the proportion of the time when the eye closure degree of the user in the consecutive-frame eye state is less than the corresponding eye aspect ratio threshold of the user in the normal eye state, etc., to judge whether there is eye fatigue. Through the diversification of judgment indexes, the diversification of eye fatigue detection is realized.

[0086] In a possible design, the degree of eye closure is used to represent the eye aspect ratio; determining whether the user's eyes are in a fatigued state according to the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the eye states of consecutive frames through a blinking judgment strategy and / or an eye closure degree judgment strategy includes:

[0087] Determining the number of blinks within the preset time period according to the degree of eye closure of the user in the eye states of consecutive frames, and determining the first time proportion of the eye state of the user returning to the normal eye state of the user during blinking according to the degree of eye closure of the user in the normal eye state; if the number of blinks is greater than the first preset number threshold, and / or, if the first time proportion is greater than the first preset time proportion threshold, it is determined that the user's eyes are in a fatigued state; and / or,

[0088] Multiplying the degree of eye closure of the user in the normal eye state by the corresponding weight to obtain an eye aspect ratio threshold; determining the second time proportion within the preset time period during which the degree of eye closure of the user in the eye states of consecutive frames is less than the eye aspect ratio threshold according to the degree of eye closure of the user in the eye states of consecutive frames; if the second time proportion is greater than the first preset time proportion threshold, it is determined that the user's eyes are in a fatigued state.

[0089] In this embodiment, the statistics of the number of blinks here is based on setting a threshold approaching 0. If the degree of eye closure is less than or equal to this threshold, it means that the user is in a blinking state, and the images approaching 0 within consecutive frames are regarded as one blink.

[0090] Specifically, for the blinking judgment strategy (Method 1, Method 2) and / or the eye closure degree judgment strategy (Method 3), it can be implemented in the following multiple ways:

[0091] Method 1: By judging whether the degree of eye closure in the eye states of each frame of the user in consecutive frames approaches 0, calculating the number of blinks of the user within the preset time period (for example, within 1 hour), if the number of blinks is greater than 60 * 15 (the frame rate of image acquisition is 20 / s, and the normal number of blinks per minute is less than 15), it can be judged that the user's eyes are in a fatigued state;

[0092] Method 2: By calculating the comparison between the first time proportion of the eye state of the user returning to the normal eye state of the user during blinking and the first preset time proportion threshold (for example, taking the 1-hour statistics as an example, the threshold here can be 30 minutes), if the first time proportion is greater than the first preset time proportion threshold, it can be judged that the user's eyes are in a fatigued state;

[0093] Method 3: Respectively compare the degree of eye closure in each frame of the user's eye state in consecutive frames with the eye aspect ratio threshold (here it refers to 0.5 * the degree of eye closure A of the user in the normal eye state), and calculate the second time ratio that the degree of eye closure in the eye state of consecutive frames is less than the eye aspect ratio threshold. If the second time ratio is greater than the first preset time ratio threshold, it can be determined that the user's eyes are in a fatigued state.

[0094] Method 4: Free combination of Method 1, Method 2, and Method 3, which will not be elaborated here.

[0095] Therefore, through any of the above methods, it can be determined whether the user's eyes are in a fatigued state, achieving diversified detection and ensuring the accuracy of detection.

[0096] In a possible design, determining the number of blinks within the preset time period according to the degree of eye closure in the eye state of consecutive frames of the user, and determining the first time ratio that the eye state returns to the normal eye state of the user during the blinking process according to the degree of eye closure in the normal eye state of the user, includes:

[0097] Taking the degree of eye closure in the normal eye state of the user and the degree of eye closure being 0 as reference lines respectively, and drawing the eye aspect ratio curve corresponding to the user according to the degree of eye closure in the eye state of consecutive frames of the user and the degree of eye closure in the normal eye state of the user; the horizontal axis of the eye aspect ratio curve is time, and the vertical axis is the eye aspect ratio.

[0098] According to the eye aspect ratio curve, determine the number of blinks, the first time ratio, and the second time ratio respectively.

[0099] In this embodiment, for the convenience of statistical calculation, it can be presented in the form of a curve graph, which is intuitive and not prone to errors. Specifically, taking the degree of eye closure in the normal eye state of the user and the degree of eye closure being 0 as reference lines respectively, the horizontal axis of the eye aspect ratio curve is time, and the vertical axis is the eye aspect ratio. Draw the degree of eye closure in the eye state of consecutive frames of the user and the degree of eye closure in the normal eye state of the user to obtain the eye aspect ratio curve corresponding to the user. Among them, the number of blinks, the first time ratio, and the second time ratio can also be determined through the eye aspect ratio curve. The first time ratio is the time ratio that the eye state returns to the normal eye state of the user during the blinking process, and the second time ratio is the time ratio that the degree of eye closure (aspect ratio) is less than the eye aspect ratio threshold.

[0100] In a possible design, the method further includes:

[0101] If it is determined that the user's eyes are in a fatigued state, determine the fatigue level according to at least one of the number of blinks, the first time ratio, and the second time ratio;

[0102] According to the correlation between the fatigue level and the perceived distance of the virtual image displayed in the head-mounted device from the eyes, adjust the perceived distance of the currently displayed virtual image in the head-mounted device from the eyes;

[0103] Wherein, the correlation is determined by the minimum distance and the maximum distance supported by the head-mounted device and the degree of eye closure of the user in a normal eye state.

[0104] In this embodiment, the current perceived distance can be adjusted based on the determined correlation between the fatigue level and the perceived distance of the virtual image displayed in the head-mounted device from the eyes; it can also be that if it is determined that the user's eyes are in a fatigued state and the user wears the head-mounted device continuously for more than a certain duration (such as 2 hours) and the display focal planes are all located near the farthest distance, the display screen can be forcibly exited to remind the user to take a rest. Based on different users, through this eye fatigue detection method, the personalized needs of users are met, rather than judging whether there is eye fatigue by calculating the degree of eye closure corresponding to different users according to a unified standard, thereby realizing the strategy of relieving eye fatigue.

[0105] In a possible design, the head-mounted device is used to support the user to customize the opening of the fatigue detection function; before obtaining the eye reference image of the user currently wearing the head-mounted device, the method further includes:

[0106] Respond to the instruction operation of the user currently wearing the head-mounted device to turn on the fatigue detection function;

[0107] Prompt the user to look straight ahead to collect the eye reference image of the user.

[0108] In this embodiment, it supports the user to customize whether to turn on the fatigue monitoring function and configure it according to needs, which can meet the personalized needs of users.

[0109] Exemplarily, as shown in Figure 5 When the eyes are fatigued, the three-dimensional image displayed by the VR glasses can be switched to a farther virtual image plane (focal plane) for display, so that the human eyes can focus on display planes at different distances, thereby relieving eye fatigue and preventing myopia.

[0110] Wherein, as shown in Figure 6As shown, taking a head-mounted device as an example, first, add user settings in the system (here referring to the head-mounted device): whether to perform fatigue detection. If it is enabled, the infrared camera of the head-mounted device is turned on for data collection, and the user is prompted to face forward to collect data on the eye closure state under normal conditions (here referring to collecting the infrared eye images of the user under normal eye conditions). This data is input into the network for processing, and the eye feature points in the normal state are output, and the aspect ratio A of the eyes is calculated. At the same time, the infrared camera continuously collects data and inputs the data into the network for processing, outputs the eye feature points, calculates the aspect ratio of the eyes, records the aspect ratio of the eyes, and draws a curve of the aspect ratio of the eyes. Based on the aspect ratio A of the eyes in the normal state and the drawn curve of the aspect ratio of the eyes, perform the following operations:

[0111] 1. Count the number of times the aspect ratio of the eyes approaches 0 within each hour (determine the number of blinks or calculate the number of frames within one hour, and calculate the number of times the aspect ratio of the eyes in each frame approaches 0) and the proportion of time to return to the normal level (here referring to the first-time proportion); 2. Count the number of times the aspect ratio of the eyes is less than 0.5A within each hour (calculate the number of frames within one hour and calculate the aspect ratio A of the eyes in each frame) and the proportion of time (here referring to the second-time proportion).

[0112] 1. If the number of times the aspect ratio approaches 0 is greater than 15 * 60 (frame rate / 20s); 2. If the time when the aspect ratio of the eyes is less than 0.5A within 1 hour is greater than 30 minutes, determine eye fatigue and send a notification to the system. The system will switch the displayed 3D image to a farther virtual image plane (focal plane) for display, and end the detection.

[0113] Specifically, for the normal state, the perceived distance of the VR virtual image from the eyes is Near: 2 meters, and the farthest achievable focal plane display distance (here referring to the perceived distance of the virtual image from the eyes) is set to Far: 6 meters (the specific distance value can be adjusted according to the diopter of each VR glasses).

[0114] If it is detected that within 1 hour, the time when the eye closure degree (degree of eye closure) is less than 0.5 * the closure degree (degree of eye closure) in the normal state is 30 minutes or the number of blinks It is higher than the initial set value I0 (0 < It - I0 < 10), and the eye fatigue level is defined as L0 = 0.5 (maximum is 1), then send a notification to the system to adjust the perceived distance of the virtual image from the eyes to Near + (Far - Near) * L0 = 4.

[0115] If it is detected that within 1 hour, the time when the eye closure degree is less than 0.5 times the closure degree in the normal state is X minutes, or the number of blinks It is higher than the initial set value I0 (10 < It - I0 < 50), then it is determined that the eye fatigue level is L1 = 0.5 + 0.5 * (X - 30) / 30 or L1 = 0.5 + 0.5 * (It - I0) / 50, and then a notification is sent to the system to adjust the perceived distance of the virtual image from the eyes to Near + (Far - Near) * L0.

[0116] If it lasts for 2 hours and the display focal plane (virtual image) is always located near the farthest position, the user can be reminded to take a break by forcibly exiting the display screen.

[0117] In this application, the eye closure state of the user in the normal state is used as a reference benchmark, avoiding misjudgment caused by the deviation of the image acquisition angle due to different wearing angles of the VR glasses, adding the number of blinks as a supplementary judgment of fatigue, and reducing the probability of misjudgment; by defining the eye fatigue level, the perceived distance of the virtual image from the eyes is adjusted according to the virtual image focal plane adjustment algorithm to relieve eye fatigue, and the three-dimensional display focal plane is switched to effectively relieve eye fatigue and prevent myopia or the deepening of myopia.

[0118] In order to implement the eye fatigue detection method, this embodiment provides an eye fatigue detection device. Refer to Figure 7 , Figure 7 which is a schematic structural diagram of the eye fatigue detection device provided by the embodiment of the present application; the eye fatigue detection device includes: an image acquisition module 701, an image processing module 702, and a fatigue detection module 703.

[0119] Among them, the image acquisition module 701 is used to obtain the eye reference image of the user currently wearing the head-mounted device, and after obtaining the eye reference image, obtain the consecutive frame eye images within a preset time period; the eye reference image is used to represent the eye image of the user in the normal state of the eyes.

[0120] The image processing module 702 is used to determine the eye closure degree of the user in the normal state of the eyes and the eye closure degree of the user in the consecutive frame eye states respectively through the eye fatigue detection model according to the eye reference image and the consecutive frame eye images; among them, the eye fatigue detection model is obtained by training a deep learning network model.

[0121] The fatigue detection module 703 is used to determine whether the eyes of the user are in a fatigued state according to the eye closure degree of the user in the normal state of the eyes and the eye closure degree of the user in the consecutive frame eye states.

[0122] In this embodiment, an image acquisition module 701, an image processing module 702, and a fatigue detection module 703 are used to obtain a reference eye image of a user currently wearing the head-mounted device, and after obtaining the reference eye image, obtain consecutive frame eye images within a preset time period; the reference eye image is used to represent the eye image of the user in a normal eye state; further, according to the reference eye image and the consecutive frame eye images, an eye fatigue detection model is used to respectively determine the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive frame eye state; wherein, the eye fatigue detection model is obtained by training a deep learning network model; further, according to the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive frame eye state, it is determined whether the user's eyes are in a fatigued state. Therefore, this application uses the user's own eye image in the normal eye state as a reference image, performs logical processing with the consecutively acquired eye images within a subsequent preset time period, and respectively determines the degree of eye closure of the user in different eye states through the eye fatigue detection model, realizing the judgment of eye fatigue based on the user's personalized degree of eye closure, rather than calculating the corresponding degree of eye closure of different users according to a unified standard or benchmark to determine whether there is eye fatigue, thereby improving the accuracy of fatigue detection and meeting personalized needs.

[0123] The eye fatigue detection device provided in this embodiment can be used to execute the technical solution of the above-mentioned eye fatigue detection method embodiment, and its implementation principle and technical effect are similar, and will not be elaborated here in this embodiment.

[0124] In a possible design, the fatigue detection module 703 is specifically configured to:

[0125] According to the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive frame eye state, determine whether the user's eyes are in a fatigued state through a blinking judgment strategy and / or an eye closure degree judgment strategy.

[0126] In a possible design, the degree of eye closure is used to represent the eye aspect ratio; the fatigue detection module 703 is specifically configured to:

[0127] Determine the number of blinks within the preset time period according to the degree of eye closure of the user in the eye states of consecutive frames, and determine the first time proportion of the eye state returning to the normal eye state of the user during the blinking process according to the degree of eye closure of the user in the normal eye state; if the number of blinks is greater than the first preset number threshold, and / or, if the first time proportion is greater than the first preset time proportion threshold, then determine that the user's eyes are in a fatigued state; and / or,

[0128] Multiply the degree of eye closure of the user in the normal eye state by the corresponding weight to obtain an eye aspect ratio threshold; determine the second time proportion within the preset time period during which the degree of eye closure of the user in the eye states of consecutive frames is less than the eye aspect ratio threshold according to the degree of eye closure of the user in the eye states of consecutive frames; if the second time proportion is greater than the first preset time proportion threshold, then determine that the user's eyes are in a fatigued state.

[0129] In a possible design, the fatigue detection module 703 is specifically configured to:

[0130] Take the degree of eye closure of the user in the normal eye state and the degree of eye closure of 0 as reference lines respectively, and draw an eye aspect ratio curve corresponding to the user according to the degree of eye closure of the user in the eye states of consecutive frames and the degree of eye closure of the user in the normal eye state; the horizontal axis of the eye aspect ratio curve is time, and the vertical axis is the eye aspect ratio;

[0131] Determine the number of blinks, the first time proportion, and the second time proportion respectively according to the eye aspect ratio curve.

[0132] In a possible design, the device further includes: a fatigue relief processing module; the fatigue relief processing module is configured to:

[0133] When it is determined that the user's eyes are in a fatigued state, determine a fatigue level according to at least one of the number of blinks, the first time proportion, and the second time proportion;

[0134] Adjust the perceived distance between the currently displayed virtual image in the head-mounted device and the eyes according to the association relationship between the fatigue level and the perceived distance between the virtual image displayed in the head-mounted device and the eyes;

[0135] Wherein, the association relationship is determined by the minimum distance, the maximum distance supported by the head-mounted device, and the degree of eye closure of the user in the normal eye state.

[0136] In a possible design, the head-mounted device is used to support the user to customize and enable the fatigue detection function; the device further includes: a fatigue detection start module; the fatigue detection start module is used for:

[0137] Before obtaining the eye reference image of the user currently wearing the head-mounted device, in response to the instruction operation of the user currently wearing the head-mounted device, the fatigue detection function is enabled;

[0138] Prompt the user to look straight ahead to collect the eye reference image of the user.

[0139] In a possible design, the image processing module 702 is specifically used for:

[0140] Input the eye reference image and the eye images of the continuous frames into the eye fatigue detection model respectively, to obtain the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the eye states of the continuous frames; or,

[0141] Input the eye reference image and the eye images of the continuous frames into the eye fatigue detection model respectively, to obtain the positions of the feature points of the user in the normal eye state and the positions of the feature points of the user in the eye states of each frame in the continuous frames; according to the positions of the feature points of the user in the normal eye state and the positions of the feature points of the user in the eye states of each frame in the continuous frames, through the eye closure degree model, obtain the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the eye states of the continuous frames respectively; wherein, the eye closure degree model is determined by the positions of the feature points.

[0142] To implement the above-mentioned eye fatigue detection method, this embodiment provides a head-mounted device, in combination with Figure 1 and Figure 2 As shown, the head-mounted device includes a data acquisition device and the eye fatigue detection device; wherein, the eye fatigue detection device is used to execute the method described in any item of the first aspect;

[0143] The data acquisition device includes a camera; the camera is used to collect images of the eyes at any moment or in any eye state.

[0144] The head-mounted device provided in this embodiment can be used to execute the technical solutions of the above method embodiments, and its implementation principles and technical effects are similar, which will not be elaborated here in this embodiment.

[0145] To implement the method of the above embodiment, this embodiment provides an electronic device. Figure 8 It is a schematic structural diagram of the electronic device provided in the embodiment of the present application. As Figure 8As shown in the figure, the electronic device of this embodiment includes: a processor 801 and a memory 802; among them, the memory 802 is used to store computer-executable instructions; the processor 801 is used to execute the computer-executable instructions stored in the memory to implement each step executed in the above embodiment. For specific details, please refer to the relevant descriptions in the foregoing method embodiments.

[0146] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the processor executes the computer-executable instructions, the above method is implemented.

[0147] An embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by the processor, the above method is implemented.

[0148] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be in an electrical, mechanical or other form. In addition, in each embodiment of the present application, each functional module can be integrated in a processing unit, or each module can exist physically alone, or two or more modules can be integrated in a unit. The unit formed by the above modules can be implemented in the form of hardware, or in the form of a hardware plus a software functional unit.

[0149] The integrated module implemented in the form of software functional modules can be stored in a computer-readable storage medium. The above software functional modules stored in a storage medium include several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to execute some steps of the methods in various embodiments of the present application. It should be understood that the above processor may be a central processing unit (English: Central Processing Unit, abbreviated: CPU), or may also be other general-purpose processors, digital signal processors (English: Digital Signal Processor, abbreviated: DSP), application specific integrated circuits (English: Application Specific Integrated Circuit, abbreviated: ASIC), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0150] The memory may include high-speed RAM memory and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disc, etc. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus. The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0151] An exemplary storage medium is coupled to a processor, enabling the processor to read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a master device.

[0152] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: various media such as ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An eye fatigue detection method, characterized in that, Applied to a head-mounted device or an electronic device, the method includes: Obtaining a reference eye image of a user currently wearing the head-mounted device, and after obtaining the reference eye image, obtaining consecutive-frame eye images within a preset time period; the reference eye image is used to represent the eye image of the user in a normal eye state; According to the reference eye image and the consecutive-frame eye images, respectively determining the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive-frame eye state through an eye fatigue detection model; wherein, the eye fatigue detection model is obtained by training a deep learning network model; Determining whether the user's eyes are in a fatigued state according to the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive-frame eye state.

2. The method according to claim 1, wherein The determining whether the user's eyes are in a fatigued state according to the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive-frame eye state includes: Determining whether the user's eyes are in a fatigued state according to the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive-frame eye state through a blinking judgment strategy and / or an eye closure degree judgment strategy.

3. The method according to claim 2, characterized in that The degree of eye closure is used to represent the eye aspect ratio; the determining whether the user's eyes are in a fatigued state according to the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the consecutive-frame eye state through a blinking judgment strategy and / or an eye closure degree judgment strategy includes: Determining the number of blinks within the preset time period according to the degree of eye closure of the user in the consecutive-frame eye state, and determining the first time ratio of the eye state of the user returning to the normal eye state of the user during blinking according to the degree of eye closure of the user in the normal eye state; if the number of blinks is greater than a first preset number threshold, and / or, if the first time ratio is greater than a first preset time ratio threshold, then determining that the user's eyes are in a fatigued state; and / or, Multiplying the degree of eye closure of the user in the normal eye state by a corresponding weight to obtain an eye aspect ratio threshold; determining the second time ratio of the degree of eye closure of the user in the consecutive-frame eye state being less than the eye aspect ratio threshold within the preset time period according to the degree of eye closure of the user in the consecutive-frame eye state; if the second time ratio is greater than a first preset time ratio threshold, then determining that the user's eyes are in a fatigued state.

4. The method according to claim 3, wherein The determining the number of blinks within the preset time period according to the degree of eye closure of the user in the consecutive-frame eye state, and determining the first time ratio of the eye state of the user returning to the normal eye state of the user during blinking according to the degree of eye closure of the user in the normal eye state includes: Taking the degree of eye closure of the user in the normal eye state and the degree of eye closure of 0 as the reference lines respectively, draw the corresponding eye aspect ratio curve of the user according to the degree of eye closure of the user in the eye states of consecutive frames and the degree of eye closure of the user in the normal eye state; the horizontal axis of the eye aspect ratio curve is time, and the vertical axis is the eye aspect ratio; Determine the number of blinks, the first time ratio, and the second time ratio respectively according to the eye aspect ratio curve.

5. The method according to claim 3 or 4, characterized in that, The method further includes: If it is determined that the eyes of the user are in a fatigued state, determine the fatigue level according to at least one of the number of blinks, the first time ratio, and the second time ratio; Adjust the perceived distance between the currently displayed virtual image in the head-mounted device and the eyes according to the correlation between the fatigue level and the perceived distance between the virtual image displayed in the head-mounted device and the eyes; Wherein, the correlation is determined by the minimum distance and the maximum distance supported by the head-mounted device and the degree of eye closure of the user in the normal eye state.

6. The method according to any one of claims 1-4, characterized in that, The head-mounted device is used to support the user to customize and turn on the fatigue detection function; before obtaining the reference eye image of the user currently wearing the head-mounted device, the method further includes: Respond to the instruction operation of the user currently wearing the head-mounted device to turn on the fatigue detection function; Prompt the user to look straight ahead to collect the reference eye image of the user.

7. The method according to any one of claims 1 to 4, characterized in that, The step of respectively determining the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the eye states of consecutive frames through the eye fatigue detection model according to the reference eye image and the eye images of consecutive frames includes: Input the reference eye image and the eye images of consecutive frames into the eye fatigue detection model respectively to obtain the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the eye states of consecutive frames; or, Input the reference eye image and the eye images of consecutive frames into the eye fatigue detection model respectively to obtain the positions of the feature points of the user in the normal eye state and the positions of the feature points of the user in the eye state of each frame in the consecutive frames; according to the positions of the feature points of the user in the normal eye state and the positions of the feature points of the user in the eye state of each frame in the consecutive frames, respectively obtain the degree of eye closure of the user in the normal eye state and the degree of eye closure of the user in the eye states of consecutive frames through the eye closure degree model; wherein, the eye closure degree model is determined by the positions of the feature points.

8. An eye fatigue detection device, characterized in that, Applied to a head-mounted device or an electronic device; the device includes: An image acquisition module, configured to acquire a reference eye image of the user currently wearing the head-mounted device, and after acquiring the reference eye image, acquire eye images of consecutive frames within a preset time period; the reference eye image is used to represent the eye image of the user in the normal eye state; An image processing module, configured to determine, according to the eye reference image and the eye images of consecutive frames, the degree of eye closure of the user in a normal eye state and the degree of eye closure of the user in the eye states of consecutive frames respectively through an eye fatigue detection model; wherein, the eye fatigue detection model is obtained by training a deep learning network model; A fatigue detection module, configured to determine whether the eyes of the user are in a fatigued state according to the degree of eye closure of the user in a normal eye state and the degree of eye closure of the user in the eye states of consecutive frames.

9. A head-mounted device, characterized in that, Comprising: A data acquisition device and the eye fatigue detection device as described above; wherein, the eye fatigue detection device is configured to execute the method according to any one of claims 1-7; The data acquisition device includes a camera; the camera is configured to acquire images of the eyes at any moment or in any eye state.

10. An electronic device, characterized in that, Comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 7.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 7.

Images