Visual fatigue detection method, device, terminal equipment and system

Abstract

The invention is suitable for the technical field of intelligent equipment, and provides an asthenopia detection method, device, terminal equipment and system.The method comprises the steps that multiple pieces of eye image data of a user within a preset time period are obtained, and the eyeballs of the user continuously move within the preset time period; image analysis is carried out on the multiple pieces of eye image data to obtain multi-dimensional eye data of the user, and the multi-dimensional eye data at least comprises blink data, pupil reaction data and eye movement behavior data; performing eye fatigue detection on the user according to the multi-dimensional eye data to obtain an asthenopia detection result of the user; if the asthenopia detection result represents that the eyes of the user are in the fatigue state, asthenopia prompt information corresponding to the asthenopia detection result is output, and the asthenopia prompt information is used for prompting the user to conduct eye use management. According to the method, the data basis of asthenopia detection can be enriched, and the accuracy of asthenopia detection results is improved, so that the overall reliability of asthenopia detection is improved.

Description

Technical Field

[0001] This application belongs to the field of intelligent device technology, and in particular relates to a method, device, terminal equipment and system for detecting visual fatigue. Background Technology

[0002] Visual fatigue, also known as eye strain, is a common condition with main symptoms including eye fatigue, headache, blurred vision, and dry eyes. It can seriously affect users who engage in activities that require intense visual focus for extended periods, such as those who work with computers, read, or drive, reducing productivity and quality of life.

[0003] However, the existing methods for detecting visual fatigue have a relatively limited data base, which limits the accuracy of the results and reduces the overall reliability of visual fatigue detection. Summary of the Invention

[0004] This application provides a method, apparatus, terminal device, and system for detecting visual fatigue, which can enrich the data foundation for visual fatigue detection, improve the accuracy of visual fatigue detection results, and thus enhance the overall reliability of visual fatigue detection.

[0005] In a first aspect, embodiments of this application provide a method for detecting visual fatigue, including:

[0006] Acquire multiple eye image data of the user within a predetermined time period, during which the user's eyeballs move continuously;

[0007] Image analysis is performed on multiple eye image data to obtain multi-dimensional eye data of users. Multi-dimensional eye data includes at least blink data, pupil response data and eye movement behavior data.

[0008] Based on multi-dimensional eye data, eye fatigue is detected in users to obtain the results of their visual fatigue detection.

[0009] If the visual fatigue detection result indicates that the user's eyes are in a state of fatigue, then the visual fatigue prompt message corresponding to the visual fatigue detection result will be output. The visual fatigue prompt message is used to remind the user to manage their eye use.

[0010] Optionally, image analysis is performed on multiple eye image datasets to obtain multi-dimensional eye data of the user, including:

[0011] By analyzing the eyelid positions of multiple eye image data, the user's blinking data can be obtained;

[0012] Pupil position is analyzed from multiple eye image data to obtain the user's pupil response data;

[0013] Eye movement analysis is performed on multiple eye image data to calculate the user's eye movement behavior data.

[0014] Optionally, the eyelid position of multiple eye image data is analyzed to obtain the user's blinking data, including:

[0015] An image segmentation algorithm was used to identify eyelids from multiple eye image datasets to obtain the eyelid positions in the multiple eye image datasets;

[0016] Based on the eyelid position in multiple eye image data, calculate the palpebral fissure width of the user in each eye image data, and calculate the variation index corresponding to the palpebral fissure width;

[0017] Based on the eyelid position in multiple eye image data, blinking action is detected in the eye image data to obtain the user's blinking action;

[0018] The duration and interval of a user's blinks are calculated based on the user's blinking action, and the variation indexes corresponding to the duration and interval of blinks are calculated respectively.

[0019] Optionally, the pupil position of multiple eye image data is analyzed to obtain the user's pupil response data, including:

[0020] Image segmentation algorithms are used to identify pupils in multiple eye image datasets to obtain the pupil positions in the multiple eye image datasets;

[0021] Based on the pupil position in multiple eye image datasets, the pupil size of the user in each eye image dataset is calculated, along with the corresponding variation index for the pupil size.

[0022] Optionally, eye-tracking analysis is performed on multiple eye image data to calculate the user's eye-tracking behavior data, including:

[0023] Deep learning algorithms are used to detect gaze actions on multiple eye image data to obtain the user's gaze actions;

[0024] The number of times a user gazes and the duration of that gaze are calculated based on the user's gaze actions.

[0025] Based on the number of fixations and the duration of fixation, the user's eye movement data is determined. The eye movement data includes at least the average fixation duration and the mean square error of the fixation duration.

[0026] Optionally, eye fatigue detection can be performed on users based on multi-dimensional eye data to obtain the user's visual fatigue detection results, including:

[0027] The upper limit critical flash fusion frequency and the lower limit critical flash fusion frequency detected by the flashing device are obtained. The upper limit critical flash fusion frequency is used to characterize the maximum frequency of the continuous light source perceived by the user, and the lower limit critical flash fusion frequency is used to characterize the minimum frequency of the flashing light source perceived by the user.

[0028] The average value of the upper limit critical flash fusion frequency and the lower limit critical flash fusion frequency is determined as the user's average flash fusion frequency;

[0029] Based on multi-dimensional eye data and average flash fusion frequency, eye fatigue is detected in users, and the results of the eye fatigue detection are obtained. The results include one of the following: no eye fatigue, mild eye fatigue, moderate eye fatigue, and severe eye fatigue.

[0030] Secondly, embodiments of this application provide an eye fatigue detection device, comprising:

[0031] The acquisition module is used to acquire multiple eye image data of the user within a predetermined time period, wherein the user's eyeballs move continuously within the predetermined time period;

[0032] The image analysis module is used to perform image analysis on multiple eye image data to obtain the user's multi-dimensional eye data, which includes at least blink data, pupil response data, and eye movement behavior data.

[0033] The visual fatigue detection module is used to detect eye fatigue in users based on multi-dimensional eye data and obtain the user's visual fatigue detection results;

[0034] The visual fatigue alert module is used to output visual fatigue alert information corresponding to the visual fatigue detection result if the visual fatigue detection result indicates that the user's eyes are in a state of fatigue. The visual fatigue alert information is used to remind the user to manage their eye use.

[0035] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any of the first aspects.

[0036] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any of the first aspects.

[0037] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute any of the methods described in the first aspect above.

[0038] Sixthly, embodiments of this application provide an eye fatigue detection system, including:

[0039] A head-mounted device used to collect multiple eye image data of a user within a predetermined time period, during which the user's eyeballs move continuously;

[0040] The terminal device is used to perform image analysis on multiple eye image data to obtain multi-dimensional eye data of the user. The multi-dimensional eye data includes at least blink data, pupil response data, and eye movement behavior data. Based on the multi-dimensional eye data, the device performs eye fatigue detection on the user to obtain the user's visual fatigue detection result. If the visual fatigue detection result indicates that the user's eyes are in a state of fatigue, the device outputs visual fatigue prompt information corresponding to the visual fatigue detection result. The visual fatigue prompt information is used to remind the user to manage their eye use.

[0041] Optionally, the visual fatigue detection system also includes:

[0042] The flashing device is used to detect the user's upper and lower critical flash fusion frequencies;

[0043] The terminal device is also used to determine the average of the upper limit critical flash fusion frequency and the lower limit critical flash fusion frequency as the user's average flash fusion frequency; based on multi-dimensional eye data and the average flash fusion frequency, it performs eye fatigue detection on the user and obtains the user's visual fatigue detection results.

[0044] This application provides a method, apparatus, terminal device, and system for detecting eye fatigue. The method includes: acquiring multiple eye image data of a user within a predetermined time period, wherein the user's eyeballs move continuously during the predetermined time period; performing image analysis on the multiple eye image data to obtain multi-dimensional eye data of the user, the multi-dimensional eye data including at least blink data, pupillary response data, and eye movement behavior data; performing eye fatigue detection on the user based on the multi-dimensional eye data to obtain the user's eye fatigue detection result; if the eye fatigue detection result indicates that the user's eyes are in a fatigued state, then outputting eye fatigue prompt information corresponding to the eye fatigue detection result, the eye fatigue prompt information being used to prompt the user to manage eye use. Using the above technical solution, by performing image analysis on multiple eye image data of the user to obtain multi-dimensional eye data of the user, wherein the multi-dimensional eye data includes at least blink data, pupillary response data, and eye movement behavior data, the data foundation for eye fatigue detection can be enriched, the accuracy of eye fatigue detection results can be improved, thereby enhancing the overall reliability of eye fatigue detection. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic flowchart of an embodiment of an eye fatigue detection method provided in this application;

[0047] Figure 2 This is a schematic flowchart of a visual fatigue detection method provided in another embodiment of this application;

[0048] Figure 3 This is a schematic diagram of the structure of a head-mounted device according to an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the structure of a flashing device provided in an embodiment of this application;

[0050] Figure 5 This is a structural block diagram of an eye fatigue detection device provided in one embodiment of this application;

[0051] Figure 6 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0052] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0053] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0054] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0055] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0056] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0057] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0058] It should be noted that the information collection process (such as the eye image collection process, fingerprint information collection process, etc.) / feature extraction process involved in this application is carried out with the user's knowledge and permission. That is, the information collection process / feature extraction process complies with the requirements of laws and regulations and does not constitute an act that harms the public interest.

[0059] The visual fatigue detection method provided in this application can be applied to terminal devices such as mobile phones, tablets, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of terminal device.

[0060] It can be argued that existing methods for detecting visual fatigue typically rely on single physiological indicators and lack a comprehensive data foundation, leading to an incomplete understanding of visual fatigue and thus limiting the effectiveness of visual fatigue detection. Furthermore, existing methods lack objective and repeatable diagnostic tools. For example, measuring key flicker fusion values ​​usually requires expert operation, making it impractical for users to operate the system themselves without professional assistance, thus being unfriendly to ordinary users in visual fatigue detection.

[0061] Based on this, the embodiments of this application provide a method for detecting visual fatigue, which can use a combination of data such as blink index, pupil response, eye movement behavior and critical flicker fusion frequency to detect and evaluate visual fatigue, thereby providing users with targeted visual fatigue prompts and realizing scientific eye management for users.

[0062] Figure 1 This is a schematic flowchart of an embodiment of an eye fatigue detection method provided in this application. It is intended as an example and not a limitation. This method can be applied to terminal devices, such as... Figure 1 As shown, the method includes:

[0063] S101. Acquire multiple eye image data of the user within a predetermined time period, wherein the user's eyeballs move continuously within the predetermined time period.

[0064] The scheduled time period can be a pre-specified time interval, such as a periodic time period with a set duration at each interval, or a pre-specified fixed time period. During the scheduled time period, the user will continuously use their eyes, and the user's eyeballs will continuously move.

[0065] The method of acquiring multiple eye image data of the user is not limited. For example, multiple eye images can be continuously captured by an infrared camera or other image acquisition device mounted on a head-mounted device. Alternatively, eye video data can be captured and collected during a visual task within a predetermined time period (such as a specified 1 minute), and then the captured eye video data can be analyzed frame by frame to obtain multiple eye image data of the user.

[0066] S102. Perform image analysis on multiple eye image data to obtain multi-dimensional eye data of the user. The multi-dimensional eye data includes at least blink data, pupil response data and eye movement behavior data.

[0067] Multidimensional eye data can be understood as data related to the user's eyes obtained from different analytical dimensions. For example, multidimensional eye data can include blink data, pupillary response data, and eye movement behavior data, as well as other types of related data. Blink data can refer to data related to the user's blinking, such as the number of blinks, average blink duration, and percentage of eye closure. Pupil response data can refer to data related to pupillary response, such as pupil size and pupillary accommodation speed. Eye movement behavior data can refer to the user's eye movement data, such as movement trajectory, number of fixations, and eye movement distance. Of course, various types of multidimensional eye data can also include other data besides those mentioned above; this embodiment does not limit this.

[0068] Specifically, after obtaining multiple eye image data of the user, this embodiment can perform image analysis on the multiple eye image data to obtain multi-dimensional eye data of the user. The specific means of obtaining multi-dimensional eye data are not limited. For example, an image analysis model can be used to directly input the multiple eye image data of the user into the image analysis model to output the multi-dimensional eye data of the user. The image analysis model can be a pre-trained neural network model used to output the multi-dimensional eye data corresponding to the eye image data. The specific training process of the image analysis model will not be further elaborated here. Alternatively, this embodiment can also obtain the user's eye data in various dimensions by performing certain calculations on the multiple eye image data separately. The calculation methods corresponding to different dimensions can be different.

[0069] S103. Perform eye fatigue detection on users based on multi-dimensional eye data to obtain the user's visual fatigue detection results.

[0070] After obtaining the user's multi-dimensional eye data through the above steps, the user's eye fatigue can be detected directly based on the obtained multi-dimensional eye data, thereby obtaining the user's visual fatigue detection results. Furthermore, it can be combined with other user data to comprehensively complete the user's eye fatigue detection.

[0071] In some embodiments, eye fatigue detection is performed on the user based on multi-dimensional eye data to obtain the user's visual fatigue detection results, including:

[0072] The upper limit critical flash fusion frequency and the lower limit critical flash fusion frequency detected by the flashing device are obtained. The upper limit critical flash fusion frequency is used to characterize the maximum frequency of the continuous light source perceived by the user, and the lower limit critical flash fusion frequency is used to characterize the minimum frequency of the flashing light source perceived by the user.

[0073] The average value of the upper limit critical flash fusion frequency and the lower limit critical flash fusion frequency is determined as the user's average flash fusion frequency;

[0074] Based on multi-dimensional eye data and average flash fusion frequency, eye fatigue is detected in users, and the results of the eye fatigue detection are obtained. The results include one of the following: no eye fatigue, mild eye fatigue, moderate eye fatigue, and severe eye fatigue.

[0075] The flashing device can be used to detect the upper and lower critical flash fusion frequencies of the user. The flashing device can be composed of a red light-emitting diode (LED) lamp with a wavelength of 660nm and a brightness adjustment device. The red light-emitting diode (LED) lamp has a flashing frequency of 20-70Hz, and the minimum adjustable range of the flashing frequency is 1Hz. The brightness adjustment device can be used to adjust the light source intensity to suit the user's habits.

[0076] The specific steps for the flashing device to detect the critical flash fusion frequency may include: the user operating a wireless remote control to gradually increase the red light frequency until the human eye perceives a continuous light source, and recording this frequency as the upper critical flash fusion frequency; and gradually decreasing the red light frequency until the human eye perceives a flickering light source, recording this frequency as the lower critical flash fusion frequency. Furthermore, the flashing device can connect to an image acquisition device (such as a head-mounted device) via Bluetooth, or to a terminal device (such as a computer) via a serial cable, thereby transmitting the detected upper and lower critical flash fusion frequencies to the head-mounted device's memory card or a designated storage address on the computer for subsequent processing.

[0077] In a specific implementation, the terminal device can acquire the upper and lower critical flash fusion frequencies detected by the flicker device and calculate the user's average flash fusion frequency. Based on multi-dimensional eye data and the calculated average flash fusion frequency, the device can comprehensively detect the user's eye fatigue and obtain an eye fatigue detection result. This embodiment does not limit the specific methods of eye fatigue detection. For example, multi-dimensional eye data and the average flash fusion frequency can be imported into a machine learning model to comprehensively assess the degree of eye fatigue and output an eye fatigue detection result. Alternatively, the collected data can be compared with their respective benchmark data to comprehensively evaluate visual fatigue and thus fully assess the degree of visual fatigue. For instance, the user's eye fatigue detection classification result can be determined based on the number of eye data points that exceed their respective benchmark data points. If all of the user's eye data points exceed their respective benchmark data points, the user's eye fatigue detection result can be considered severe eye fatigue. The benchmark data can be pre-configured data thresholds used to individually measure the user's eye fatigue state.

[0078] S104. If the visual fatigue detection result indicates that the user's eyes are in a state of fatigue, then output the visual fatigue prompt information corresponding to the visual fatigue detection result. The visual fatigue prompt information is used to remind the user to manage their eye use.

[0079] In a specific implementation, the terminal device can detect and provide feedback in real time, offering users immediate alerts and suggestions. For example, it can output different visual fatigue warning messages based on different visual fatigue detection results, thereby enabling timely intervention to manage and reduce visual fatigue. The output format of the visual fatigue warning message is not limited; it can include pop-ups or voice prompts, as long as it achieves the effect of prompting users to manage their eye use.

[0080] This embodiment provides a method for detecting eye fatigue. It acquires multiple eye image data of a user within a predetermined time period, during which the user's eyeballs continuously move. Image analysis is performed on the multiple eye image data to obtain multi-dimensional eye data of the user, including at least blink data, pupillary response data, and eye movement behavior data. Eye fatigue is detected based on this multi-dimensional eye data to obtain the user's eye fatigue detection result. If the eye fatigue detection result indicates that the user's eyes are in a state of fatigue, eye fatigue warning information corresponding to the detection result is output to prompt the user to manage their eye use. This method, by analyzing multiple eye image data of a user to obtain multi-dimensional eye data, including at least blink data, pupillary response data, and eye movement behavior data, enriches the data foundation for eye fatigue detection, improves the accuracy of the detection results, and thus enhances the overall reliability of eye fatigue detection.

[0081] Figure 2 This is a flowchart illustrating a visual fatigue detection method according to another embodiment of this application. This embodiment further optimizes the multi-dimensional eye data of the user by performing image analysis on multiple eye image data: analyzing the eyelid positions of multiple eye image data to obtain the user's blinking data; analyzing the pupil positions of multiple eye image data to obtain the user's pupillary response data; and performing eye movement analysis on multiple eye image data to calculate the user's eye movement behavior data. Figure 2 As shown, the method includes:

[0082] S201. Acquire multiple eye image data of the user within a predetermined time period, wherein the user's eyeballs move continuously within the predetermined time period.

[0083] S202. Analyze the eyelid position of multiple eye image data to obtain the user's blinking data.

[0084] As an example, blink data can be obtained by analyzing and calculating the eyelid positions of multiple eye image data. For instance, blink data can be obtained directly using a specific image analysis model, or an image segmentation algorithm can be used to identify eyelids in multiple eye image data to obtain the eyelid positions. Then, different blink data can be calculated based on the eyelid positions in the multiple eye image data. For example, the palpebral fissure width of the user in each eye image data can be calculated based on the eyelid positions in the multiple eye image data, as well as the corresponding variation index. Alternatively, blinking action detection can be performed on the eye image data based on the eyelid positions in the multiple eye image data to obtain the user's blinking action. Then, based on the user's blinking action, the blink duration and blink interval can be calculated, the mean square error of the blink duration can be calculated, the number of blinks and the percentage of eye closure can be counted, and the variation index corresponding to the blink duration and blink interval can be calculated respectively. Among them, the average blink duration can be the average of the duration of all blinking actions; the mean square error of blink duration (MSE) is the average of the squares of the differences between the duration of all blinks and the average blink duration, used to understand the variation in blink duration; the number of blinks is the total number of times the eyes close and open within a specific time; the percentage of eye closure refers to the percentage of time the eyes are closed, used to statistically measure the frequency and duration of eye closure.

[0085] S203. Analyze the pupil position of multiple eye image data to obtain the user's pupil response data.

[0086] As an example, pupil response data can be obtained by analyzing and calculating the pupil positions of multiple eye image data. For instance, pupil response data can be obtained directly based on a specific image analysis model, or an image segmentation algorithm can be used to identify pupils in multiple eye image data to obtain pupil positions. Then, based on the pupil positions in multiple eye image data, different pupil response data can be calculated. For example, pupil size and pupil accommodation speed can be measured. Pupil size can be the average pupil diameter measured per unit time, and pupil accommodation speed can refer to the rate at which the pupil adapts to changes in light intensity, i.e., the speed at which the pupil size changes when the eye reacts to changes in light or viewing distance, as well as the calculation of the variation index corresponding to pupil size.

[0087] In one feasible implementation, the process of calculating the variation index may include: plotting time distribution curves for pupil size, palpebral fissure width, blink duration, and blink interval based on the time series. The time distribution curve can be a dynamic change curve with time as the horizontal axis and the values ​​of each parameter as the vertical axis. Then, the standard deviation, variance, and coefficient of variation of each parameter within one minute are calculated and extracted. The standard deviation can be used to calculate the dispersion of the parameter value relative to its mean; the variance can be used to calculate the fluctuation range of the parameter value to assess the stability of the data; and the coefficient of variation can be used to calculate the ratio of the standard deviation to the mean, reflecting the relative fluctuation of the parameter value, expressed as a percentage.

[0088] S204. Perform eye movement analysis on multiple eye image data to calculate the user's eye movement behavior data.

[0089] As an example, eye movement analysis can be performed directly on multiple eye image datasets using a specific image analysis model to calculate the user's eye movement behavior data. Alternatively, deep learning algorithms can be used to detect gaze movements on multiple eye image datasets. By detecting the eye movement trajectory, the user's gaze movements can be detected. Based on these gaze movements, the number of gazes, gaze duration, and eye distance can be calculated. Furthermore, based on the number of gazes and gaze duration, data such as the user's average gaze duration and the mean squared error of gaze duration can be determined. The number of gazes can be the number of times the eye is stationary or makes slight movements within a very small range; the average gaze duration is the average of all gaze durations; the mean squared error of gaze duration (MSE) is the average of the squares of the differences between all gaze durations and the average gaze duration, used to measure the variability of gaze time; and the eye distance is the total distance the eye moves over a period of time, such as the distance between consecutive gaze points.

[0090] S205. Based on blink data, pupil response data, and eye movement behavior data, perform eye fatigue detection on the user to obtain the user's visual fatigue detection results.

[0091] S206. If the visual fatigue detection result indicates that the user's eyes are in a state of fatigue, then output the visual fatigue prompt information corresponding to the visual fatigue detection result. The visual fatigue prompt information is used to remind the user to manage their eye use.

[0092] This embodiment provides a method for detecting visual fatigue. By analyzing the eyelid position of multiple eye image data, analyzing the pupil position of multiple eye image data, and performing eye movement analysis on multiple eye image data, it can provide feasible technical means to obtain a rich data foundation and further improve the accuracy of visual fatigue detection results.

[0093] Corresponding to the visual fatigue detection method in the above embodiments, this application also provides a visual fatigue detection system, including:

[0094] A head-mounted device used to collect multiple eye image data of a user within a predetermined time period, during which the user's eyeballs move continuously;

[0095] The terminal device is used to perform image analysis on multiple eye image data to obtain multi-dimensional eye data of the user. The multi-dimensional eye data includes at least blink data, pupil response data, and eye movement behavior data. Based on the multi-dimensional eye data, the device performs eye fatigue detection on the user to obtain the user's visual fatigue detection result. If the visual fatigue detection result indicates that the user's eyes are in a state of fatigue, the device outputs visual fatigue prompt information corresponding to the visual fatigue detection result. The visual fatigue prompt information is used to remind the user to manage their eye use.

[0096] In some embodiments, the visual fatigue detection system further includes:

[0097] The flashing device is used to detect the user's upper and lower critical flash fusion frequencies;

[0098] The terminal device is also used to determine the average of the upper limit critical flash fusion frequency and the lower limit critical flash fusion frequency as the user's average flash fusion frequency; based on multi-dimensional eye data and the average flash fusion frequency, it performs eye fatigue detection on the user and obtains the user's visual fatigue detection results.

[0099] As an example, a user can wear a head-mounted device and browse text, videos, or matching templates displayed on the terminal device screen. The head-mounted device can establish a communication connection with the terminal device. Figure 3 This is a schematic diagram of the structure of a head-mounted device according to an embodiment of this application. Figure 3 (a) and (b) are the side view and front view of the head-mounted device, respectively. The high-resolution infrared camera 1 mounted on the head-mounted device can be configured below and in front of the user's eyes to perform binocular visual positioning of the eyes. It is used to collect multiple eye image data of the user within a predetermined time period and transmit the collected multiple eye image data to the terminal device. By capturing detailed eye images and videos, it is crucial for subsequent accurate detection of blinking, eye movement and pupil size, and accurate tracking and analysis. The head-mounted device can also be equipped with a distance sensor to measure the distance between the user and the terminal device screen to ensure that the user maintains the optimal viewing distance for measuring visual fatigue.

[0100] Meanwhile, users can control the start and stop of the flashing device by using the buttons on the wireless remote control, measure the upper and lower critical flash fusion frequencies, and transmit the measured upper and lower critical flash fusion frequencies to the terminal device. Figure 4 This is a schematic diagram of the structure of a flashing device provided in an embodiment of this application, as shown below. Figure 4 As shown, the flashing device can be placed on a desktop. The user can operate the equipped wireless remote control to gradually increase the red light frequency until the human eye perceives a continuous light source, and record this frequency as the upper limit critical flash fusion frequency; gradually decrease the red light frequency until the human eye perceives a flashing light source, and record this frequency as the lower limit critical flash fusion frequency.

[0101] The processor on the terminal device side (such as a computer) can be used to control the acquisition and storage of the above data, and transmit the acquired eye image data and critical flash fusion frequency measurement results to the storage device; the memory on the terminal device side may include a designated storage path of the computer connected to the head-mounted device and the built-in memory card of the head-mounted device, for storing the acquired eye image data and critical flash fusion frequency measurement results.

[0102] The computer program on the terminal device side can calculate the average value of the upper and lower critical flash fusion frequencies and determine the average value as the user's average flash fusion frequency. Subsequently, based on multimodal data such as the average flash fusion frequency, blink index, pupil response, eye movement behavior, and time distribution curve variation index, it can analyze and output the classification results of visual fatigue degree, including no visual fatigue, mild visual fatigue, moderate visual fatigue, and severe visual fatigue. Finally, if the visual fatigue detection result indicates that the user's eyes are in a state of fatigue, it can also output visual fatigue prompt information corresponding to the visual fatigue detection result to remind the user to manage their eye use.

[0103] In some embodiments, this embodiment may also develop a user-friendly interface or system that displays eye fatigue results in a clear and easy-to-understand manner, providing users with real-time feedback and actionable insights, while ensuring compatibility with various devices and operating systems.

[0104] As can be seen from the above description, the visual fatigue detection method and system provided in this embodiment integrates multiple physiological indicators, providing a comprehensive and accurate assessment of visual fatigue. This is an improvement over current systems that lack such comprehensive data analysis, and is essential for medical diagnosis and ergonomic assessment. In particular, the assessment of nerve fatigue by measuring the critical flicker fusion frequency adds an extra dimension to the analysis. By allowing users to control the measurement of the critical flicker fusion frequency, a more personalized and accurate assessment is provided, improving the overall robustness of the visual fatigue detection system.

[0105] Meanwhile, deep learning algorithms process and analyze the captured high-precision data such as blink indicators, pupillary responses, and eye movement behaviors, significantly improving the reliability and depth of the analysis, providing a more comprehensive understanding of visual fatigue, and thus improving the accuracy of visual fatigue assessment.

[0106] Furthermore, existing technologies often lack real-time feedback, which reduces their effectiveness in preventing eye strain. This embodiment provides users with immediate feedback, enabling timely intervention to manage eye strain.

[0107] In summary, the visual fatigue detection method and system provided in this embodiment can be used in workplace ergonomics to monitor and manage visual fatigue, thereby improving productivity and employee well-being. For example, it can assist in medical diagnosis by helping to diagnose and manage diseases related to visual fatigue and eye strain; it can also provide users with insights into their visual health in personal health monitoring and help manage eye fatigue caused by prolonged screen use; in the gaming and entertainment field, this embodiment can enhance user experience by monitoring and managing visual fatigue in gaming and VR environments; this embodiment can also help students and educators manage eye fatigue caused by prolonged use of digital learning tools, thus providing assistance in educational environments.

[0108] Corresponding to the visual fatigue detection method in the above embodiment, Figure 5 This is a structural block diagram of an eye fatigue detection device provided in one embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0109] Reference Figure 5 The device includes:

[0110] The acquisition module 301 is used to acquire multiple eye image data of the user within a predetermined time period, wherein the user's eyeballs move continuously within the predetermined time period;

[0111] The image analysis module 302 is used to perform image analysis on multiple eye image data to obtain the user's multi-dimensional eye data, which includes at least blink data, pupil response data and eye movement behavior data.

[0112] The visual fatigue detection module 303 is used to detect eye fatigue in users based on multi-dimensional eye data and obtain the visual fatigue detection results of users;

[0113] The visual fatigue warning module 304 is used to output visual fatigue warning information corresponding to the visual fatigue detection result if the visual fatigue detection result indicates that the user's eyes are in a state of fatigue. The visual fatigue warning information is used to remind the user to manage their eye use.

[0114] This embodiment provides an eye fatigue detection device. An acquisition module acquires multiple eye image data of a user within a predetermined time period, during which the user's eyeballs continuously move. An image analysis module performs image analysis on the multiple eye image data to obtain multi-dimensional eye data of the user, including at least blink data, pupillary response data, and eye movement behavior data. An eye fatigue detection module performs eye fatigue detection on the user based on the multi-dimensional eye data to obtain the user's eye fatigue detection result. If the eye fatigue detection result indicates that the user's eyes are in a state of fatigue, an eye fatigue prompt module outputs eye fatigue prompt information corresponding to the eye fatigue detection result, which is used to prompt the user to manage their eye use. Using this device, by performing image analysis on multiple eye image data of the user to obtain multi-dimensional eye data, including at least blink data, pupillary response data, and eye movement behavior data, the data foundation for eye fatigue detection can be enriched, the accuracy of eye fatigue detection results can be improved, and thus the overall reliability of eye fatigue detection can be enhanced.

[0115] Optionally, the image analysis module includes:

[0116] The eyelid analysis unit is used to analyze the eyelid position of multiple eye image data to obtain the user's blink data;

[0117] The pupil analysis unit is used to analyze the pupil position of multiple eye image data to obtain the user's pupil response data;

[0118] The eye-tracking analysis unit is used to perform eye-tracking analysis on multiple eye image data and calculate the user's eye-tracking behavior data.

[0119] Optionally, the eyelid analysis unit is specifically used for:

[0120] An image segmentation algorithm was used to identify eyelids from multiple eye image datasets to obtain the eyelid positions in the multiple eye image datasets;

[0121] Based on the eyelid position in multiple eye image data, calculate the palpebral fissure width of the user in each eye image data, and calculate the variation index corresponding to the palpebral fissure width;

[0122] Based on the eyelid position in multiple eye image data, blinking action is detected in the eye image data to obtain the user's blinking action;

[0123] The duration and interval of a user's blinks are calculated based on the user's blinking action, and the variation indexes corresponding to the duration and interval of blinks are calculated respectively.

[0124] Optionally, the pupil analysis unit is specifically used for:

[0125] Image segmentation algorithms are used to identify pupils in multiple eye image datasets to obtain the pupil positions in the multiple eye image datasets;

[0126] Based on the pupil position in multiple eye image datasets, the pupil size of the user in each eye image dataset is calculated, along with the corresponding variation index for the pupil size.

[0127] Optionally, the eye-tracking analysis unit is specifically used for:

[0128] Deep learning algorithms are used to detect gaze actions on multiple eye image data to obtain the user's gaze actions;

[0129] The number of times a user gazes and the duration of that gaze are calculated based on the user's gaze actions.

[0130] Based on the number of fixations and the duration of fixation, the user's eye movement data is determined. The eye movement data includes at least the average fixation duration and the mean square error of the fixation duration.

[0131] Optionally, the visual fatigue detection module is specifically used for:

[0132] The upper limit critical flash fusion frequency and the lower limit critical flash fusion frequency detected by the flashing device are obtained. The upper limit critical flash fusion frequency is used to characterize the maximum frequency of the continuous light source perceived by the user, and the lower limit critical flash fusion frequency is used to characterize the minimum frequency of the flashing light source perceived by the user.

[0133] The average value of the upper limit critical flash fusion frequency and the lower limit critical flash fusion frequency is determined as the user's average flash fusion frequency;

[0134] Based on multi-dimensional eye data and average flash fusion frequency, eye fatigue is detected in users, and the results of the eye fatigue detection are obtained. The results include one of the following: no eye fatigue, mild eye fatigue, moderate eye fatigue, and severe eye fatigue.

[0135] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0137] This application also provides a terminal device. Figure 6 This is a schematic diagram of the structure of a terminal device provided in one embodiment of this application, as shown below. Figure 6 As shown, the terminal device includes: at least one processor 401, a memory 402, an input device 403, an output device 404, and a computer program stored in the memory 402 and executable on at least one processor 401. When the processor 401 executes the computer program, it implements the steps in any of the above-described method embodiments.

[0138] Input device 403 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the terminal device. Output device 404 may include display devices such as a display screen.

[0139] This application also provides a computer-readable storage medium storing a computer program, which, when executed by processor 401, implements the steps in the above-described method embodiments.

[0140] This application also provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the above-described method embodiments.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 401, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable storage media cannot be electrical carrier signals or telecommunication signals.

[0142] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0143] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0144] In the embodiments provided in this application, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for detecting visual fatigue, characterized in that, include: Acquire multiple eye image data of a user within a predetermined time period, wherein the user's eyeballs move continuously within the predetermined time period; Image analysis is performed on the multiple eye image data to obtain the user's multi-dimensional eye data, which includes at least blink data, pupil response data, and eye movement behavior data; Based on the multi-dimensional eye data, the user's eye fatigue is detected, and the user's visual fatigue detection result is obtained; If the visual fatigue detection result indicates that the user's eyes are in a state of fatigue, then visual fatigue prompt information corresponding to the visual fatigue detection result is output, and the visual fatigue prompt information is used to prompt the user to manage their eye use.

2. The method for detecting visual fatigue as described in claim 1, characterized in that, The step of performing image analysis on the multiple eye image data to obtain the user's multi-dimensional eye data includes: The blinking data of the user is obtained by analyzing the eyelid positions of the multiple eye image data. The pupil position of the user is analyzed from the multiple eye image data to obtain the user's pupil response data; Eye movement analysis is performed on the multiple eye image data to calculate the user's eye movement behavior data.

3. The method for detecting visual fatigue as described in claim 2, characterized in that, The step of analyzing the eyelid positions of the multiple eye image data to obtain the user's blinking data includes: An image segmentation algorithm is used to identify eyelids in multiple eye image data to obtain the eyelid positions in the multiple eye image data; Based on the eyelid position in multiple eye image data, the palpebral fissure width of the user in each eye image data is calculated, and the variation index corresponding to the palpebral fissure width is calculated; Based on the eyelid position in multiple eye image data, blinking action detection is performed on the eye image data to obtain the user's blinking action; The blink duration and blink interval of the user are calculated based on the user's blinking action, and the variation index corresponding to the blink duration and blink interval are calculated respectively.

4. The method for detecting visual fatigue as described in claim 2, characterized in that, The analysis of pupil position from the multiple eye image data to obtain the user's pupil response data includes: An image segmentation algorithm is used to identify pupils in multiple eye image data to obtain the pupil positions in the multiple eye image data; Based on the pupil position in multiple eye image data, the pupil size of the user in each eye image data is calculated, and the variation index corresponding to the pupil size is calculated.

5. The method for detecting visual fatigue as described in claim 2, characterized in that, The step of performing eye-tracking analysis on the multiple eye image data to calculate the user's eye-tracking behavior data includes: A deep learning algorithm is used to detect gaze actions on multiple eye image data to obtain the user's gaze actions; The number of gazes and the duration of gazes of the user are calculated based on the user's gaze actions; Based on the number of fixations and the duration of fixation, the user's eye movement behavior data is determined, and the eye movement behavior data includes at least the average fixation duration and the mean square error of the fixation duration.

6. The method for detecting visual fatigue as described in claim 1, characterized in that, The step of performing eye fatigue detection on the user based on the multi-dimensional eye data to obtain the user's visual fatigue detection result includes: The upper limit critical flash fusion frequency and the lower limit critical flash fusion frequency detected by the flashing device are obtained, wherein the upper limit critical flash fusion frequency is used to characterize the maximum frequency of the continuous light source perceived by the user, and the lower limit critical flash fusion frequency is used to characterize the minimum frequency of the flashing light source perceived by the user. The average value of the upper limit critical flash fusion frequency and the lower limit critical flash fusion frequency is determined as the user's average flash fusion frequency; Based on the multi-dimensional eye data and the average flash fusion frequency, eye fatigue is detected in the user to obtain the user's visual fatigue detection result. The visual fatigue detection result includes one of the following: no visual fatigue, mild visual fatigue, moderate visual fatigue, and severe visual fatigue.

7. A visual fatigue detection device, characterized in that, include: The acquisition module is used to acquire multiple eye image data of a user within a predetermined time period, wherein the user's eyeballs move continuously within the predetermined time period; The image analysis module is used to perform image analysis on the multiple eye image data to obtain the user's multi-dimensional eye data, which includes at least blink data, pupil response data and eye movement behavior data. The visual fatigue detection module is used to perform visual fatigue detection on the user based on the multi-dimensional eye data, and obtain the visual fatigue detection result of the user; The visual fatigue warning module is used to output visual fatigue warning information corresponding to the visual fatigue detection result if the visual fatigue detection result indicates that the user's eyes are in a state of fatigue. The visual fatigue warning information is used to remind the user to manage their eye use.

8. A terminal device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the terminal device to implement the method as described in any one of claims 1-6.

9. A visual fatigue detection system, characterized in that, include: A head-mounted device for collecting multiple eye image data of a user within a predetermined time period, wherein the user's eyeballs move continuously during the predetermined time period; A terminal device is used to perform image analysis on the multiple eye image data to obtain multi-dimensional eye data of the user, wherein the multi-dimensional eye data includes at least blink data, pupil response data, and eye movement behavior data; to perform eye fatigue detection on the user based on the multi-dimensional eye data to obtain the user's visual fatigue detection result; if the visual fatigue detection result indicates that the user's eyes are in a fatigued state, then output visual fatigue prompt information corresponding to the visual fatigue detection result, wherein the visual fatigue prompt information is used to prompt the user to manage eye use.

10. The visual fatigue detection system as described in claim 9, characterized in that, Also includes: A flashing device is used to detect the user's upper limit critical flash fusion frequency and lower limit critical flash fusion frequency; The terminal device is also used to determine the average value of the upper limit critical flash fusion frequency and the lower limit critical flash fusion frequency as the user's average flash fusion frequency; and to perform eye fatigue detection on the user based on the multi-dimensional eye data and the average flash fusion frequency to obtain the user's visual fatigue detection result.

Images