A pupil data fatigue detection method, system and device based on infrared camera

Abstract

The application discloses a pupil data fatigue detection method, system and device based on infrared camera, and relates to the technical field of target detection. Eye infrared image data of a testee is acquired; an image processing algorithm is used to process the eye infrared image data to obtain position data and edge data of a pupil; according to the position data and the edge data of the pupil, a pupil diameter position change data set and a pupil center position change data set in a T time period are obtained; the pupil diameter position change data set and the pupil center position change data set in the T time period are compared with a visual analogue scale of fatigue severity to obtain a pupil data fatigue detection result. The application improves the detection accuracy of the visual fatigue degree of the testee.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method, system and device for fatigue detection of pupil data based on infrared imaging. Background Technology

[0002] By collecting infrared image data of the eye and extracting pupil parameters, various parameters about the pupil can be obtained, such as pupil diameter, pupillary variability, and pupillary adaptation speed. These parameters reflect the physiological state and activity of the eyeball. Simultaneously, by using assessment tools such as the Visual Analogue Scale for Fatigue (VAS-F) and the Eye Complaints Questionnaire (ECQ) to evaluate the subjects, assessment data related to visual fatigue, such as subjective fatigue sensations and eye discomfort, can be obtained.

[0003] When establishing a visual fatigue model in a product, pupil parameters are correlated with visual fatigue assessment data. Statistical analysis and machine learning methods can be used to explore the relationship between pupil parameters and visual fatigue. For example, changes in pupil diameter may be related to the degree of visual fatigue; a larger pupil diameter may indicate a higher level of visual fatigue. Increased pupil variability may be related to the degree of eye fatigue, while a slower pupil adaptation speed may be related to visual fatigue caused by prolonged use of electronic devices. Traditional pupil detection methods suffer from high subjectivity, significant limitations, and low accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, and device for detecting pupil fatigue based on infrared imaging, so as to improve the accuracy of detecting the visual fatigue level of the test subject.

[0005] To achieve the above objectives, embodiments of the present invention provide the following solutions:

[0006] A method for fatigue detection of pupil data based on infrared imaging, comprising:

[0007] Acquire infrared image data of the test subject's eyes;

[0008] The infrared image data of the eye is processed using an image processing algorithm to obtain the position data and edge data of the pupil;

[0009] Based on the pupil position data and edge data, a dataset of pupil diameter position change and a dataset of pupil center position change within time period T are obtained;

[0010] The pupil diameter position change dataset and pupil center position change dataset within the time period T are compared with the fatigue severity visual analog scale to obtain the pupil data fatigue detection results.

[0011] Optionally, the infrared image data of the eye is processed using an image processing algorithm to obtain pupil position data and edge data, specifically including:

[0012] The eye infrared image data is processed using constrained Haar-like features to obtain a rectangular pupil region;

[0013] The rectangular pupil region was located using morphological methods and RANSAC fitting to obtain an elliptical pupil region.

[0014] Based on the elliptical pupil region, the position data and edge data of the pupil are obtained;

[0015] The response value of the Haar-like feature is used to characterize the pupil-iris contrast features;

[0016] The contrast characteristics of the pupil-iris are represented as the difference between the mean values ​​of the outer feature region i1 = (x-2w, y-2w, 3w) and the inner feature region i0 = (x, y, w).

[0017] Where (x, y) represents the coordinates of the upper left corner of the internal feature region, and w represents half the width of the internal feature region; the specific formula is:

[0018]

[0019] in, This represents the sum of pixel intensities within the internal feature region; This represents the sum of pixel intensities in the external feature region; Indicates the area of ​​the internal feature region. This represents the area of ​​the external feature region.

[0020] Optionally, the infrared image data of the eye includes: pupil luminance data, pupil region data, and eye region data.

[0021] Optionally, the global constraint on the pupil luminance data is:

[0022]

[0023] Where PU represents the mean value of the feature intensity of the internal feature region; PIC represents the mean value of the feature intensity of the external feature region.

[0024] The global constraints for the pupil region data and eye region data are:

[0025] The infrared image data of the eye is downsampled to a preset resolution, and the width w of the internal feature region is constrained according to the proportional relationship; the specific formula is as follows:

[0026] stw∈[W / 14,H / 2];

[0027] Where W represents the width of the infrared image of the eye; H represents the height of the infrared image of the eye.

[0028] Optionally, the formula for calculating the constrained Haar-like features is:

[0029]

[0030] Where 'a' is a hyperparameter.

[0031] Optionally, the pupil diameter position change dataset includes: average pupil diameter, pupil fluctuation degree, and pupil adaptation speed;

[0032] The average pupil diameter is used to characterize the average pupil diameter across all frames in the pupil diameter position change data within time period T;

[0033] The pupil variability is used to characterize the standard deviation of pupil diameter changes across all frames in the pupil diameter position change data within time period T.

[0034] Pupil adaptation velocity is used to characterize the average absolute value of the slope of the straight line formed by the extreme points of all frames in the pupil diameter position change data within time interval T; the calculation formula is:

[0035]

[0036] Where N represents the number of extreme points within the time interval T, and S n S represents the pupil diameter in the nth frame. n+1 F represents the pupil diameter in the (n+1)th frame. n F represents the time point of the nth frame. n+1 This represents the time point of the (n+1)th frame.

[0037] Optionally, the pupil center position change dataset includes: eye closure percentage, blink parameters, and saccade parameters;

[0038] Blinking parameters include blink time and blink count; blink time represents the average blink time over a time interval T; blink count represents the total number of blinks over a time interval T.

[0039] The saccade parameters include: saccade duration and saccade speed; saccade duration represents the average saccade duration within time period T; saccade speed represents the maximum saccade speed within time period T.

[0040] To achieve the above objectives, embodiments of the present invention also provide the following solutions:

[0041] A pupil fatigue detection system based on infrared camera data includes:

[0042] The image acquisition module is used to acquire infrared image data of the test subject's eyes;

[0043] A preprocessing module, connected to the image acquisition module, is used to process the infrared image data of the eye using an image processing algorithm to obtain pupil position data and edge data;

[0044] The feature extraction module, connected to the preprocessing module, is used to obtain a dataset of pupil diameter position change and a dataset of pupil center position change within a time period T based on the pupil position data and edge data.

[0045] The output module, connected to the feature extraction module, is used to compare the pupil diameter position change dataset and the pupil center position change dataset within the T time period with the fatigue severity visual analog scale to obtain the pupil data fatigue detection result.

[0046] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned method for fatigue detection of pupil data based on infrared imaging.

[0047] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the aforementioned pupil data fatigue detection method based on infrared imaging.

[0048] In this embodiment of the invention, the image acquisition module employs a non-contact data acquisition method, enabling the capture of image data of the eye region without physical contact. Compared to traditional contact methods, this non-contact acquisition method is more convenient and comfortable, improving the acceptability and participation of the test.

[0049] The preprocessing module uses image processing algorithms to obtain accurate pupil position and edge data; this makes pupil parameter extraction more accurate and reliable, and improves the accuracy of fatigue detection.

[0050] Data sets of pupil diameter position changes and pupil center position changes over time period T were compared with a visual analog scale (VAS) to assess the degree of visual fatigue in test subjects and their responses to related questions. By comprehensively analyzing infrared image data and assessment data, a more comprehensive assessment of visual fatigue status can be achieved. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A schematic flowchart of the pupil data fatigue detection method based on infrared imaging provided in an embodiment of the present invention;

[0053] Figure 2 A detailed structural diagram of the pupil data fatigue detection system based on infrared imaging provided in an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of a simulated office provided for an embodiment of the present invention.

[0055] Symbol explanation:

[0056] Image acquisition module-1, preprocessing module-2, feature extraction module-3, output module-4, light source-5, screen-6, infrared camera-7. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] The purpose of this invention is to provide a method, system, and device for fatigue detection based on pupil data using infrared imaging, for non-contact detection of human fatigue, in order to solve the problem of low accuracy in the detection of visual fatigue levels of test subjects in existing methods.

[0059] An infrared camera-based pupil data fatigue detection system collects pupil data using an infrared camera and extracts pupil-related parameters through data processing and analysis, such as mean pupil diameter, pupil variability, pupil adaptation speed, percentage of eye closure, average eye closure time, number of eye closures, average saccade speed, and peak saccade speed. Based on these parameters, the system can determine the fatigue level of the test subject. This system enables accurate and reliable fatigue monitoring and has broad application prospects, particularly suitable for fields such as transportation, security monitoring, and industrial production.

[0060] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] Figure 1 An exemplary flowchart of the above-described fatigue detection method based on infrared camera pupil data is shown. The steps are described in detail below.

[0062] Step 1: Acquire infrared image data of the subject's eyes;

[0063] The infrared image data of the eye includes: pupil luminance data, pupil area data, and eye area data.

[0064] The global constraint for the pupil luminance data is:

[0065]

[0066] Where PU represents the mean value of the feature intensity of the internal feature region; PIC represents the mean value of the feature intensity of the external feature region.

[0067] The global constraints for the pupil region data and eye region data are:

[0068] The infrared image data of the eye is downsampled to a preset resolution, and the width w of the internal feature region is constrained according to the proportional relationship; the specific formula is as follows:

[0069] stw∈[W / 14,H / 2];

[0070] Where W represents the width of the infrared image of the eye; H represents the height of the infrared image of the eye.

[0071] In one example, the infrared image data of the test subject's eyes can be collected using an infrared camera device. Infrared imaging technology allows for non-contact capture of images of the eye area, and the acquired image data is transmitted to a computer for subsequent image processing and analysis. During data collection, assessment tools such as the Visual Analogue Scale for Fatigue (VAS-F) and the Eye Complaints Questionnaire (ECQ) are used to conduct relevant evaluations and answer questions. These assessment tools can be used to obtain evaluation data related to visual fatigue. The collected visual fatigue data will be used to build a subsequent visual fatigue model.

[0072] In another example, based on classic pupil images, embodiments of the present invention propose:

[0073] Pupil Darkness: As the darkest area in the entire image, the pupil's intensity value is always the lowest in the entire image and is less affected by lighting.

[0074] Pupil area: The pupil area is always larger than the area covered by eyelashes;

[0075] Eye region: The pupil image includes the entire corner of the eye, meaning the width of the image is greater than the width of the palpebral fissure, and the height of the image is greater than the distance between the center points of the upper and lower eyelids.

[0076] To address the issue of eyelashes obscuring the pupil, this embodiment of the invention constrains the pupil and eye regions, primarily limiting the range of w in the (x, y, w) three-dimensional space. The peak diameter of the human pupil is 2mm and 8mm, and the average width of the human eye fissure is 28mm. Furthermore, the image in this embodiment is first downsampled to a resolution of 640×360, and the width w of the internal pixel region can be constrained based on the proportional relationship.

[0077] Since multi-objective optimization models are difficult to solve, this embodiment of the invention uses a linear weighting method to transform the multi-objective optimization model into a single-objective optimization model. Therefore, no additional normalization processing is required.

[0078] Step 2: Process the infrared image data of the eye using an image processing algorithm to obtain the position data and edge data of the pupil; specifically including:

[0079] The eye infrared image data is processed using constrained Haar-like features to obtain a rectangular pupil region (internal pixel region);

[0080] Morphological methods and RANSAC fitting were used to locate the pupil region of the rectangle, resulting in an elliptical pupil region (outer pixel region).

[0081] Based on the elliptical pupil region, the position data and edge data of the pupil are obtained;

[0082] The response value of the Haar-like feature is used to characterize the pupil-iris contrast features;

[0083] The contrast characteristics of the pupil-iris are represented as the difference between the mean values ​​of the outer feature region i1 = (x-2w, y-2w, 3w) and the inner feature region i0 = (x, y, w).

[0084] Where (x, y) represents the coordinates of the upper left corner of the internal feature region, and w represents half the width of the internal feature region; the specific formula is:

[0085]

[0086] in, This represents the sum of pixel intensities within the internal feature region; This represents the sum of pixel intensities in the external feature region; Indicates the area of ​​the internal feature region. This represents the area of ​​the external feature region.

[0087] In one example, the width of the outer feature region is three times that of the inner feature region.

[0088] The formula for calculating the constrained Haar-like features is as follows:

[0089]

[0090] Where 'a' is a hyperparameter.

[0091] In one example, an image processing algorithm is used to process infrared image data of the eye to locate and extract the position and edge of the pupil. This step determines the pupil region and uses it for subsequent parameter extraction. This embodiment of the invention proposes a constrained Haar-like feature to detect the pupil region, in order to cope with noise factors such as changes in external lighting and eyelash obstruction.

[0092] The response value of the Haar-like feature reflects the pupil-iris contrast characteristic, which is expressed as the difference between the mean values ​​of the outer feature region i1 = (x - 2w, y - 2w, 3w) and the inner feature region i0 = (x, y, w), where (x, y) represents the coordinates of the upper left corner of the inner feature region, and w represents half the width of the inner feature region. The pupil-iris contrast characteristic has a larger response value than other contrast characteristics (iris-sclera contrast characteristic, sclera-eyelid contrast characteristic, and iris-eyelid contrast characteristic).

[0093] Haar-like features can locate a square region of width w, where the contrast in feature intensity is highest inside and outside the square region, which may be the pupil area. However, to find elliptical pupils, further processing of the coarse localization results is needed. This can be achieved through morphological methods of the image and RANSAC fitting. These methods can accurately determine the diameter and center position of the pupil.

[0094] In existing technologies, traditional Haar-like models may incorrectly detect areas such as eyelashes, corners of the eyes, and irises due to factors such as lighting, shooting angle, and excessively long eyelashes. This is mainly because variations in light intensity can lead to higher contrast between the corner of the eye and eyelid, and between the iris and sclera, compared to the pupil and iris. Furthermore, differences in eyelash length and grayscale can result in higher contrast between the eyelashes and their surroundings compared to the pupil and iris. Therefore, this invention improves upon Haar-like features by proposing constrained Haar-like features to address the problems caused by these variations in conditions.

[0095] Step 3: Based on the pupil position data and edge data, obtain the pupil diameter position change dataset and pupil center position change dataset within the time period T;

[0096] The pupil diameter position change dataset includes: average pupil diameter, pupil fluctuation degree, and pupil adaptation speed;

[0097] The average pupil diameter is used to characterize the average pupil diameter across all frames in the pupil diameter position change data within time period T;

[0098] The pupil variability is used to characterize the standard deviation of pupil diameter changes across all frames in the pupil diameter position change data within time period T, and is used to represent the intensity of the pupil's response to stimuli.

[0099] Pupil adaptation velocity is used to characterize the average absolute value of the slope of the straight line formed by the extreme points of all frames in the pupil diameter position change data within time interval T; the calculation formula is:

[0100]

[0101] Where N represents the number of extreme points within the time interval T, and S n S represents the pupil diameter in the nth frame. n+1 F represents the pupil diameter in the (n+1)th frame. n F represents the time point of the nth frame. n+1 This represents the time point of the (n+1)th frame.

[0102] In one example, extreme points include both maxima and minima. The slope of the line formed by the extreme points is the slope of the line connecting adjacent pairs of maxima and minima.

[0103] The dataset of pupil center position change includes: percentage of eye closure, blinking parameters, and saccade parameters;

[0104] Blinking parameters include blink time and blink count; blink time represents the average blink time over a time interval T; blink count represents the total number of blinks over a time interval T, used to indicate the frequency of blinking.

[0105] The saccade parameters include: saccade duration and saccade speed; saccade duration represents the average saccade duration within time period T; saccade speed represents the maximum saccade speed within time period T, used to indicate the speed of saccades.

[0106] In one example, blink frames in the pupil center position change dataset are determined using the following two conditions:

[0107] (1) Ellipse aspect ratio (ρ): measures the roundness of the pupil, that is, the ratio between the minor axis and the major axis of the pupil ellipse. This index tends to be a more rounded ellipse. Generally speaking, if the aspect ratio of the ellipse is greater than 3, the fitted ellipse is considered not to be the pupil.

[0108] (2) Ellipse contour contrast (γ): Based on pupil luminance, it is assumed that the region fitted by the ellipse must be the area with the lowest pixel intensity value in the surrounding region. By calculating the linear equation passing through the edge point of the ellipse and the center of the pupil, the average intensity of the inner and outer line segments of the ellipse is compared. If the average intensity of the inner segment is lower than that of the outer segment, then the pupil luminance assumption is satisfied.

[0109] Specifically, the entire circumference is divided into 36 line segments with a step size of 10 degrees. For each line segment, the equation of the straight line from the edge point of the ellipse to the center of the pupil is calculated. A certain length of line segment is taken on both the inner and outer sides, and the average intensity of the line segments on both sides is calculated and compared. If more than half of the line segments satisfy the ellipse darkness assumption, then the fitted ellipse is considered to be the pupil ellipse.

[0110] The expression for the judgment condition (η) is:

[0111] η = ρ & γ;

[0112] Among them, the ellipse aspect ratio (ρ), ellipse contour contrast (γ), and constraint condition (η) are all logical variables. If the judgment condition (η) is 0, it means that the frame is in a blinking state.

[0113] The above eight parameters can objectively describe the changes in pupil response during the experiment, thereby better assessing the visual fatigue state of attention at this time.

[0114] Step 4: Compare the pupil diameter position change dataset and pupil center position change dataset within the time period T with the fatigue severity visual analog scale to obtain the pupil data fatigue detection results.

[0115] In one example, the pupil diameter is the major axis of the pupil ellipse, and a coordinate system is established with the top-left pixel of the eye's infrared image as the origin. Based on the visual fatigue assessment using the eight pupil parameters mentioned above, a visual fatigue classifier can be constructed using Support Vector Machine (SVM), Random Forest, or Artificial Neural Network (ANN). The pupil parameters are used as features in the input layer, and fatigue classification and prediction are performed through multiple intermediate and output layers. The specific method chosen depends on the size of the dataset, the nature of the features, and the requirements for accuracy and efficiency.

[0116] In summary, in this embodiment of the invention, the image acquisition module employs a non-contact data acquisition method, enabling the capture of image data of the eye region without physical contact. Compared to traditional contact methods, this non-contact acquisition method is more convenient and comfortable, improving the acceptability and participation of the test.

[0117] The preprocessing module uses image processing algorithms to obtain accurate pupil position and edge data; this makes pupil parameter extraction more accurate and reliable, and improves the accuracy of fatigue detection.

[0118] Data sets of pupil diameter position changes and pupil center position changes over time period T were compared with a visual analog scale (VAS) to assess the degree of visual fatigue in test subjects and their responses to related questions. By comprehensively analyzing infrared image data and assessment data, a more comprehensive assessment of visual fatigue status can be achieved.

[0119] Example 1:

[0120] To better verify the effectiveness of the pupil data fatigue detection method based on infrared imaging, this embodiment of the invention uses six publicly available datasets to analyze the model results, containing a total of 26,034 images. These datasets were recorded during road driving experiments using a Dikablis head-mounted moving eye tracker. These datasets are highly challenging due to issues such as frequently changing lighting conditions, eyelid and eyelash obstruction of the pupil, motion blur, light reflection, and low contrast between the pupil and iris, as shown in Table 1 below.

[0121] The pupil fatigue detection method based on infrared imaging proposed in this invention differs from existing methods. The Starburst, ExCuSe, and ElSe algorithms were compared across the six datasets mentioned above, and the performance of the infrared camera-based pupil data fatigue detection methods was reported in terms of detection rate with a 5-pixel error. This is shown in Table 2 below:

[0122] Table 1

[0123]

[0124] Table 2

[0125]

[0126] As shown in Table 1, for five of the six datasets, the pupil fatigue detection method based on infrared imaging proposed in this embodiment is significantly superior to other algorithms, even in the worst dataset, Data2. The detection rate was 47.13% on Starburst and ExCuSe datasets. In the remaining five datasets, the pupil fatigue detection method based on infrared imaging proposed in this invention demonstrated excellent detection rates, with an average detection rate greater than 60%. Specifically, on Data4 dataset, the detection rate was 20.53% higher than the second-best ElSe algorithm. This shows that the pupil fatigue detection method based on infrared imaging proposed in this invention performs well even under conditions of low light, mascara, eyeshadow, and other interference.

[0127] Example 2:

[0128] This embodiment uses a simulated office as an example, with the following specific layout: Figure 3 As shown, to stabilize the subject's head position, a neck rest was placed on the table, with a distance of 50cm between the neck rest and the screen to ensure the subject could work at a suitable distance. An infrared camera for collecting pupil data was also placed on the table. The infrared camera used was an Alluvium 1800U, and to ensure the clarity of the pupil images, a 12V6 infrared LED from Vismin was used.

[0129] The pupil data fatigue detection system based on infrared imaging includes the following operations:

[0130] The experiment used an Alluvium 1800U infrared camera to save pupil images for each frame during the experiment. Fatigue data was collected every 10 minutes using the ECQ scale on the screen, and finally a database corresponding to pupil images and visual fatigue was established.

[0131] Pupil localization: In order to reduce the complexity of the pupil data fatigue detection system based on infrared camera, the pupil image is first downsampled to a resolution of 640×360. In order to reduce the influence of factors such as lighting, shooting angle, and excessively long eyelashes, this embodiment of the invention uses constrained Haar-like features to localize the pupil region.

[0132]

[0133] Since the objective function has no analytical solution, a sliding window approach must be used. Although the embodiments of this invention constrain w, the Haar-like feature still searches for the maximum response value in three-dimensional space (feature position (x, y) and feature width w), with a time complexity of O(W×H×H×S), where S is the area of ​​the square region. A square region is used instead of an elliptical region as the Haar-like feature because the square region can be solved using an integral image, eliminating the complexity of convolution and reducing the original time complexity to O(W×H×H).

[0134] The integral image stores the sum of the intensities of all pixels within the rectangle formed by the top-left pixel and the pixel itself, i.e.:

[0135]

[0136] Integral graphs can be calculated incrementally:

[0137] SAT(x+w,y+h)=SAT(x,y+h)+SAT(x+w,y)-SAT(x,y)+I(x+w,y+h);

[0138] Based on the principle of integral graphs, the values ​​of PIC and PU can be calculated:

[0139]

[0140]

[0141] Constrained Haar-like features can only find the approximate area of ​​the pupil. In order to obtain pupil data in a more advanced way, the obtained pupil area needs to be precisely located.

[0142] Precise positioning: In this embodiment of the invention, the diameter and center position of the pupil can be accurately determined by morphological methods and RANSAC fitting of the image. Alternatively, other methods such as ellipse fitting and edge detection can be used for positioning.

[0143] Image binarization mainly involves obtaining the mean value PU. This is achieved by scanning the pixel intensity value of each pixel in the image; if the intensity value is greater than PU, it is set to 255, otherwise it is set to 0.

[0144]

[0145] Image dilation primarily expands the boundary points of binarized objects, filling in image holes caused by occlusion.

[0146]

[0147] Where k is the dilated elliptical structure. When the center of the dilated elliptical structure is translated to a point (x, y) in the BIN image, and the structure intersects with BIN at least at one point, the point (x, y) is retained. Then, the edge pixels of the pupil can be extracted by performing Canny filtering on the dilated image.

[0148] Finally, the RANSAC algorithm fits the contour points of the pupil to obtain the parameters of the ellipse. Specifically:

[0149] Step S1: Randomly select 5 data points from the edge pixels to fit the ellipse equation and calculate the ellipse equation parameters. The general equation of an ellipse is:

[0150] Ax 2 +Bxy+Cy 2 +Dx+Ey+F=0;

[0151] Step S2: Use the obtained ellipse model to evaluate the data points and determine whether they conform to the interior point set of the ellipse model.

[0152] Step S3: Is the number of interior points calculated greater than the number of interior points in the previous step? If so, update the optimal interior point set and the optimal ellipse model.

[0153] Step S4: Determine whether the optimal set of interior points has reached the expected number. If so, exit the loop and output the optimal ellipse model. Otherwise, continue to execute the above operation steps S1 to S3 until the maximum number of loops.

[0154] It is worth noting that during steps S1 and S2, it is necessary to determine whether the parameters of the ellipse being solved are valid, for example, they must satisfy the condition A*C-(B / 2)^2>0. Furthermore, in step S2, thresholds for algebraic and geometric distances can be set to determine whether a data point belongs to an interior point set. Please refer to Table 3:

[0155] Table 3

[0156]

[0157] Participants are required to select appropriate options or assign scores based on their own circumstances, according to the descriptions in the scale. By answering these questions, a comprehensive visual fatigue score is obtained, used to assess the degree of visual fatigue in an individual. The degree of visual fatigue is described on a percentage scale.

[0158] Eight pupil parameters and a fatigue score on a percentage scale were input into the fatigue model for training. The model can employ Support Vector Machine (SVM), Random Forest, or Artificial Neural Network (ANN), among others. This invention uses linear regression, SVM, decision tree, and random forest as examples, with the correlation coefficient R² and mean squared error MSE used to describe the model results, as shown in Table 4.

[0159] Table 4

[0160]

[0161] The results show that the eight pupil parameters extracted in the embodiments of the present invention have a high correlation with visual fatigue, indicating that the pupil parameter model proposed in this invention can effectively determine the degree of visual fatigue.

[0162] Here are some common alternatives:

[0163] For visual fatigue models, various other models such as artificial neural networks can be used as alternatives. A suitable model can be selected based on the correlation coefficient R2 and the mean squared error MSE.

[0164] For the method of extracting parameters, the following is adopted: Other pupil recognition models such as Starburst and ExCuSe do not achieve the same level of accuracy as the method proposed in this invention.

[0165] To achieve the above objectives, embodiments of the present invention also provide the following solutions:

[0166] Please refer to the following: A pupil fatigue detection system based on infrared camera data. Figure 2 ,include:

[0167] Image acquisition module 1 is used to acquire infrared image data of the test subject's eyes;

[0168] Preprocessing module 2 is connected to image acquisition module 1. Preprocessing module 2 is used to process the infrared image data of the eye using image processing algorithms to obtain pupil position data and edge data.

[0169] Feature extraction module 3 is connected to preprocessing module 2. Feature extraction module 3 is used to obtain a dataset of pupil diameter position change and a dataset of pupil center position change within time period T based on the pupil position data and edge data.

[0170] The output module 4 is connected to the feature extraction module 3. The output module 4 is used to compare the pupil diameter position change dataset and the pupil center position change dataset within the T time period with the fatigue severity visual analog scale to obtain the pupil data fatigue detection result.

[0171] Furthermore, the present invention also provides an electronic device, which may include: a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The processor can call a computer program stored in the memory to execute the aforementioned pupil data fatigue detection method based on infrared imaging.

[0172] Furthermore, when the computer program in the aforementioned memory is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0173] Furthermore, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the aforementioned pupil data fatigue detection method based on infrared imaging.

[0174] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0175] This document uses specific examples to illustrate the principles and implementation methods of the embodiments of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of the embodiments of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the embodiments of the present invention. In summary, the content of this specification should not be construed as a limitation on the embodiments of the present invention.

Claims

1. A method for detecting pupil data fatigue based on infrared camera, characterized in that, The method comprises the following steps: acquiring eye infrared image data of a testee; processing the eye infrared image data by using an image processing algorithm to obtain pupil position data and edge data, specifically comprising: processing the eye infrared image data by using a Haar-like feature with a constraint condition to obtain a rectangular pupil region; positioning the rectangular pupil region by using a morphological method and a RANSAC fitting to obtain an elliptical pupil region; obtaining the pupil position data and the edge data according to the elliptical pupil region; a response value of the Haar-like feature is used to represent a pupil-iris contrast feature; the pupil-iris contrast feature is represented as a difference between a mean value of an external feature region i1=(x-2w, y-2w, 3w) and a mean value of an internal feature region i0=(x, y, w); wherein (x, y) represents a coordinate of a top-left corner of the internal feature region, and w represents a half of a width of the internal feature region; a specific formula is as follows: ； wherein, represents the sum of pixel intensities of the internal feature region; represents the sum of pixel intensities of the external feature region; represents the area of the internal feature region, represents the area of the external feature region; the eye infrared image data comprises: pupil darkness data, pupil region data and eye region data; the global constraint of the pupil darkness data is: ； wherein, PU represents a mean value of a feature intensity of the internal feature region, and PIC represents a mean value of a feature intensity of the external feature region; a global constraint of the pupil region data and the eye region data is that the eye infrared image data is down-sampled to a preset resolution, and the internal feature region width w is constrained according to a proportional relationship; a specific formula is as follows: ； wherein, W represents a width of the eye infrared image, and H represents a height of the eye infrared image; obtaining a pupil diameter position change data set and a pupil center position change data set in a T time period according to the pupil position data and the edge data; comparing the pupil diameter position change data set and the pupil center position change data set in the T time period with a visual analogue scale of fatigue severity to obtain a pupil data fatigue detection result. 2.The pupil data fatigue detection method based on infrared camera according to claim 1, characterized in that, The formula for calculating the Haar-like feature with a constraint condition is as follows: ； wherein, a is a hyperparameter. 3.The pupil data fatigue detection method based on infrared camera according to claim 1, wherein, The pupil diameter position change data set comprises: an average pupil diameter, a pupil fluctuation degree and a pupil adaptation speed; the average pupil diameter is used to represent an average value of pupil diameters of all frames in the pupil diameter position change data in the T time period; the pupil fluctuation degree is used to represent a standard deviation of pupil diameter changes of all frames in the pupil diameter position change data in the T time period; the pupil adaptation speed is used to represent an average value of absolute values of slopes of straight lines formed by extreme points in the pupil diameter position change data in the T time period; a calculation formula is as follows: ； Wherein, N represents the number of extreme points in T time period, S n represents the pupil diameter of the n frame, S n+1 represents the pupil diameter of the n+1 frame, F n represents the time point of the n frame, F n+1 represents the time point of the n+1 frame. 4.The pupil data fatigue detection method based on infrared camera according to claim 1, wherein, The pupil center position change data set comprises: a closed-eye percentage, a blink parameter and a saccade parameter; the blink parameter comprises a blink time and a blink frequency; the blink time represents an average value of blink times in the T time period; the blink frequency represents a total sum of blink frequencies in the T time period; the saccade parameter comprises: a saccade duration and a saccade speed; the saccade duration represents an average saccade duration in the T time period; the saccade speed represents a maximum saccade speed in the T time period.

5. An infrared camera-based pupil data fatigue detection system, characterized by, The method comprises the following steps: an image acquisition module is configured to acquire eye infrared image data of a testee; The preprocessing module is connected with the image acquisition module, and is configured to process the eye infrared image data by using an image processing algorithm to obtain position data and edge data of the pupil, specifically including: processing the eye infrared image data by using a Haar-like feature with a constraint condition to obtain a rectangular pupil region; positioning the rectangular pupil region by using a morphological method and a RANSAC fitting to obtain an elliptical pupil region; obtaining the position data and the edge data of the pupil according to the elliptical pupil region; a response value of the Haar-like feature is used to represent a contrast feature of the pupil-iris; the contrast feature of the pupil-iris is represented as a difference between a mean value of an external feature region i1=(x-2w, y-2w, 3w) and a mean value of an internal feature region i0=(x, y, w); wherein (x, y) represents a coordinate of a top-left corner of the internal feature region, and w represents a half of a width of the internal feature region; a specific formula is as follows: ； wherein, represents the sum of pixel intensities of the internal feature region; represents the sum of pixel intensities of the external feature region; represents the area of the internal feature region, represents the area of the external feature region; the eye infrared image data comprises: pupil darkness data, pupil region data and eye region data; the global constraint of the pupil darkness data is: ； wherein, PU represents a mean value of a feature intensity of the internal feature region; PIC represents a mean value of a feature intensity of the external feature region; a global constraint of the pupil region data and the eye region data is that the eye infrared image data is down-sampled to a preset resolution, and the internal feature region width w is constrained according to a proportional relationship; a specific formula is as follows: ； wherein, W represents a width of the eye infrared image; H represents a height of the eye infrared image; The feature extraction module is connected with the preprocessing module, and is configured to obtain a pupil diameter position change data set and a pupil center position change data set in a T time period according to the position data and the edge data of the pupil. The output module is connected with the feature extraction module, and is configured to compare the pupil diameter position change data set and the pupil center position change data set in the T time period with a visual analog scale of fatigue severity to obtain a pupil data fatigue detection result.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the infrared camera-based pupil data fatigue detection method according to any one of claims 1-4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed to implement the infrared camera-based pupil data fatigue detection method according to any one of claims 1-4.

Images