A method and system for detecting eye movement video quality

Abstract

This application proposes a method and system for detecting the quality of eye movement videos. The method detects the face position, face key points, eye key points, and eye position in the eye movement video through a Haar cascade classifier, a histogram of oriented gradients (HOG), or a deep learning model. The method further calculates the head integrity, the proportion of the head occupied, the actual head orientation, and the actual degree of eye openness. The method then scores the basic information of the video, the lighting conditions of the video, the user's recording process, and the user's seriousness, and obtains a comprehensive score of the video quality based on the above scores. This application analyzes the physical characteristics and content of the eye movement video to more accurately and comprehensively evaluate the video quality, effectively solving the problem of uneven eye movement video quality in existing eye tracking methods. By automatically removing eye movement videos of poor quality, the accuracy and stability of the eye tracking results can be ensured, and the efficiency and reliability of data collection can be improved.

Description

Technical Field

[0001] The present application relates to the field of eye tracking technology, and in particular to a method and system for detecting the quality of eye movement videos. Background Art

[0002] In the field of eye tracking, existing video quality assessment methods include focus image / video compression algorithms, methods based on autocorrelation analysis, objective video quality assessment algorithms, "MOSp metric" video quality measurement methods, and subjective video quality assessment methods.

[0003] Focus image / video compression algorithm is a commonly used video quality assessment method. This method evaluates the quality by quantifying the degree of distortion of the image or video. Commonly used indicators include PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index). These indicators only consider the physical characteristics of the image and ignore the accurate simulation of human subjective perception. Therefore, they cannot fully and accurately evaluate the quality of eye movement videos.

[0004] Video quality assessment method based on autocorrelation analysis, which evaluates quality by analyzing the spatial autocorrelation in images or videos, mainly focuses on temporal and spatial distortion, ignoring more cognitive factors such as color information and eye movements that need to be paid attention to in eye tracking.

[0005] Objective video quality assessment is a method based on machine learning and artificial intelligence. It uses a trained model to map subjective scores to predict video quality. However, this method is subject to certain subjectivity and subjective errors, and eye tracking video quality assessment for eye tracking applications has not been deeply studied.

[0006] The "MOSp metric" is a method for measuring eye-tracking video quality based on subjective human evaluation. It uses subjects to subjectively rate eye-tracking videos and then builds a model to convert these subjective ratings into objective quality indicators. However, this method requires significant time and financial investment, and the evaluation results are subject to certain degrees of subjectivity and subjective errors.

[0007] In summary, although existing video quality assessment methods provide some references, they still have the problem of being unable to fully and accurately assess the quality of eye movement videos. The quality of eye movement videos recorded by existing eye tracking methods is unstable, including low video resolution, unstable frame rate, inconsistent exposure and other problems, resulting in the inability to guarantee the accuracy of eye tracking results.

[0008] There are some uncontrollable factors in the video data collection process. Some users failed to accurately align their heads with the camera when recording eye movement videos, resulting in their heads being restricted in the camera's field of view; some users' heads were not fully exposed to the camera's collection range, making eye tracking impossible; in addition, some users' eyes were not opened to a greater extent, further affecting the accuracy of eye tracking; during the eye movement experiment, some users did not complete the task seriously. These factors will affect the subsequent data analysis of the experiment.

[0009] The accuracy of eye tracking is directly affected by video quality. When the quality of facial video is poor, the accuracy of eye tracking results will decrease and cannot meet application requirements.

[0010] To address the above issues, this application proposes a method and system for detecting eye movement video quality. Summary of the Invention

[0011] This application proposes the following technical solutions to address one or more technical deficiencies in the above-mentioned prior art.

[0012] Based on the first aspect of the present application, a method for detecting the quality of an eye movement video is proposed, comprising:

[0013] S1: Extract the frame rate and resolution of the eye movement video, analyze the basic information of the eye movement video and evaluate the basic information of the video. The evaluation formula is:

[0014] Score1=(F×W f +R×W r +S×W s ) / (W f +W r +W s );

[0015] Among them, Score1 represents the evaluation score of the basic information of the video, F, R, and S represent the frame rate score, resolution score, and frame rate stability score respectively, and W f 、W r 、W s Represent the frame rate score weight, resolution score weight, and frame rate stability score weight respectively;

[0016] S2: converting the eye movement video into a grayscale image, performing brightness analysis on the grayscale image to obtain an actual light value, and then performing numerical mapping to obtain an actual brightness value;

[0017] Use Haar cascade detector to detect the face position, divide the frontal face area by locating the key points of the face, and obtain the brightness of the frontal face area and the background brightness through image processing algorithm;

[0018] The lighting conditions are evaluated based on the actual brightness value, the brightness of the frontal facial area, and the background brightness. The calculation formula is:

[0019] Score2=(L×W l +M×W m ) / (W l +W m );

[0020] L=1-|L s -L0| / L0;

[0021] M=MIN(L zm / 2×L bj , 1);

[0022] Among them, Score2 represents the light condition score, L represents the brightness score, L s Indicates the actual brightness value, L0 indicates the standard brightness, L zm Indicates the brightness of the frontal face area, L bj represents background brightness, M represents facial backlight score, W l and W m Represents the brightness score weight and the facial backlight score weight respectively;

[0023] S3: Analyze the head position, eye position, head orientation, head integrity, and eye openness in the eye movement video, and calculate the head position score, head integrity score, head orientation score, and eye openness score respectively to obtain the user recording process score;

[0024] S4: Calculate the user's conscientiousness score based on the time it takes the user to complete the eye movement;

[0025] S5: Calculate the comprehensive score of video quality. If the comprehensive score of video quality is higher than the preset excellent score line, it is judged as a high-quality video. If the comprehensive score of video quality is lower than the preset passing score line, it is judged as a low-quality video and automatically screened out.

[0026] Furthermore, the calculation formula of the frame rate score is:

[0027] F=(F s -F min ) / (F max -F min );

[0028] Among them, F represents the frame rate score, F s Indicates the actual frame rate, F min Indicates the minimum standard frame rate, which is the minimum frame rate threshold that the eye movement analysis system can accept. maxIt represents the highest standard frame rate, which is the frame rate threshold that the eye movement analysis system is expected to achieve perfect;

[0029] The calculation formula of the frame rate stability score is:

[0030] S=(S0-S min ) / (S max -S min );

[0031] Among them, S represents the frame rate stability score, S0 represents the frame rate stability, which is calculated based on the fluctuation of the video frame rate. min Indicates the minimum standard stability, which is the worst stability threshold that the eye movement analysis system can accept. max represents the highest standard of stability, which is the threshold of stability that the eye movement analysis system is expected to achieve perfect;

[0032] The calculation formula of the resolution score is:

[0033] R=(R s -R min ) / (R max -R min );

[0034] Among them, R represents the resolution score, R s Indicates the actual resolution, R min Indicates the lowest standard resolution, which is the lowest resolution threshold acceptable to the eye movement analysis system. max Indicates the highest standard resolution, which is the resolution threshold at which an eye tracking analysis system is expected to be perfect.

[0035] Furthermore, a Haar cascade classifier based on the Viola-Jones framework, a histogram of oriented gradients (HOG), or a deep learning model is used to detect the face position, face key points, eye key points, and eye position in the eye movement video.

[0036] Furthermore, the calculation formulas for the user recording process scores are:

[0037] Score3=(A1×W A1 +A2×W A2 +A3×W A3 +A4×W A4 ) / (W A1 +W A2 +W A3 +W A4 )θ represents;

[0038] A1=1-|head proportion-standard proportion| / standard proportion;

[0039] A2 = actual head area occupied on the screen / entire head area;

[0040] A3=1-|actual head orientation-correct orientation| / correct orientation;

[0041] A4=(actual opening degree - minimum standard opening degree) / (maximum standard opening degree - minimum standard opening degree);

[0042] Among them, Score3 represents the user recording process score, A1, A2, A3 and A4 represent the head position score, head integrity score, head orientation score and eye opening degree score respectively, W A1 、W A2 、W A3 and W A4 They represent the weight of the head position score, the weight of the head integrity score, the weight of the head orientation score, and the weight of the eye opening degree score respectively;

[0043] The proportion occupied by the head is the ratio of the number of pixels in the head area to the total number of pixels in the eye movement video.

[0044] Furthermore, the head integrity is obtained by the following method:

[0045] The Haar cascade algorithm is used to initially screen the face location and verify the face location using the MTCNN model. The detection window size is adjusted based on the video resolution using dynamic anchor frame scaling technology. The largest of the detected face locations is selected as the head location, and the area around the head location is converted into a rectangular bounding box as the head region.

[0046] Extracting the boundary of the head area, fitting the head area actually occupied on the screen using a geometric projection model, and dynamically correcting the area estimate of the head area actually occupied on the screen to obtain a final head area actually occupied on the screen;

[0047] The head completeness is the ratio of the estimated area of ​​the actually occupied head region to the area of ​​the entire head region.

[0048] Based on the preliminary screening of the Haar cascade algorithm, the MTCNN model was introduced for detailed verification to solve the problem of false detection of a single algorithm in complex lighting or occlusion scenes.

[0049] The dynamic anchor frame scaling technology is used to adjust the detection window scale to avoid missed detections caused by fixed windows.

[0050] Dynamically correcting the estimated area of ​​the head area actually occupied on the screen can reduce the error accumulation caused by posture changes in traditional methods and significantly improve measurement stability in complex scenarios.

[0051] Furthermore, the actual head orientation is determined by determining the nose tip position based on the facial key points, calculating the horizontal coordinate difference and the vertical coordinate difference between the nose tip position coordinate and the head area center coordinate, and calculating the head orientation angle. The calculation formula is:

[0052] θ=atan2(Δy,Δx)×180 / π;

[0053] Where θ represents the head orientation angle, Δy is the vertical coordinate difference between the nose tip position coordinate and the head area center coordinate, and Δx is the horizontal coordinate difference between the nose tip position coordinate and the head area center coordinate.

[0054] Furthermore, the positions of key points of the human eye are detected based on the position of the human eye, and the key points of the human eye include the inner corner, outer corner, upper eyelid and lower eyelid of the eye. The distance between the upper eyelid and the lower eyelid is used as the height of the eye, and the distance between the inner corner and the outer corner is used as the opening width of the eye. The actual degree of opening is the ratio of the opening width to the height of the eye.

[0055] Furthermore, the calculation formula for the user's seriousness score is:

[0056] Score4=1-|D s -D0| / D0;

[0057] Among them, Score4 represents the user's seriousness score, D s It represents the actual time it takes for the user to complete the eye movement, and D0 represents the standard completion time, which should be reasonably set according to the difficulty and complexity of the task;

[0058] The calculation formula of the comprehensive video quality score is:

[0059] Score=(Score1×W1+Score2×W2+Score3×W3+Score4×W4) / (W1+W2+W3+W4);

[0060] Among them, Score is the comprehensive score of video quality, Score1 represents the evaluation score of basic video information, Score2 represents the lighting condition score, Score3 represents the user recording process score, Score4 represents the user's conscientiousness score, W1, W2, W3, and W4 represent the weights of the basic video information evaluation score, the lighting condition score, the user recording process score, and the user's conscientiousness score, respectively.

[0061] Based on the second aspect of the present application, a system for detecting the quality of eye movement videos is also proposed, comprising:

[0062] Video basic information evaluation module: extracts the frame rate and resolution of the eye movement video, analyzes the basic information of the eye movement video and evaluates the basic information of the video. The evaluation formula is:

[0063] Score1=(F×W f +R×W r +S×W s ) / (W f +W r +W s );

[0064] Among them, Score1 represents the evaluation score of the basic information of the video, F, R, and S represent the frame rate score, resolution score, and frame rate stability score respectively, and W f 、W r 、W s Represent the frame rate score weight, resolution score weight, and frame rate stability score weight respectively;

[0065] Light condition assessment module: converts the eye movement video into a grayscale image, performs brightness analysis on the grayscale image to obtain an actual light value, and then performs numerical mapping to obtain an actual brightness value;

[0066] Use Haar cascade detector to detect the face position, divide the frontal face area by locating the key points of the face, and obtain the brightness of the frontal face area and the background brightness through image processing algorithm;

[0067] The lighting conditions are evaluated based on the actual brightness value, the brightness of the frontal facial area, and the background brightness. The calculation formula is:

[0068] Score2=(L×W l +M×W m ) / (W l +W m );

[0069] L=1-|L s -L0| / L0;

[0070] M=MIN(L zm / 2×L bj , 1);

[0071] Among them, Score2 represents the light condition score, L represents the brightness score, L s Indicates the actual brightness value, L0 indicates the standard brightness, L zm Indicates the brightness of the frontal face area, L bj represents background brightness, M represents facial backlight score, W l and W m Represents the brightness score weight and the facial backlight score weight respectively;

[0072] Recording process evaluation module: Analyze the head position, eye position, head orientation, head integrity and eye openness in the eye movement video, calculate the head position score, head integrity score, head orientation score and eye openness score respectively, and obtain the user's recording process score;

[0073] User seriousness scoring module: calculates the user's seriousness score based on the time it takes for the user to complete eye movement;

[0074] Comprehensive evaluation module: calculates the comprehensive score of video quality. If the comprehensive score of video quality is higher than the preset excellent score line, it is judged as a high-quality video. If the comprehensive score of video quality is lower than the preset passing score line, it is judged as a low-quality video and automatically screened out.

[0075] This application saves a lot of human resources, improves the efficiency and reliability of data collection, and effectively solves the problem of uneven eye movement video quality in existing eye tracking methods. By automatically removing eye movement videos of poor quality, the accuracy and stability of eye tracking results are ensured.

[0076] Based on the third aspect of the present application, a computer program product is further proposed, which has one or more computer programs thereon, and when the computer program product is executed by a computer processor, implements any one of the methods described above.

[0077] The technical effect of this application is that: by analyzing the physical characteristics and content of eye movement videos, this application can more accurately and comprehensively evaluate the video quality, effectively solving the problem of uneven eye movement video quality in existing eye tracking methods. By automatically removing eye movement videos of poor quality, the accuracy and stability of eye tracking results can be ensured, and the efficiency and reliability of data collection can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Other features, objects and advantages of the present application will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings.

[0079] Figure 1 is an overall flow chart of a method for detecting eye movement video quality provided according to an embodiment of the present application;

[0080] Figure 2 1 is a general framework diagram of an eye movement video quality detection system provided according to an embodiment of the present application;

[0081] Figure 3 It is a structural diagram of a computer system suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0082] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0083] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0084] Reference below Figure 1 , Figure 1 A method for detecting eye movement video quality is shown, comprising:

[0085] S1: Extract the frame rate and resolution of the eye movement video, analyze the basic information of the eye movement video and evaluate the basic information of the video. The evaluation formula is:

[0086] Score1=(F×W f +R×W r +S×W s ) / (W f +W r +W s );

[0087] Among them, Score1 represents the evaluation score of the basic information of the video, F, R, and S represent the frame rate score, resolution score, and frame rate stability score respectively, and W f 、W r 、W s Represent the frame rate score weight, resolution score weight, and frame rate stability score weight respectively;

[0088] S2: converting the eye movement video into a grayscale image, performing brightness analysis on the grayscale image to obtain an actual light value, and then performing numerical mapping to obtain an actual brightness value;

[0089] Use Haar cascade detector to detect the face position, divide the frontal face area by locating the key points of the face, and obtain the brightness of the frontal face area and the background brightness through image processing algorithm;

[0090] The lighting conditions are evaluated based on the actual brightness value, the brightness of the frontal facial area, and the background brightness. The calculation formula is:

[0091] Score2=(L×W l +M×W m ) / (W l +W m );

[0092] L=1-|Ls -L0| / L0;

[0093] M=MIN(L zm / 2×L bj , 1);

[0094] Among them, Score2 represents the light condition score, L represents the brightness score, L s Indicates the actual brightness value, L0 indicates the standard brightness, L zm Indicates the brightness of the frontal face area, L bj represents background brightness, M represents facial backlight score, W l and W m Represents the brightness score weight and the facial backlight score weight respectively;

[0095] S3: Analyze the head position, eye position, head orientation, head integrity, and eye openness in the eye movement video, and calculate the head position score, head integrity score, head orientation score, and eye openness score respectively to obtain the user recording process score;

[0096] S4: Calculate the user's conscientiousness score based on the time it takes the user to complete the eye movement;

[0097] S5: Calculate the comprehensive score of video quality. If the comprehensive score of video quality is higher than the preset excellent score line, it is judged as a high-quality video. If the comprehensive score of video quality is lower than the preset passing score line, it is judged as a low-quality video and automatically screened out.

[0098] It should be noted that the calculation formula for the frame rate score is:

[0099] F=(F s -F min ) / (F max -F min );

[0100] Among them, F represents the frame rate score, F s Indicates the actual frame rate, F min Indicates the minimum standard frame rate, which is the minimum frame rate threshold that the eye movement analysis system can accept. max It represents the highest standard frame rate, which is the frame rate threshold that the eye movement analysis system is expected to achieve perfect;

[0101] The calculation formula of the frame rate stability score is:

[0102] S=(S0-S min ) / (S max -S min );

[0103] Among them, S represents the frame rate stability score, S0 represents the frame rate stability, which is calculated based on the fluctuation of the video frame rate. min Indicates the minimum standard stability, which is the worst stability threshold that the eye movement analysis system can accept. max represents the highest standard of stability, which is the threshold of stability that the eye movement analysis system is expected to achieve perfect;

[0104] The calculation formula of the resolution score is:

[0105] R=(R s -R min ) / (R max -R min );

[0106] Among them, R represents the resolution score, R s Indicates the actual resolution, R min Indicates the lowest standard resolution, which is the lowest resolution threshold acceptable to the eye movement analysis system. max Indicates the highest standard resolution, which is the resolution threshold at which an eye tracking analysis system is expected to be perfect.

[0107] It should be noted that the face position, face key points, eye key points and eye position in the eye movement video are detected using a Haar cascade classifier based on the Viola-Jones framework, a histogram of oriented gradients (HOG) or a deep learning model.

[0108] It should be noted that the calculation formulas for the user recording process scores are:

[0109] Score3=(A1×W A1 +A2×W A2 +A3×W A3 +A4×W A4 ) / (W A1 +W A2 +W A3 +W A4 );

[0110] A1=1-|head proportion-standard proportion| / standard proportion;

[0111] A2 = actual head area occupied on the screen / entire head area;

[0112] A3=1-|actual head orientation-correct orientation| / correct orientation;

[0113] A4=(actual opening degree - minimum standard opening degree) / (maximum standard opening degree - minimum standard opening degree);

[0114] Among them, Score3 represents the user recording process score, A1, A2, A3 and A4 represent the head position score, head integrity score, head orientation score and eye opening degree score respectively, W A1 、W A2 、W A3 and W A4 They represent the weight of the head position score, the weight of the head integrity score, the weight of the head orientation score, and the weight of the eye opening degree score respectively;

[0115] The proportion occupied by the head is the ratio of the number of pixels in the head area to the total number of pixels in the eye movement video.

[0116] It should be noted that the header integrity is obtained by the following method:

[0117] The Haar cascade algorithm is used to initially screen the face location and verify the face location using the MTCNN model. The detection window size is adjusted based on the video resolution using dynamic anchor frame scaling technology. The largest of the detected face locations is selected as the head location, and the area around the head location is converted into a rectangular bounding box as the head region.

[0118] Extracting the boundary of the head area, fitting the head area actually occupied on the screen using a geometric projection model, and dynamically correcting the area estimate of the head area actually occupied on the screen to obtain a final head area actually occupied on the screen;

[0119] The head completeness is the ratio of the estimated area of ​​the actually occupied head region to the area of ​​the entire head region.

[0120] It should be noted that the MTCNN model was introduced for detailed verification based on the preliminary screening using the Haar cascade algorithm, which solved the problem of false detection of a single algorithm in complex lighting or occlusion scenes. The dynamic anchor frame scaling technology was used to adjust the detection window scale to avoid missed detections caused by fixed windows. The dynamic correction of the area estimate of the head area actually occupied on the screen can reduce the error accumulation caused by posture changes in traditional methods, and significantly improve the measurement stability in complex scenes.

[0121] It should be noted that the actual head orientation is to determine the nose tip position based on the facial key points, calculate the horizontal and vertical coordinate differences between the nose tip position coordinates and the head area center coordinates, and calculate the head orientation angle. The calculation formula is:

[0122] θ=atan2(Δy,Δx)×180 / π;

[0123] Where θ represents the head orientation angle, Δy is the vertical coordinate difference between the nose tip position coordinate and the head area center coordinate, and Δx is the horizontal coordinate difference between the nose tip position coordinate and the head area center coordinate.

[0124] It should be noted that the positions of the key points of the human eye are detected based on the position of the human eye. The key points of the human eye include the inner corner, outer corner, upper eyelid and lower eyelid of the eye. The distance between the upper eyelid and the lower eyelid is used as the height of the eye, and the distance between the inner corner and the outer corner is used as the opening width of the eye. The actual degree of opening of the eye is the ratio of the opening width to the eye height.

[0125] It should be noted that the calculation formula for the user's seriousness score is:

[0126] Score4=1-|D s -D0| / D0;

[0127] Among them, Score4 represents the user's seriousness score, D s It represents the actual time it takes for the user to complete the eye movement, and D0 represents the standard completion time, which is reasonably set according to the difficulty and complexity of the task;

[0128] The calculation formula of the comprehensive video quality score is:

[0129] Score=(Score1×W1+Score2×W2+Score3×W3+Score4×W4) / (W1+W2+W3+W4);

[0130] Among them, Score is the comprehensive score of video quality, Score1 represents the evaluation score of basic video information, Score2 represents the lighting condition score, Score3 represents the user recording process score, Score4 represents the user's conscientiousness score, W1, W2, W3, and W4 represent the weights of the basic video information evaluation score, the lighting condition score, the user recording process score, and the user's conscientiousness score, respectively.

[0131] In a specific embodiment, a three-stage cascade network is used to train the MTCNN deep learning model, which integrates a classification loss function (for distinguishing faces from non-faces), a bounding box loss function (for regressing the position and size of the face bounding box), and a key point localization loss function (for locating facial key points) to achieve face detection.

[0132] The total loss function formula for the MTCNN deep learning model training is:

[0133] ,

[0134] in, is the classification loss function, is the bounding box regression loss, is the keypoint localization loss, and Weight coefficients used to balance different tasks;

[0135] ,

[0136] in, is the true label ( Represents a face, indicates non-human face), is the face probability predicted by the model;

[0137] ,

[0138] in, is the sample size, are the predicted bounding box parameters, is the true bounding box parameter, and the SmoothL1 loss is defined as:

[0139] ,

[0140] in, is the number of key points, are the predicted keypoint coordinates, are the true keypoint coordinates.

[0141] In a specific embodiment, HOG directional gradient histogram is used to detect faces. The face gradient directional histogram of the local image is extracted to calculate the gradient amplitude and direction of each pixel in the image. When extracting image features, the image is divided into several cell units and the histogram of the gradient direction in each interval is counted. Finally, a support vector machine classifier is used to distinguish between faces and non-faces. A negative sample mining optimization model is introduced to perform face detection.

[0142] The formula for calculating the gradient amplitude of each pixel of the image is:

[0143] ,

[0144] The formula for calculating the gradient direction of each pixel in the image is:

[0145] ,

[0146] in, and They are the gradients of the image in the x and y directions, which can be calculated by methods such as the Sobel operator;

[0147] The calculation formula for the histogram of the gradient direction in each interval is:

[0148] ,

[0149] Among them, p represents a single pixel in the image, cell represents a local area unit, It is The histogram values ​​of the direction intervals, It is Direction range, is the indicator function.

[0150] In a specific embodiment, a Haar cascade classifier based on the Viola-Jones framework is trained to extract a large number of Haar-like rectangular features, and the pixel sum of each region is calculated using an integral image. Finally, an AdaBoost algorithm is used to iteratively select key features, and the weight of each weak classifier is corrected according to the error rate. Multiple weak classifiers are cascaded into a strong classifier to perform face detection.

[0151] The calculation formula of the Haar-like rectangle feature is:

[0152] ,

[0153] Among them, Haar Feature represents Haar-like rectangular feature;

[0154] The calculation formula of the integral graph is:

[0155] ,

[0156] ,

[0157] Integral(x, y) represents the cumulative sum of the grayscale values ​​of all pixels in the rectangular area from the image origin (0,0) to the current point (x, y); Sum represents the sum of the grayscale values ​​of pixels in any rectangular area calculated by the integral image;

[0158] The weight update formula of the weak classifier is:

[0159] ,

[0160] ,

[0161] in, Represents pixel points The gray value of Indicates that it is located The grayscale value of the pixel, It is The error rate of weak classifiers, is the weight of the classifier, It is Weak classifiers, is the total number of weak classifiers, and H(x) represents a strong classifier composed of a weighted combination of multiple weak classifiers.

[0162] In a specific embodiment, a Haar cascade classifier is used to extract local texture differences in the human eye region, and a rectangular Haar template is designed based on the eye features of the collected positive sample image of the human eye and the background negative sample image. For example, the upper and lower eyelids are captured based on the horizontal edge features, and the corners of the eyes are captured based on the vertical features.

[0163] The pixel difference of the human eye area is calculated using the integral image, and the key eye Haar features of the human eye and the background are screened using the AdaBoast algorithm to construct a Haar cascade classifier. The Haar cascade classifier excludes non-eye areas based on preliminary eye features, uses key eye Haar features to determine the pupil position, and then locates the coordinates of the human eye position through a sliding window.

[0164] In a specific embodiment, the human eye area is identified by the gradient direction distribution of the eye area, the human eye area image is normalized to a fixed size of 32x32 pixels, and the gradient amplitude and direction of each pixel in the human eye area image are calculated, the human eye area image is divided into 8x8 cells, the gradient histogram of 9 directions of each cell in the human eye area image is counted, L2-Hys normalization is performed on 2x2 cells as blocks, HOG feature vectors are generated, and input into a linear vector machine, classification training is performed on human eye positive sample images and background negative sample images, and then overlapping detection frames are eliminated through non-maximum suppression, and the sliding window step size and scale are adjusted to determine the human eye position.

[0165] In a specific embodiment, binocular bounding boxes and category labels are annotated in an eye region image, a multi-scale feature map of the eye region image is extracted using a DarkNet-53 backbone network, an anchor frame is preset for the feature map, the anchor frame size is determined according to the size of the eye, a multi-task loss function, a CloU Loss loss function, and a Focal Loss loss function are used to train a deep learning model, the deep learning model detects the final eye bounding box according to a set confidence level, performs threshold filtering and non-maximum suppression post-processing to determine the eye position, and outputs the eye position coordinates;

[0166] The calculation formula of the CloU function is as follows:

[0167] ,

[0168] Among them, CIoU Loss represents the CloU loss function, which is a function used to measure the difference between the model prediction value and the true value. IoU is the intersection-over-union ratio of the predicted bounding box and the true bounding box. is the square of the Euclidean distance between the center point of the predicted bounding box and the true bounding box, is the square of the diagonal length of the smallest rectangle enclosing the two bounding boxes, It is a measure of the consistency between the aspect ratio of the predicted bounding box and the true bounding box;

[0169] ,

[0170] in, and h gt are the width and height of the ground-truth bounding box, and h pred is the width and height of the predicted bounding box, is a coefficient used to balance the effects of center point distance and aspect ratio;

[0171] The calculation formula of the Focal Loss function is:

[0172] ,

[0173] Among them, L focal (p,y) represents the weighted cross entropy loss value for a single sample, is the true label ( ),when When represents a positive sample, It represents a negative sample, is the probability of positive samples predicted by the model Is the balance coefficient, which is used to adjust the weight of positive and negative samples, and is usually set to 0.25. It is an adjustment factor used to control the focal strength of Focal Loss, and its value is usually 2.

[0174] It should be noted that this application analyzes the physical characteristics and content of eye movement videos to conduct a more accurate and comprehensive assessment of video quality, effectively solving the problem of uneven eye movement video quality in existing eye tracking methods. By automatically removing eye movement videos of poor quality, the accuracy and stability of eye tracking results can be ensured, and the efficiency and reliability of data collection can be improved.

[0175] Reference below Figure 2 , Figure 2 The present invention shows a system for detecting the quality of eye movement videos, which includes a basic video information evaluation module a, a lighting condition evaluation module b, a recording process evaluation module c, a user seriousness scoring module d, and a comprehensive evaluation module e.

[0176] In a specific embodiment, the video basic information evaluation module a is configured to: extract the frame rate and resolution of the eye movement video, analyze the basic information of the eye movement video to perform video basic information evaluation, and the evaluation formula is:

[0177] Score1=(F×W f +R×W r +S×W s ) / (W f +W r +W s );

[0178] Among them, Score1 represents the evaluation score of the basic information of the video, F, R, and S represent the frame rate score, resolution score, and frame rate stability score respectively, and W f 、W r 、W s They represent the frame rate score weight, resolution score weight, and frame rate stability score weight respectively.

[0179] In a specific embodiment, the light condition assessment module b is configured to: convert the eye movement video into a grayscale image, perform brightness analysis on the grayscale image to obtain an actual light value, and then perform numerical mapping to obtain an actual brightness value;

[0180] Use Haar cascade detector to detect the face position, divide the frontal face area by locating the key points of the face, and obtain the brightness of the frontal face area and the background brightness through image processing algorithm;

[0181] The lighting conditions are evaluated based on the actual brightness value, the brightness of the frontal facial area, and the background brightness. The calculation formula is:

[0182] Score2=(L×W l +M×W m ) / (W l +W m );

[0183] L=1-|L s -L0| / L0;

[0184] M=MIN(L zm / 2×L bj , 1);

[0185] Among them, Score2 represents the light condition score, L represents the brightness score, L s Indicates the actual brightness value, L0 indicates the standard brightness, L zm Indicates the brightness of the frontal face area, L bj represents background brightness, M represents facial backlight score, W l and Wm Represent the brightness score weight and the facial backlight score weight respectively.

[0186] In a specific embodiment, the recording process evaluation module c is configured to: analyze the head position, eye position, head orientation, head integrity and eye openness in the eye movement video, and calculate the head position score, head integrity score, head orientation score and eye openness score respectively to obtain the user recording process score.

[0187] In a specific embodiment, the user seriousness scoring module d is configured to calculate the user's seriousness score based on the time it takes for the user to complete the eye movement.

[0188] In a specific embodiment, the comprehensive evaluation module e is configured to: calculate the comprehensive score of video quality; if the comprehensive score of video quality is higher than the preset excellent score line, it is judged as a high-quality video; if the comprehensive score of video quality is lower than the preset passing score line, it is judged as a low-quality video and the video is automatically screened out.

[0189] It should be noted that compared with the existing technical methods that only focus on the physical characteristics of the video, the eye movement video quality detection method of this application provides a more comprehensive evaluation. In addition to considering basic information such as video resolution and frame rate, it also focuses on the evaluation factors within the video content, such as lighting conditions, face position and orientation, etc., as well as the capture of human eye information. This application also takes into account the user's seriousness in completing the task. By comprehensively evaluating these factors, the method of this application can provide a more accurate, comprehensive and objective eye movement video quality evaluation, thereby improving the accuracy and reliability of eye tracking results.

[0190] It should be noted that by adopting computer vision and image processing technology, eye movement videos of poor quality can be detected quickly and accurately and automatically eliminated; compared with traditional manual screening methods, the automated operation of this application saves a lot of human resources, improves the efficiency and reliability of data collection, and effectively solves the problem of uneven eye movement video quality in existing eye tracking methods. By automatically removing eye movement videos of poor quality, the accuracy and stability of eye tracking results are ensured, which is very important for applications that rely on eye tracking data, such as user experience research, medical diagnosis, etc. This application can provide high-quality eye tracking data.

[0191] Reference below Figure 3 , which shows a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. Figure 3 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0192] like Figure 3 As shown, the computer system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 302 or programs loaded from a storage unit 308 into a random access memory (RAM) 303. Various programs and data required for system operation are also stored in RAM 303. CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to bus 304.

[0193] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, mouse, and the like; an output section 307 including devices such as a liquid crystal display (LCD) and speakers; a storage section 308 including devices such as a hard disk; and a communication section 309 including a network interface card such as a LAN card or a modem. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. Removable media 311, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is installed in the drive 310 as needed, so that computer programs read from the media can be installed in the storage section 308 as needed.

[0194] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for executing the method illustrated in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 309 and / or installed from removable media 311. When executed by the central processing unit (CPU) 301, the computer program performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take various forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0195] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0196] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0197] The modules described in the embodiments of the present application may be implemented by software or hardware.

[0198] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed by the electronic device, the electronic device: extracts the frame rate and resolution of the eye movement video, analyzes the basic information of the eye movement video and performs basic information evaluation of the video, converts the eye movement video into a grayscale image, performs brightness analysis on the grayscale image to obtain the actual light value and then performs numerical mapping to obtain the actual brightness value; uses a Haar cascade detector to detect the position of the face, divides the frontal facial area by locating the key points of the face, and obtains the brightness of the frontal facial area and the background brightness through an image processing algorithm; according to the actual brightness value, the frontal facial area The brightness and background brightness are used to evaluate the lighting conditions, and the head position, eye position, head orientation, head integrity and eye openness in the eye movement video are analyzed. The head position score, head integrity score, head orientation score and eye openness score are calculated respectively, and the user recording process score is further calculated. The user's conscientiousness score is calculated based on the time it takes for the user to complete the eye movement. The comprehensive score of the video quality is calculated. If the comprehensive score of the video quality is higher than the preset excellent score line, the video is judged to be a high-quality video. If the comprehensive score of the video quality is lower than the preset passing score line, the video is judged to be a low-quality video and is automatically screened out.

[0199] Finally, it should be noted that the above description is merely a preferred embodiment of this application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention herein is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for detecting eye movement video quality, characterized in that: include: S1: Extract the frame rate and resolution of the eye movement video, analyze the basic information of the eye movement video and evaluate the basic information of the video. The evaluation formula is: ； Among them, Score1 represents the evaluation score of the basic information of the video, F, R, and S represent the frame rate score, resolution score, and frame rate stability score respectively, and W f 、W r 、W s Represent the frame rate score weight, resolution score weight, and frame rate stability score weight respectively; S2: converting the eye movement video into a grayscale image, performing brightness analysis on the grayscale image to obtain an actual light value, and then performing numerical mapping to obtain an actual brightness value; Use Haar cascade detector to detect the face position, divide the frontal face area by locating the key points of the face, and obtain the brightness of the frontal face area and the background brightness through image processing algorithm; The lighting conditions are evaluated based on the actual brightness value, the brightness of the frontal facial area, and the background brightness. The calculation formula is: ； ； ； Among them, Score2 represents the light condition score, L represents the brightness score, L s Indicates the actual brightness value, L0 indicates the standard brightness, L zm Indicates the brightness of the frontal face area, L bj represents background brightness, M represents facial backlight score, W l and W m Represents the brightness score weight and the facial backlight score weight respectively; S3: Analyze the head position, eye position, head orientation, head integrity, and eye openness in the eye movement video, and calculate the head position score, head integrity score, head orientation score, and eye openness score respectively to obtain the user recording process score; The calculation formulas for the user recording process scores are: ； A1=1-|head proportion-standard proportion| / standard proportion; A2 = actual head area occupied on the screen / entire head area; A3=1-|actual head orientation-correct orientation| / correct orientation; A4=(actual opening degree - minimum standard opening degree) / (maximum standard opening degree - minimum standard opening degree); Among them, Score3 represents the user recording process score, A1, A2, A3 and A4 represent the head position score, head integrity score, head orientation score and eye opening degree score respectively, W A1 、W A2 、W A3 and W A4 They represent the weight of the head position score, the weight of the head integrity score, the weight of the head orientation score, and the weight of the eye opening degree score respectively; The proportion occupied by the head is the ratio of the number of pixels in the head area to the total number of pixels in the eye movement video; The actual head orientation is to determine the nose tip position based on the facial key points, calculate the horizontal coordinate difference and the vertical coordinate difference between the nose tip position coordinate and the head area center coordinate, and calculate the head orientation angle. The calculation formula is: ； Where θ represents the head orientation angle, Δy is the vertical coordinate difference between the nose tip position coordinate and the head area center coordinate, and Δx is the horizontal coordinate difference between the nose tip position coordinate and the head area center coordinate; S4: Calculate the user's conscientiousness score based on the time it takes the user to complete the eye movement; S5: Calculate the comprehensive score of video quality. If the comprehensive score of video quality is higher than the preset excellent score line, it is judged as a high-quality video. If the comprehensive score of video quality is lower than the preset passing score line, it is judged as a low-quality video and automatically screened out.

2. The method according to claim 1, characterized in that The calculation formula of the frame rate score is: ； Among them, F represents the frame rate score, F s Indicates the actual frame rate, F min Indicates the minimum standard frame rate, which is the minimum frame rate threshold that the eye movement analysis system can accept. max It represents the highest standard frame rate, which is the frame rate threshold that the eye movement analysis system is expected to achieve perfect; The calculation formula of the frame rate stability score is: ； Among them, S represents the frame rate stability score, S0 represents the frame rate stability, which is calculated based on the fluctuation of the video frame rate. min Indicates the minimum standard stability, which is the worst stability threshold that the eye movement analysis system can accept. max represents the highest standard of stability, which is the threshold of stability that the eye movement analysis system is expected to achieve perfect; The calculation formula of the resolution score is: ； Among them, R represents the resolution score, R s Indicates the actual resolution, R min Indicates the lowest standard resolution, which is the lowest resolution threshold acceptable to the eye movement analysis system. max Indicates the highest standard resolution, which is the resolution threshold at which an eye tracking analysis system is expected to be perfect.

3. The method according to claim 1, characterized in that A Haar cascade classifier based on the Viola-Jones framework, a histogram of oriented gradients (HOG), or a deep learning model is used to detect the face position, face key points, eye key points, and eye position in the eye movement video.

4. The method according to claim 1, wherein The header integrity is obtained by the following method: The Haar cascade algorithm is used to initially screen the face location and verify the face location using the MTCNN model. The detection window size is adjusted based on the video resolution using dynamic anchor frame scaling technology. The largest of the detected face locations is selected as the head location, and the area around the head location is converted into a rectangular bounding box as the head region. Extracting the boundary of the head area, fitting the head area actually occupied on the screen using a geometric projection model, and dynamically correcting the area estimate of the head area actually occupied on the screen to obtain a final head area actually occupied on the screen; The head completeness is the ratio of the estimated area of ​​the actually occupied head region to the area of ​​the entire head region.

5. The method according to claim 1, wherein The positions of key points of the human eye are detected according to the position of the human eye, where the key points of the human eye include the inner corner, outer corner, upper eyelid and lower eyelid of the eye. The distance between the upper eyelid and the lower eyelid is used as the height of the eye, and the distance between the inner corner and the outer corner is used as the opening width of the eye. The actual degree of opening is the ratio of the opening width to the eye height.

6. The method according to claim 1, characterized in that The calculation formula for the user's seriousness score is: ； Among them, Score4 represents the user's seriousness score, D s It represents the actual time it takes for the user to complete the eye movement, and D0 represents the standard completion time, which should be reasonably set according to the difficulty and complexity of the task; The calculation formula of the comprehensive video quality score is: ； Among them, Score is the comprehensive score of video quality, Score1 represents the evaluation score of basic video information, Score2 represents the lighting condition score, Score3 represents the user recording process score, Score4 represents the user's conscientiousness score, W1, W2, W3, and W4 represent the weights of the basic video information evaluation score, the lighting condition score, the user recording process score, and the user's conscientiousness score, respectively.

7. A system for detecting the quality of eye movement videos, characterized in that: include: Video basic information evaluation module: extracts the frame rate and resolution of the eye movement video, analyzes the basic information of the eye movement video and evaluates the basic information of the video. The evaluation formula is: ； Among them, Score1 represents the evaluation score of the basic information of the video, F, R, and S represent the frame rate score, resolution score, and frame rate stability score respectively, and W f 、W r 、W s Represent the frame rate score weight, resolution score weight, and frame rate stability score weight respectively; Light condition assessment module: converts the eye movement video into a grayscale image, performs brightness analysis on the grayscale image to obtain an actual light value, and then performs numerical mapping to obtain an actual brightness value; Use Haar cascade detector to detect the face position, divide the frontal face area by locating the key points of the face, and obtain the brightness of the frontal face area and the background brightness through image processing algorithm; The lighting conditions are evaluated based on the actual brightness value, the brightness of the frontal facial area, and the background brightness. The calculation formula is: ； ； ； Among them, Score2 represents the light condition score, L represents the brightness score, L s Indicates the actual brightness value, L0 indicates the standard brightness, L zm Indicates the brightness of the frontal face area, L bj represents background brightness, M represents facial backlight score, W l and W m Represents the brightness score weight and the facial backlight score weight respectively; Recording process evaluation module: Analyze the head position, eye position, head orientation, head integrity and eye openness in the eye movement video, calculate the head position score, head integrity score, head orientation score and eye openness score respectively, and obtain the user's recording process score; The calculation formulas for the user recording process scores are: ； A1=1-|head proportion-standard proportion| / standard proportion; A2 = actual head area occupied on the screen / entire head area; A3=1-|actual head orientation-correct orientation| / correct orientation; A4=(actual opening degree - minimum standard opening degree) / (maximum standard opening degree - minimum standard opening degree); Among them, Score3 represents the user recording process score, A1, A2, A3 and A4 represent the head position score, head integrity score, head orientation score and eye opening degree score respectively, W A1 、W A2 、W A3 and W A4 They represent the weight of the head position score, the weight of the head integrity score, the weight of the head orientation score, and the weight of the eye opening degree score respectively; The proportion occupied by the head is the ratio of the number of pixels in the head area to the total number of pixels in the eye movement video; The actual head orientation is to determine the nose tip position based on the facial key points, calculate the horizontal coordinate difference and the vertical coordinate difference between the nose tip position coordinate and the head area center coordinate, and calculate the head orientation angle. The calculation formula is: ； Where θ represents the head orientation angle, Δy is the vertical coordinate difference between the nose tip position coordinate and the head area center coordinate, and Δx is the horizontal coordinate difference between the nose tip position coordinate and the head area center coordinate; User seriousness scoring module: calculates the user's seriousness score based on the time it takes for the user to complete eye movement; Comprehensive evaluation module: calculates the comprehensive score of video quality. If the comprehensive score of video quality is higher than the preset excellent score line, it is judged as a high-quality video. If the comprehensive score of video quality is lower than the preset passing score line, it is judged as a low-quality video and automatically screened out.

8. A computer program product having one or more computer programs thereon, characterized in that: When the computer program product is executed by a computer processor, the method according to any one of claims 1 to 6 is implemented.

Images