Equipment operator fatigue state detection method based on visual feature information fusion

Abstract

The present invention discloses a method for detecting the fatigue state of equipment operators based on the fusion of visual feature information. The method selects the eyes, mouth, and head contours from various fatigue representations as a detection combination, and makes a fatigue judgment on the equipment operator based on the detection combination. A feature cascade classifier is used to detect the human eyes, and a particle filter algorithm is used to track the eye position; grayscale processing is used to detect the mouth and modify the critical point to improve the detection accuracy, and then the head state is judged by the relative position of the feature points. By selecting the above three methods and combining them into a total detection method, a comprehensive detection of equipment operators in different states is performed. After the detection, parameters are obtained and a fatigue detection model for equipment operators is established; a support vector machine is used to perform machine learning on the model to analyze in real time whether the operator is fatigued at this time, which well meets the application demand of analyzing the representation parameters under different states to derive the degree of fatigue.

Description

Technical Field

[0001] The present invention belongs to the field of fatigue state detection, and in particular relates to a method for detecting the fatigue state of an equipment operator based on visual feature information fusion. Background Art

[0002] Current research on fatigue detection focuses primarily on fatigue driving, with most focusing on vehicle behavior, behavioral, and physiological characteristics. The former primarily measures steering wheel deviation and the vehicle's distance from the road's center white line to predict fatigue, but is not practical for current equipment. While the latter two are widely used for fatigue detection, some rely on physiological parameters such as EEG (electroencephalogram), EKG (electrocardiogram), and EMG (electromyogram). MIT has developed a system called Smart Car, which incorporates various sensors to measure EKG and EMG. HARKEN, developed by Nottingham Trent University in the UK, can monitor a driver's breathing rate and heart rate to detect fatigue. However, similar research results generally suffer from high cost and difficulty in real-time monitoring. Furthermore, since physiological signal detection often requires contact with the subject, it can be disruptive to the user and hinders widespread adoption.

[0003] Among the currently widely used facial detection methods, the eyelid closure time (PERCLOS) is the most effective parameter for fatigue characterization. A binocular camera is used to calculate pupil position and size, and PERCLOS calculations are achieved by taking advantage of the difference in the human eye's retina's reflection of infrared light of different wavelengths. However, single-feature fatigue detection methods have limitations, and over-reliance on facial recognition, light brightness, and a large number of calculations make it difficult to achieve good results. Furthermore, in the detection of head movement patterns, the algorithm is mainly targeted at vehicle driving, and the trajectory of head movement judgment is too simple to be compatible with fatigue detection for device operation. Simulation testing and statistics of the data set need to be re-performed, and the algorithm needs to be optimized to adapt to the purpose of detection in device scenarios. Summary of the Invention

[0004] The present invention provides a method for detecting the fatigue status of equipment operators based on visual feature information fusion, extending fatigue detection and recognition from fatigue driving detection to equipment operation fatigue detection, re-optimizing the algorithm and performing special targeted processing on the movement trajectory of the head, thereby obtaining a robust fatigue detection model.

[0005] The technical solution to realize the present invention is: a method for detecting fatigue status of equipment operators based on visual feature information fusion, comprising the following steps:

[0006] Step 1: According to the "Group Standard of the Chinese Association of Traditional Chinese Medicine", collect videos of the operator's head information in the awake and fatigue states specified in the standard. The head information includes the head outline, eyes and mouth.

[0007] Step 2: Perform eye detection on the video based on the Haar-Like feature cascade classifier detection method, and perform eye tracking based on the particle filter of mean shift theory. After particle filter resampling, particles with small weights are discarded and particles with large weights are retained. New particles are derived from particles with large weights, and all derived new particles have the same weights. Then, the mean shift theory is used to iterate each newly generated particle. After the iteration, the position of each particle is updated so that each particle converges near the target. Continuous tracking is used to determine the real-time position of the eye, thereby obtaining real-time eye blink parameters through the existing eyelid closure parameter, namely the PERCLOS measurement principle; and at the same time, proceed to step 3.

[0008] Step 3. Determine the position of the mouth area using the ASM method, use the area within the mouth feature points as the input image, and convert the input RGB image into a grayscale image; perform image enhancement on the grayscale image, adjust its grayscale value until the grayscale value intensity between the mouth area and the background exceeds the set threshold; binarize the image of the mouth area, remove noise and fill the blank area of ​​the mouth through morphological reconstruction, extract the height value of the mouth area and the ratio of black and white pixels near the mouth, evaluate the degree of opening, compare the evaluation result with the standard yawning critical point, obtain information on whether yawning is occurring at this time, and use whether yawning is occurring in the current state as the mouth parameter; proceed to step 4.

[0009] Step 4. Extract facial feature parameters based on SIFT feature points, that is, the relative position parameters of the eyes and mouth in the triangular feature area of ​​the eyes and mouth, take the triangular feature area as the region of interest ROIs, and extract the SIFT feature points of the eyes and mouth in the region of interest ROIs based on the matching algorithm of the SIFT descriptor; first determine the main gradient direction of the SIFT feature point descriptor, that is, the peak position of the pixel gradient histogram in the SIFT feature point field; then use a 4×4×8 128-dimensional vector as the descriptor of the SIFT feature point, and extract the eye and mouth SIFT feature point sets from the image to be processed through feature point matching; obtain the Euclidean distance between the SIFT feature point descriptor of the template image ROIs and the feature point descriptor of the region of interest ROIs through feature point matching; then compare the ratio of the nearest Euclidean distance p value and the second nearest Euclidean distance value with the threshold value. If p is less than the threshold value, the feature point of the region of interest ROIs is the SIFT feature point, and the head swing angle, that is, the head parameter, is determined by the acquired feature points; go to step 5.

[0010] Step 5: Based on the blink parameters, mouth parameters, and head parameters obtained in the above steps, a fatigue detection model for equipment operators is established based on a support vector machine through hybrid optimization of a genetic algorithm and a particle swarm algorithm.

[0011] Step 6: Using the equipment operator fatigue detection model, determine whether the operator in the video containing head information is awake or fatigued.

[0012] Compared with the prior art, the present invention has the following significant advantages:

[0013] (1) The most rigorous eyelid closure parameter (PERCLOS) is selected as the main detection basis for more efficient eye positioning and tracking.

[0014] (2) Comprehensive mouth parameters are used for auxiliary detection, and grayscale processing is used to improve the accuracy of mouth recognition, effectively improving the recognition of the mouth.

[0015] (3) By extracting facial SIFT feature points and analyzing the feature points, the rotation angles of the three coordinate axes corresponding to the head movement trajectory are detected to better meet the targeted detection of device operations.

[0016] (4) The three characterization parameters selected by the above-mentioned technical detection are selected from among the numerous characterization parameters, which not only avoids the high error generated when using a single characterization parameter for judgment, but also avoids the problem of difficulty in balancing the weights when too many characterization parameters are detected at the same time.

[0017] (5) According to the fatigue monitoring table, the parameters of equipment operation are counted to complete the data set, and the influence and judgment of multiple parameters on fatigue are analyzed based on the SVM vector machine to fill the gap in the data set of equipment operating in different states. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is the detection result diagram of the Haar-Like feature cascade classifier.

[0019] Figure 2 Figure 2 is a diagram showing the changes in mouth state, where Figure (a) is a grayscale image processing diagram of the mouth state, Figure (b) is an image enhancement effect diagram of the mouth state, Figure (c) is a binarization processing result diagram of the mouth state, and Figure (d) is a preprocessing result diagram of the mouth state.

[0020] Figure 3 Figure 2 is a simulation of head analysis based on SIFT feature point extraction of facial feature parameters, where (a) is a facial view, (b) is the facial orientation diagram when the head is swung directly downward, (c) is the facial orientation diagram when the head is swung to the lower right, and (d) is the facial orientation diagram when the head is swung to the lower left.

[0021] Figure 4 This is a flow chart of the equipment operator fatigue status detection method based on visual feature information fusion. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0023] Combine Figure 4 A method for detecting fatigue status of equipment operators based on visual feature information fusion, the method comprising the following steps:

[0024] Step 1: Select eyes, mouth, and head contours from various fatigue representations as a detection combination. According to the "Group Standard of the Chinese Association of Traditional Chinese Medicine", collect videos of the operator with the above detection combinations in the awake and fatigue states according to the standard.

[0025] Step 2: Figure 1 As shown in the figure, the detection method based on the Haar-Like feature cascade classifier is used to detect human eyes in the above video, and the particle filter based on the mean shift theory is used to track eyes. After the particle filter is resampled, particles with small weights are discarded and particles with large weights are retained. New particles are derived from particles with large weights, and all the new particles derived have the same weights. Then, the mean shift theory is used to iterate each newly generated particle. After the iteration, the position of each particle is updated so that each particle converges to the vicinity of the target. Uninterrupted tracking is used to determine the real-time position of the eye, so that the real-time eye blink parameter can be obtained through the existing eyelid closure parameter, namely the PERCLOS measurement principle. The target can be tracked with a small number of particles, which improves the real-time performance of tracking, as shown in the figure.

[0026] Step 2-1, particle sampling, during initialization, determine the position x0 of the eye target, and take the number of particles as N s , t represents the particle number, and its initial weight value is 1 / N s , the sum of the weights of all particles is 1, and each particle represents a possible motion state of the target, that is, a possible position x of the target t =x0+bξ, where ξ is a random number in [-1,1], indicating the number of possible random positions, and the coefficient b is a constant. Calculate the histogram of the target search window centered on each particle H(b) = (n b ,u b ,∑ b ), when the center of the target area is x0 at the beginning, the second-order histogram Target feature B = 1,…,N s , the mean vector Covariance matrix Represents the histogram coefficient; go to step 2-2.

[0027] Step 2-2, particle state transfer, using a first-order system to achieve eye tracking; particle coordinates (X t , Y t ) is the first-order system state transfer equation: The coefficients A1, B1, A2, and B2 are constants, and the parameter w t-1 is a random number in [0,1], go to step 2-3.

[0028] Step 2-3: Particle update. Observe the similarity between the possible target state represented by each particle and the target's true state. Particles that are close to the target's true state are given a larger weight, and particles that are not close to the target's true state are given a smaller weight. Go to step 2-4.

[0029] Step 2-4: Calculate the weighted value of each particle to determine the final position of the target: x new , go to steps 2-5.

[0030] Step 2-5: Resampling: By resampling the probability density function represented by the particles and the corresponding weights, the number of particles with larger weights is increased, and then proceed to step 2-6.

[0031] Step 2-6: Particle optimization based on mean shift. According to MeanShift theory, the mean shift vector points to the direction with the greatest similarity, that is, the position most similar to the target. The mean shift iteration is used to update the position of the particle.

[0032] Go to step 3.

[0033] Step 3: Determine the position of the mouth area by the ASM method, take the area within the mouth feature points as the input image, and convert the input RGB image into a grayscale image, thereby reducing the image processing time. Figure 2 As shown in Figure (a); perform image enhancement on the grayscale image and adjust its grayscale value until the grayscale intensity between the mouth area and the background exceeds the set threshold; perform binarization on the image of the mouth area, as shown in Figure (a); perform image enhancement on the grayscale image and adjust its grayscale value until the grayscale intensity between the mouth area and the background exceeds the set threshold; Figure 2 (b) Figure and Figure 2 As shown in Figure (c), morphological reconstruction is used to remove noise and fill the blank area of ​​the mouth. The height value of the mouth area and the ratio of black and white pixels near the mouth are extracted to evaluate the degree of opening. The results are shown as follows: Figure 2 In image (d), the evaluation results are compared with the standard yawning threshold. To address the phenomenon that operator fatigue gradually increases during equipment operation, we decided to change the fixed threshold to two targeted thresholds. The comparison of whether the operator exceeds the thresholds is used to obtain information on whether the operator yawns. The details are as follows:

[0034] The height value of the mouth area and the ratio of black and white pixels near the mouth must meet three basic conditions. The specific formula is as follows:

[0035] NBS>Th1

[0036] NWC>Th2

[0037] h>Th3

[0038] Where NBS indicates that the ratio of the number of black pixels in the current frame to the reference frame must be greater than a threshold value Th1; NWC indicates that the ratio of the number of black pixels in the mouth area of ​​the current frame to the number of white pixels in the area around the mouth must be greater than a threshold value Th2; h indicates the quotient of the mouth area, and the height must be greater than a threshold value Th3. Only when all three conditions are met at the same time is the detected specific frame determined to be a yawn.

[0039] Because operators have to operate the equipment for a long time, the critical point for yawning detection needs to be more strictly adjusted after yawning is detected to improve the accuracy of subsequent real-time detection and ensure that fatigue can be accurately detected during long-term operation. Therefore, after the first yawn is detected, the basic condition formula is modified to:

[0040] NBS>0.8*Th1

[0041] NWC>0.8*Th2

[0042] h>0.8*Th3

[0043] This critical point will be used as the standard for subsequent cyclic testing until the operator stops the operation and interrupts the testing, and then the original critical point is restored.

[0044] Go to step 4.

[0045] Step 4: Extract facial feature parameters based on SIFT feature points, that is, the relative position parameters of the eyes and mouth in the triangular feature area of ​​the eyes and mouth, such as Figure 3As shown in the figure, the relative distance between the left and right eyes and the relative distance between the left and right eyes and the mouth, namely d1, d2, and d3 in the figure, are used to analyze the facial orientation of the subject. The specific analysis details are as follows: the triangular feature area is used as the region of interest ROIs, and the SIFT feature points of the eyes and mouth in the region of interest ROIs are extracted based on the matching algorithm of the SIFT descriptor; first, the main gradient direction of the SIFT feature point descriptor is determined, that is, the peak position of the pixel gradient histogram in the SIFT feature point area; then a 4×4×8 128-dimensional vector is used as the descriptor of the SIFT feature point, and the SIFT feature point set of the eyes and mouth is extracted from the image to be processed through feature point matching; the Euclidean distance between the SIFT feature point descriptor of the template image ROIs and the feature point descriptor of the region of interest ROIs is obtained through feature point matching; then the ratio of the nearest Euclidean distance p value and the second nearest Euclidean distance value is compared with the threshold. If p is less than the threshold, the feature point of the region of interest ROIs is the SIFT feature point, and the head swing angle, that is, the head parameter, is determined by the acquired feature points.

[0046] Based on SIFT feature point matching, the SIFT feature points of the left eye, right eye and mouth are obtained and used as the processing object for extracting facial features; then the midpoint coordinates L(x L ,y L )、R(x R ,y R )、M(x M ,y M ), calculate the following three eigenvectors based on the midpoint coordinates. The specific steps are as follows:

[0047] Step 4-1, extract eye feature vector: Assume that the SIFT feature point sets of the left eye and the right eye are {(x Li ,y Li )}、{(x Rj ,y Rj )}, where the coordinate parameters of the left eye feature point set i = 1, 2, ..., I, and the coordinate parameters of the right eye feature point set j = 1, 2, ..., J;

[0048] Calculate all feature points of the left eye to L(x L ,y L ) distance d i (L), all feature points of the right eye to R(x R ,y R ) distance d j (L)

[0049]

[0050]

[0051] Where, the midpoint abscissa of the left eye feature point is The midpoint ordinate of the left eye feature point The horizontal coordinate of the midpoint of the SIFT feature point of the mouth The midpoint ordinate of the SIFT feature point of the mouth

[0052] According to d i (L), d i (R) Get the mean of the feature points of both eyes to the midpoint of both eyes And the standard deviations δ(L) and δ(R) when calculating the above distances.

[0053] Step 4-2, extract mouth feature parameters: Assume that the SIFT feature point set of the mouth is {M(k)=(x Mk ,y Mk )}, the feature point number k=1,2,……,K, K is the number of all feature points; calculate all the feature points of the eye to M(x Mk ,y Mk ) distance d K (M):

[0054]

[0055] In the formula, the horizontal axis vertical axis

[0056] Go to step 5.

[0057] Step 5: For the three characterization parameters selected for detection, namely the blink parameters, mouth parameters, and head parameters obtained in the above steps, an equal weighted combination is performed to form a unique comprehensive detection characterization parameter set. Then, the comprehensive characterization parameter set is hybrid optimized using a genetic algorithm and a particle swarm algorithm. Based on a support vector machine, an equipment operator fatigue detection model is established, as follows:

[0058] Step 5-1, parameter preprocessing. In order to eliminate the dimensional influence between the acquired parameter eigenvalues ​​and ensure the accuracy of the equipment operator fatigue detection model, it is necessary to normalize the data. The most effective data normalization method is the maximum normalization method. The blink parameters, mouth parameters, and head parameters obtained in the above steps are linearly transformed and the results are mapped to a specified interval. The results are mapped to the interval [0, 1] as follows:

[0059]

[0060] Among them, x, y are different parameter eigenvalues, x min is the minimum value of the sample data, xmax This method is used to normalize the training set and test set data to the interval [0, 1] to improve the accuracy of model classification.

[0061] Step 5-2: Optimize the kernel function parameter g and penalty parameter c, select the optimal parameters, and further improve the classification accuracy of the equipment operator fatigue detection model. The optimization method is to use the K-fold Cross Validation method (K-CV) to find the optimal function parameter g and penalty parameter c in a certain sense. First, select the values ​​of the kernel function parameter g and penalty parameter c within a certain range. Divide the blink parameters, mouth parameters, and head parameters obtained in the above steps into z groups on average. Each time, use one subset of data as a test set, and the remaining z-1 groups of subset data as training sets. In this way, constructing a classifier will obtain z support vector machine models. The average classification accuracy of the z support vector machine model test sets is used as the performance indicator of the classifier under this specified parameter. Change the parameter value again and repeat the above steps. Finally, the set of parameters that gives the highest classification accuracy of the test set is taken as the optimal parameter.

[0062] Step 5-3: Use the GA-PSO hybrid algorithm to apply the crossover and mutation concepts of the genetic algorithm to the particle swarm algorithm. When updating individual particles, first select individual particles with lower fitness and cross them with the optimal individual particle. If the fitness of the newly generated particle is higher than the previous one, the new particle replaces the original one; otherwise, the two particles are not exchanged. After all particles have completed the crossover operation, the new particle is mutated. Similarly, if the fitness of the newly generated particle is higher than the previous one, the new particle replaces the original one; otherwise, the two particles are not exchanged. Repeat these steps to iteratively update the particles until the termination condition is met, and the global optimal solution for the equipment operator fatigue detection model is obtained.

[0063] Step 6: Using the equipment operator fatigue detection model, determine whether the operator in the new video containing head information is awake or fatigued.

Claims

1. A method for detecting fatigue status of equipment operators based on visual feature information fusion, characterized in that: The method comprises the following steps: Step 1: Select the eyes, mouth, and head outline as a detection combination from various fatigue representations, and collect videos of the operator with the above detection combinations in the awake and fatigued states respectively; Step 2: Perform eye detection on the video using a Haar-Like feature cascade classifier. Perform eye tracking using a particle filter based on mean shift theory. After resampling, particles with small weights are discarded and particles with large weights are retained. New particles are derived from particles with large weights, and all derived new particles have the same weight. Then, the mean shift theory is used to iterate each newly generated particle. After the iteration, the position of each particle is updated so that each particle converges near the target. Continuous tracking is used to determine the real-time position of the eye, thereby obtaining real-time eye blink parameters using the existing eyelid closure parameters and the PERCLOS measurement principle. At the same time, the process proceeds to step 3. Step 3: Determine the position of the mouth area using the ASM method, use the area within the mouth feature points as the input image, and convert the input RGB image into a grayscale image; perform image enhancement processing on the grayscale image, adjust its grayscale value until the grayscale value intensity between the mouth area and the background exceeds the set threshold; binarize the image of the mouth area, remove noise and fill the blank area of ​​the mouth through morphological reconstruction, extract the height value of the mouth area and the ratio of black and white pixels near the mouth, evaluate the degree of opening, and compare the evaluation results with the standard yawning critical point. In view of the phenomenon that the operator's fatigue level will gradually increase during the operation of the equipment, it is decided to change the fixed critical point to two targeted critical points. Comparison is made on whether the critical point is exceeded to obtain information on whether the yawn is present at this time, and whether the current state of yawning is present is used as the mouth parameter; then proceed to step 4; Step 4, extract facial feature parameters based on SIFT feature points, including the relative position parameters of the eyes and mouth in the triangular feature area of ​​the eyes and mouth, take the triangular feature area as the region of interest ROIs, and extract the SIFT feature points of the eyes and mouth in the region of interest ROIs based on the matching algorithm of the SIFT descriptor; first determine the main gradient direction of the SIFT feature point descriptor, specifically the peak position of the gradient histogram of the pixel neighborhood of the SIFT feature point; then use the 4×4×8 128-dimensional vector as the descriptor of the SIFT feature point, and extract the eye and mouth SIFT feature point sets from the image to be processed by feature point matching; obtain the Euclidean distance between the SIFT feature point descriptor of the template image ROIs and the feature point descriptor of the region of interest ROIs by feature point matching; then, calculate the ratio of the nearest Euclidean distance p value and the second nearest Euclidean distance value, and compare the above ratio with the threshold. If p is less than the threshold, the feature point of the region of interest ROIs is the SIFT feature point, and the head swing angle is determined as the head parameter by the acquired feature points; go to step 5; Step 5: The acquired blink parameters, mouth parameters, and head parameters are combined with equal weights to form a comprehensive detection parameter set. The comprehensive parameter set is then optimized using a genetic algorithm and a particle swarm optimization algorithm. A fatigue detection model for equipment operators is established based on a support vector machine. Step 6: Using the equipment operator fatigue detection model, determine whether the operator in the video containing head information is awake or fatigued.

2. The method for detecting fatigue status of equipment operators based on visual feature information fusion according to claim 1, characterized in that: The specific steps in step 2 are as follows: Step 2-1, particle sampling, during initialization, determine the position x0 of the eye target, and take the number of particles as N s , t represents the particle number, and its initial weight value is 1 / N s , the sum of the weights of all particles is 1, and each particle represents a possible motion state of the target, that is, a possible position x of the target t =x0+bξ, where ξ is a random number in [-1,1], indicating the number of possible random positions, and the coefficient b is a constant. The histogram of the target search window centered on each particle is calculated. H(b) = (n b ,u b ,∑ b ), n b is the second-order histogram, u b is the mean vector, ∑ b is the covariance matrix; go to step 2-2; Step 2-2, particle state transfer, using a first-order system to achieve eye tracking; particle coordinates (X t , Y t ) is the first-order system state transfer equation: The coefficients A1, B1, A2, and B2 are constants, and w t-1 is a random number in [0, 1], go to step 2-3; Step 2-3: Particle update. Observe the similarity between the target state represented by each particle and the target's true state. Particles that are close to the target's true state are given a larger weight, and particles that are not close to the target's true state are given a smaller weight. Go to step 2-4. Step 2-4: Calculate the weighted value of each particle to determine the final position x of the target new , go to step 2-5; Step 2-5, resampling, by resampling the probability density function represented by the particles and the corresponding weights, increasing the number of particles with larger weights, and then proceeding to step 2-6; Step 2-6: Particle optimization based on mean shift. According to MeanShift theory, the mean shift vector points to the direction with the greatest similarity. As the position most similar to the target, the mean shift iteration is used to update the position of the particle.

3. The method for detecting fatigue status of equipment operators based on visual feature information fusion according to claim 2, characterized in that: In step 3, the height value of the mouth area and the ratio of black and white pixels near the mouth must meet three basic conditions. The specific critical points are as follows: NBS>Th1 NWC>Th2 h>Th3 Where NBS represents the ratio of the number of black pixels in the current frame to the reference frame, which must be greater than a threshold value Th1. NWC represents the ratio of the number of black pixels in the mouth area of ​​the current frame to the number of white pixels in the area around the mouth, which must be greater than a threshold value Th2. h represents the height of the mouth area, which must be greater than a threshold value Th3. Only when all three conditions are met at the same time is the detected specific frame determined to be a yawn. After the first yawn is detected, the basic condition formula is modified to: NBS>0.8*Th1 NWC>0.8*Th2 h>0.8*Th3 This critical point will be used as the standard for subsequent cyclic testing until the operator stops the operation and interrupts the testing, and then the original critical point is restored.

4. The method for detecting fatigue status of equipment operators based on visual feature information fusion according to claim 3 is characterized in that: In step 4, SIFT feature points of the left eye, right eye and mouth are obtained based on SIFT feature point matching and used as processing objects for extracting facial features; then the midpoint coordinates of the SIFT feature points of the left eye, right eye and mouth are calculated to correspond to L(x L ,y L )、R(x R ,y R )、M(x M ,y M ), calculate the following three eigenvectors based on the midpoint coordinates. The specific steps are as follows: Step 4-1, extract eye feature vector: Assume that the SIFT feature point sets of the left eye and the right eye are {(x Li ,y Li )}、{(x Rj ,y Rj )}, where the coordinate parameters of the left eye feature point set i = 1, 2, ..., I, and the coordinate parameters of the right eye feature point set j = 1, 2, ..., J; Calculate all feature points of the left eye to L(x L ,y L ) distance d i (L), all feature points of the right eye to R(x R ,y R ) distance d j (R): Where, the midpoint abscissa of the left eye feature point is The midpoint ordinate of the left eye feature point The horizontal coordinate of the midpoint of the right eye feature point The midpoint ordinate of the right eye feature point According to d i (L) Get the mean value from the left eye feature point to the left eye According to d j (R) Get the right eye feature point to the right eye mean use Calculate the standard deviation δ(L) and δ(R); Step 4-2, extract mouth feature parameters: Assume that the SIFT feature point set of the mouth is {M(k)=(x Mk ,y Mk )}, feature point number k=1,2,……,K, K is the number of all feature points; calculate the midpoint of the SIFT feature points of the mouth to M(x Mk ,y Mk ) distance d k (M): Where, the horizontal coordinate of the midpoint of the SIFT feature point of the mouth is The vertical coordinate of the midpoint of the SIFT feature point of the mouth 5. The method for detecting fatigue status of equipment operators based on visual feature information fusion according to claim 1, characterized in that: In step 5, the specific steps are as follows: Step 5-1: Use the maximum normalization method to perform linear transformation on the blink parameters, mouth parameters, and head parameters obtained in the above steps, and map the results to the [0, 1] interval. The mapping is as follows: Among them, x is each sample data, x min is the minimum value of the sample data, x max is the maximum value of the sample data; Step 5-2: Optimize the kernel function parameter g and the penalty parameter c, select the optimal parameters, and further improve the classification accuracy of the equipment operator fatigue detection model; the optimization method is to use the K-fold cross-validation method to find the optimal function parameter g and penalty parameter c; first, select the values ​​of the kernel function parameter g and the penalty parameter c, and divide the blink parameters, mouth parameters, and head parameters obtained in the above steps into z groups on average. Each time, one subset of data is used as the test set, and the remaining z-1 groups of subset data are used as the training set. In this way, the classifier will obtain z support vector machine models. The average classification accuracy of the z support vector machine model test sets is used as the performance indicator of the classifier under this specified parameter. Change the parameter value again, repeat the above steps, and finally take the set of parameters that gives the highest classification accuracy of the test set as the optimal parameter; Step 5-3: Use the GA-PSO hybrid algorithm to apply the crossover and mutation ideas of the genetic algorithm to the particle swarm algorithm. When updating individual particles, first select the individual particles with lower fitness and the individual optimal particles for crossover operation. If the fitness of the newly generated particles is higher than the previous ones, then the new particles will replace the original particles, otherwise the two particles will not be exchanged. After all the particles are crossover, the new particles are mutated. Similarly, if the fitness of the newly generated particles is higher than the previous ones, then the new particles will replace the original particles, otherwise the two particles will not be exchanged. Repeat the above steps to iteratively update the particles until the end condition is met and the global optimal solution for the equipment operator fatigue detection model is obtained.

Images