Portrait enhancement correction fatigue detection method and device in low-light multi-angle state

Abstract

The embodiment of the present application discloses a kind of portrait enhancement correction fatigue detection method and device under low light multi-angle state, it is related to safe driving technical field, fatigue detection can be carried out on the basis of low light image enhancement and portrait posture correction, to improve detection accuracy.The method comprises: obtaining first image;Based on the first image and the second image formed using adaptive texture analysis method, image enhancement processing is carried out, and third image is obtained;Wherein, the adaptive texture analysis method, based on the local binary pattern of window area is carried out feature coding;Affine transformation matrix formed based on local feature extraction is carried out portrait posture correction to the third image, and fourth image is obtained;Based on the key point of face in the fourth image, fatigue detection is carried out.The present application can be used in portrait enhancement correction fatigue detection under low light multi-angle state.

Description

Technical Field

[0001] This invention relates to the field of safe driving technology, and in particular to a method, device, and electronic device for fatigue detection of human images under low-light multi-angle conditions with enhanced correction. Background Technology

[0002] In recent years, fatigued driving has become a major hidden danger in traffic accidents, especially at night, where the proportion of traffic accidents caused by fatigued driving has been increasing year by year. However, images obtained in low-light environments tend to have high noise levels and suffer from problems such as blurred texture information and incorrect color reproduction, which pose certain difficulties for subsequent image processing and fatigue detection.

[0003] Currently, low-light image enhancement techniques include traditional algorithms such as histogram equalization and gamma transform, which introduce sampling blocks and convolution to refine features and enhance feature representation. However, when processing images under low-light conditions, due to insufficient illumination, high noise, and poor image clarity, the enhanced image features suffer from details loss, distortion, and excessive noise suppression, affecting subsequent fatigue detection processes and leading to a decrease in detection accuracy. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus and electronic device for fatigue detection of human portrait enhancement and correction under low light multi-angle conditions, which performs fatigue detection on the basis of low light image enhancement and human portrait posture correction, thereby improving the detection accuracy.

[0005] In a first aspect, embodiments of the present invention provide a method for fatigue detection and image enhancement correction under low-light multi-angle conditions, comprising: acquiring a first image; performing image enhancement processing on the first image and a second image formed using an adaptive texture analysis method to obtain a third image; wherein the adaptive texture analysis method performs feature encoding based on the local binary pattern of a window region; performing portrait pose correction on the third image based on the affine transformation matrix formed by local feature extraction to obtain a fourth image; and performing fatigue detection based on key points of the face in the fourth image.

[0006] In one implementation, the feature encoding based on the local binary pattern of the window region includes: setting the size of the window; determining the center point of the window as the center pixel; dividing the adjacent pixels into two regions: a first feature region and a second feature region based on the feature fit between the adjacent pixels and the center pixel; adjusting the values ​​of the pixels in the two regions based on the different regions, and performing feature encoding based on the adjustment results.

[0007] In one implementation, adjusting the pixel values ​​in the two regions based on the different regions and performing feature encoding based on the adjustment results includes: differentiating the values ​​of a pixel in the second feature region from the values ​​of two adjacent pixels in the first feature region; averaging the values ​​of all pixels in the first feature region to determine the trend value of the center pixel; adjusting the values ​​of all pixels in the first feature region based on the difference between the values ​​of all pixels in the first feature region and the trend value of the center pixel; determining the feature threshold of the center pixel based on the average of the adjusted values ​​of all pixels in the first feature region; comparing the adjusted value of a pixel in the first feature region with the differentiated value of a pixel in the second feature region with the feature threshold; if the value is not less than the feature threshold of the center pixel, assigning a feature encoding value of 1 to the pixel, otherwise assigning a value of 0; and determining the feature encoding of the center pixel based on the assigned feature encoding values ​​of all pixels.

[0008] In one embodiment, the step of differentiating the value of a pixel based on the two adjacent pixels of the second feature region includes: adjusting the value of the pixel based on the difference between the value of the pixel in the second feature region and the average value of the two adjacent pixels of the first feature region.

[0009] In one embodiment, determining the feature code of the center pixel based on the feature code assignment of each pixel includes: combining the feature codes of each pixel in the first feature region and the second feature region in sequence and then adding them together to determine the feature code of the center pixel.

[0010] In one implementation, the process of enhancing the image based on the first image and the second image formed using an adaptive texture analysis method to obtain a third image includes: setting the number of scales and the weight corresponding to each scale; for each scale, performing a Gaussian blur function to multiply a feature point of the first image and the corresponding point of the second image, and then calculating the reflection component; adding the reflection components calculated at different scales according to their weights to calculate the final enhanced image; and obtaining the third image based on the final enhanced image.

[0011] In one implementation, the affine transformation matrix formed based on local feature extraction is used to correct the portrait pose of the third image to obtain a fourth image, including: locating the position of the face based on the third image; locating key points of the face based on the position of the face; the key points of the face include at least one of the following: eyes, mouth, head, and nose tip; extracting local shape and texture features of the key points of the face using a local binary pattern algorithm; calculating an affine transformation matrix based on the extracted local shape and texture features, the affine transformation matrix including translation and midpoint rotation angle; and correcting the portrait pose of the third image based on the affine transformation matrix formed based on the local feature extraction to obtain the fourth image.

[0012] In one implementation, the fatigue detection based on key points of the face in the fourth image includes: determining key points of the eyes, mouth, and head based on the fourth image; determining the aspect ratio of the eyes based on the key points of the eyes; determining the mouth opening degree and mouth corner size based on the key points of the mouth; determining the vertical and horizontal degrees of the head based on the key points of the head; and comparing the changes in the eye length ratio, mouth opening degree, mouth corner size, and vertical and horizontal degrees of the head over a continuous time interval with preset thresholds to comprehensively determine the fatigue level as one of the following: alert, moderate fatigue, or severe fatigue.

[0013] Secondly, embodiments of the present invention also provide a fatigue detection device for portrait enhancement and correction under low-light multi-angle conditions, comprising: an acquisition unit for acquiring a first image; an enhancement processing unit for performing image enhancement processing on the first image and a second image formed using an adaptive texture analysis method to obtain a third image; wherein the adaptive texture analysis method performs feature encoding based on the local binary pattern of a window region; a posture correction unit for performing portrait posture correction on the third image based on an affine transformation matrix formed by local feature extraction to obtain a fourth image; and a fatigue detection unit for performing fatigue detection based on key points of the face in the fourth image.

[0014] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising: a memory and a processor, wherein the memory is used to store computer-readable instructions; and the processor is used to read the computer-readable instructions and implement the portrait enhancement correction fatigue detection method under low light multi-angle conditions as described in any one of claims 1 to 8.

[0015] The fatigue detection method, apparatus, and electronic device for portrait enhancement and correction under low-light multi-angle conditions provided in this invention can perform image enhancement processing based on texture features extracted by adaptive texture analysis methods, improving the problems of insufficient low-light enhancement or detail feature extraction. Furthermore, by performing portrait pose correction based on the affine transformation matrix formed by local feature extraction before fatigue detection, the significant fatigue detection deviation caused by asymmetric image detail loss due to image acquisition angles is greatly mitigated. Therefore, performing fatigue detection based on low-light image enhancement and portrait pose correction improves detection accuracy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram of existing image enhancement technology;

[0018] Figure 2 A flowchart of a fatigue detection method for portrait enhancement and correction under low-light multi-angle conditions provided in an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram of window region division in an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of feature encoding in an embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram of the image enhancement principle provided in an embodiment of the present invention;

[0022] Figure 6 This is a flowchart of fatigue detection provided in an embodiment of the present invention;

[0023] Figure 7 This is a schematic diagram of key points of the mouth provided in an embodiment of the present invention;

[0024] Figure 8 A flowchart of a fatigue detection method for portrait enhancement and correction under low-light multi-angle conditions provided in an embodiment of the present invention;

[0025] Figure 9 A schematic diagram of a fatigue detection device for portrait enhancement and correction under low light multi-angle conditions provided in an embodiment of the present invention;

[0026] Figure 10A schematic diagram of an electronic device provided as an embodiment of the present invention. Detailed Implementation

[0027] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0028] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0029] In a first aspect, embodiments of the present invention provide a fatigue detection method for portrait enhancement and correction under low-light multi-angle conditions, which can perform fatigue detection based on low-light image enhancement and portrait posture correction, thereby improving the detection accuracy.

[0030] like Figure 2 As shown, an embodiment of the present invention provides a method for detecting fatigue in portrait enhancement and correction under low-light multi-angle conditions, comprising:

[0031] S11, Obtain the first image.

[0032] In this step, images are acquired for fatigue detection. These images can be captured photographs or several frames extracted from a recorded video. In practice, the time intervals of multiple consecutive images in the video can also be recorded. By analyzing these consecutive images and the elapsed time, the changes in the images within a specific time period are calculated and compared with a set threshold to determine the fatigue level. The images used for fatigue detection can be in different formats, such as bmp, jpg, png, tif, gif, etc.

[0033] It should be noted that the quality of the acquired images used for fatigue detection affects the accuracy of the detection. To improve accuracy, the acquired images need to be processed in subsequent steps.

[0034] S12, based on the first image and the second image formed using an adaptive texture analysis method, image enhancement processing is performed to obtain a third image; wherein, the adaptive texture analysis method performs feature encoding based on the local binary pattern of the window region.

[0035] After acquiring the first image, this step first processes it using an adaptive texture analysis method. Adaptive texture analysis can extract key feature information from the image, such as edges and textures. This information is crucial for subsequent image enhancement, effectively enhancing the image under low-light conditions.

[0036] In this step, the adaptive texture analysis method performs feature encoding based on the local binary pattern of the window region. Specifically, the center point of the window can be determined as the center pixel. Based on the feature fit between adjacent pixels and the center pixel, the adjacent pixels are divided into multiple regions. The values ​​of pixels within different regions are adjusted, and feature encoding is performed based on the adjustment results. Using the adaptive texture analysis method, key texture information is extracted from the first image to form the second image.

[0037] To mitigate the impact of low-light conditions on images, effective image enhancement is performed based on adaptive texture analysis. Image enhancement in low light conditions can utilize traditional histogram equalization and gamma transform, or multi-scale Retinex algorithms, among others.

[0038] This application performs image augmentation processing on a first image and a second image formed using an adaptive texture analysis method to obtain a third image.

[0039] S13, Based on the affine transformation matrix formed by local feature extraction, the third image is subjected to portrait pose correction to obtain the fourth image.

[0040] In an embodiment of the present invention, the third image is subjected to portrait pose correction based on the affine transformation matrix formed by local feature extraction to obtain the fourth image. Non-forward image acquisition is prone to the problem of missing details in some feature regions, leading to a decrease in detection accuracy. In this step, affine transformation technology based on local feature extraction is used to correct the portrait pose of the enhanced image, facilitating feature point recognition for the next step of fatigue detection.

[0041] In practice, face detection is performed first to quickly and accurately locate the face in the third image. After detecting the face, key facial features such as the corners of the eyes, mouth, and nose are located. By analyzing the changes in these key features, an affine transformation matrix can be calculated. Finally, using the affine transformation matrix, each pixel in the third image can be transformed to obtain the corrected face image, resulting in the fourth image.

[0042] S14, fatigue detection is performed based on the key points of the face in the fourth image.

[0043] In this step, fatigue detection is performed based on key points of the face in the fourth image. In practice, key points such as the eyes, mouth, and head can be identified first, and the state or changes of these key points can be used to determine whether the person is in a state of fatigue or the level of fatigue, such as: alert, moderately fatigued, or severely fatigued.

[0044] The fatigue detection method for portrait enhancement and correction under low-light multi-angle conditions provided in this invention can perform image enhancement processing based on texture features extracted by adaptive texture analysis, thus improving the problems of insufficient low-light enhancement or detail feature extraction. Furthermore, by performing portrait pose correction based on the affine transformation matrix formed by local feature extraction before fatigue detection, it significantly improves the large fatigue detection deviation caused by asymmetric image detail loss due to image acquisition angle. Therefore, performing fatigue detection based on low-light image enhancement and portrait pose correction improves detection accuracy.

[0045] Specifically, in one embodiment of the present invention, step S12 can be based on the local binary pattern of the window region for feature encoding, including: setting the size of the window; determining the center point of the window as the center pixel; dividing the adjacent pixels into two regions: a first feature region and a second feature region based on the feature fit between the adjacent pixels and the center pixel; adjusting the values ​​of the pixels in the two regions based on the different regions, and performing feature encoding according to the adjustment results.

[0046] Specifically, before feature encoding based on local binary patterns of the window region, average pooling preprocessing can be performed first. By first performing average pooling on the low-light image, image noise can be initially suppressed.

[0047] In practice, the pooling window is typically 2x2 or 3x3, and the stride is usually 1 or 2. With a stride of 1, the pooling window moves one pixel at a time across the input feature map; with a stride of 2, the window moves two pixels at a time. Within the pooling window, the values ​​of all pixels are summed, then divided by the number of pixels to obtain the average value as the pooling result.

[0048] The first image is processed by average pooling. Average pooling reduces the dimensionality of the data by decreasing the image resolution, thereby reducing computational and storage requirements. By calculating the average value within a region, average pooling can fuse local feature information within that region, which helps to extract more global features. Since noise is usually randomly distributed, averaging can smooth out these random fluctuations, thus initially suppressing image noise.

[0049] When performing feature encoding based on local binary patterns of a window region, the window size must first be set. The window size can be square or circular. This invention does not impose a limitation on this. The following explanation uses a 3x3 square window as an example.

[0050] like Figure 3 As shown, within a 3x3 square window, the center point G of the window... tThe central pixel is determined, and based on the feature fit between adjacent pixels and the central pixel, the adjacent pixels are divided into two regions: region M and region N. Region M has 4 pixels, namely M1, M2, M3, and M4; region N also has 4 pixels, namely N1, N2, N3, and N4.

[0051] It should be noted that, as Figure 3 In the example shown, pixels adjacent to the center pixel are designated as region M; the remaining pixels are designated as region N. In practice, the division of regions M and N is determined based on the degree of feature fit between adjacent pixels and the center pixel in a given situation. For example, in a circular window, pixels orthogonal to the center pixel are designated as region M; the remaining pixels are designated as region N. Furthermore, the division of regions M and N may be interchanged in later calculations depending on the specific circumstances. In this example, region M is used as the reference point and region N as the adjustment point; however, depending on the actual situation, region N may be used as the reference point and region M as the adjustment point.

[0052] Based on different window regions, the values ​​of pixels within the two regions can be adjusted, and feature encoding can be performed according to the adjustment results. By dividing the adjacent pixels into two regions—a first feature region and a second feature region—based on the degree of feature fit between adjacent pixels and the center pixel, the pixel values ​​can be adjusted according to the different regions. This solves the problem that traditional local binary mode algorithms are too discretized in their encoding process for adjacent pixels, lack neighborhood encoding capabilities, and thus lose some detailed features.

[0053] In another implementation, adjusting the values ​​of pixels in the two regions and performing feature encoding based on the adjustment results includes: differentiating the value of a pixel in the second feature region based on the values ​​of two adjacent pixels in the first feature region; averaging the values ​​of all pixels in the first feature region to determine the trend value of the center pixel; adjusting the values ​​of all pixels in the first feature region based on the difference between the values ​​of all pixels in the first feature region and the trend value of the center pixel; determining the feature threshold of the center pixel based on the average of the adjusted values ​​of all pixels in the first feature region; comparing the adjusted value of a pixel in the first feature region and the differentiated value of a pixel in the second feature region with the feature threshold; if the value is not less than the feature threshold of the center pixel, assigning a feature code of 1 to the pixel, otherwise assigning a value of 0; and determining the feature code of the center pixel based on the assigned feature codes of all pixels.

[0054] like Figure 4 As shown, the values ​​of pixels in two regions are adjusted, and feature encoding is performed based on the adjustment results. Specifically, this includes the following steps:

[0055] S31. Based on the two adjacent pixels of the first feature region of a pixel in the second feature region, perform differential processing on the value of the pixel.

[0056] Specifically, the feature differentiation processing of pixels in region N can be performed using the following calculation formula:

[0057]

[0058] S32. Take the average value of each pixel in the first feature region to determine the trend value of the center pixel.

[0059] Specifically, the tendency value Avg of the center pixel Gt is determined by averaging the values ​​of all pixels in region M, using the following formula:

[0060]

[0061] S33. Adjust the values ​​of each pixel in the first feature region based on the difference between the value of each pixel in the first feature region and the trend value of the center pixel.

[0062] Specifically, the difference between the value of each pixel in region M and the intensity value of the center pixel is used to adjust the value of each pixel in the first feature region. This can be calculated using the following formula:

[0063]

[0064] S34. Based on the average value of the adjustment values ​​of each pixel in the first feature region, determine the feature threshold of the center pixel.

[0065] Specifically, the feature threshold Ln of the center pixel can be determined by averaging the adjusted values ​​of all pixels within region M, using the following formula:

[0066]

[0067] S35. Compare the adjusted value of a pixel in the first feature region and the differentiated value of a pixel in the second feature region with the feature threshold. If the value is not less than the feature threshold of the center pixel, assign the feature code of the pixel to 1; otherwise, assign it to 0.

[0068] Specifically, threshold activation functions are applied to regions M and N respectively: if the value is not less than the feature threshold of the center pixel, the feature code of that pixel is assigned a value of 1; otherwise, it is assigned a value of 0. The threshold activation functions for regions M and N can be calculated using the following formulas:

[0069]

[0070]

[0071] S36. Based on the feature encoding assignment of each pixel, determine the feature encoding of the center pixel.

[0072] Specifically, the feature code LBP of the center pixel can be determined by assigning feature codes to each pixel based on different regions within the window. Alternatively, the feature codes of each pixel in the first and second feature regions can be sequentially combined and concatenated to determine the feature code of the center pixel. Another approach is to use the following calculation formula:

[0073]

[0074] Through the above encoding process, the discrete features in traditional feature extraction are made continuous, preserving the brightness relationship between pixels and their surrounding neighborhoods. This greatly restores the texture feature extraction of human faces in low-light environments and has excellent robustness.

[0075] In another embodiment of the present invention, the value of a pixel is differentiated based on the pixels of two adjacent first feature regions of a pixel in the second feature region, including: adjusting the value of the pixel based on the difference between the value of a pixel in the second feature region and the average value of the pixels of the two adjacent first feature regions.

[0076] In another embodiment of the present invention, the feature code of the center pixel is determined based on the feature code assignment of each pixel, including: combining the feature codes of each pixel in the first feature region and the second feature region in sequence and then adding them together to determine the feature code of the center pixel.

[0077] In another embodiment of the present invention, an image enhancement process is performed based on the first image and a second image formed using an adaptive texture analysis method to obtain a third image, including: setting the number of scales and the weight corresponding to each scale; for each scale, multiplying a feature point of the first image and the corresponding point of the second image using a Gaussian blur function and then performing blurring processing, and then calculating the reflection component; adding the reflection components calculated at different scales according to the weights to calculate the final enhanced image; and obtaining the third image based on the final enhanced image.

[0078] In practice, multi-scale Retinex processing is performed on any feature point I(x,y) in the first image and its corresponding feature point F(x,y) in the second image. Through multi-scale Retinex processing, the illumination component L and reflection component R of the image can be separated at different scales. Figure 1 As shown, the basic idea of ​​Retinex theory is that illumination intensity determines the dynamic range of all pixels in the original image, while the inherent properties of the original image are determined by the object's own reflectance coefficient; that is, it is assumed that the original image is obtained by multiplying the reflected image and the illuminated image. Figure 5 As shown, this invention performs multi-scale Retinex processing based on feature texture extraction.

[0079] Specifically, for each scale k, the Gaussian blur function G is used. k (x,y) pairs The image is blurred, and then the reflection component is calculated. This can be expressed mathematically as follows:

[0080] R k ( x , y ) = log [ I ( x , y ) ⋅ F ( x , y ) ] − log [ G k ( x , y ) ⋅ I ( x , y ) ⋅ F ( x , y ) ]

[0081] Among them, R k (x,y) represents the reflection component calculated at the k-th scale.

[0082] Finally, the reflection components R calculated at different scales are... k (x,y) according to weight ω k Add them together to obtain the final enhanced image R. MFSR ( x,y ) The mathematical expression that can be used is:

[0083]

[0084] Where N represents the number of scales, ω K This represents the weight of the k-th scale.

[0085] Based on the first image and the second image generated using an adaptive texture analysis method, image enhancement processing is performed. This combines the advantages of multi-scale Retinex and feature extraction, effectively enhancing the image under low-light conditions. Through reasonable feature extraction, multi-scale Retinex processing, and reflectance component fusion steps, fine image enhancement is achieved, improving the visual effect and recognition performance of the image.

[0086] In another embodiment of the present invention, a fourth image is obtained by performing portrait pose correction on the third image based on an affine transformation matrix formed by local feature extraction, including: locating the position of a face based on the third image; locating key points of the face based on the position of the face; the key points of the face include at least one of the following: eyes, mouth, head, and nose tip; extracting local shape and texture features of the key points of the face using a local binary pattern algorithm; calculating an affine transformation matrix based on the extracted local shape and texture features, the affine transformation matrix including translation and midpoint rotation angle; and performing portrait pose correction on the third image based on the affine transformation matrix formed by local feature extraction to obtain the fourth image.

[0087] In practice, quickly and accurately locating the face based on the third image is the first step in portrait pose correction. After locating the face, key facial features such as the eyes, mouth, head, and nose tip are then identified. The location of these key features is used for subsequent calculation of the affine transformation matrix. Specifically, the MTCNN neural network can be used to detect five key regions of the face, such as the eyes and mouth.

[0088] First, the LBP (Local Binary Patterns) algorithm is used to extract local shape and texture features from the detected eye and mouth regions. This method preserves the shape and texture features of important regions such as the eyes and mouth. The LBP algorithm formula is:

[0089]

[0090] in The position of the center pixel. The grayscale value of the center pixel. The grayscale value of the neighboring pixels. (x) is a binary function (0, 1).

[0091] Next, calculate the affine transformation matrix.

[0092] Let R be the low-light enhancement MFSR Let P(x,y) be a point in the interval (x,y). After an affine transformation, let P′(x′,y′) be the corresponding point. The general form of an affine transformation can be expressed as:

[0093] [ x ′ y ′ 1 ] = [ a b c d e f 0 0 1 ] [ x y 1 ]

[0094] Among them, matrix [ a b d e ] Represents the linear transformation part, including rotation and scaling; vector [ c f ] This indicates the translation.

[0095] If the image is rotated by an angle θ around the midpoint of the line connecting the two eyes, then the rotation matrix can be expressed as:

[0096] R ( θ ) = [ cos θ − sin θ sin θ cos θ ]

[0097] Considering the translation operation, the complete affine transformation matrix can be expressed as:

[0098] A = [ cos θ − sin θ t x sin θ cos θ t y 0 0 1 ]

[0099] Among them, t x and t y This indicates the amount of translation.

[0100] After obtaining the affine transformation matrix A, it can be applied to each pixel in the original image to obtain the corrected face image. Specifically, for a point P(x,y) in the original image, its corrected coordinates P′(x′, y′) can be calculated using the following formula:

[0101] [ x ′ y ′ 1 ] = A [ x y 1 ]

[0102] Right now:

[0103]

[0104] Finally, after determining the affine transformation matrix, bilinear interpolation is performed. Following the affine transformation, bilinear texture feature weight interpolation is applied to important regions such as the eyes and mouth. The coordinates to be interpolated (x, y) are determined, and four integer coordinate points around them are found: (i, j), (i+1, j+1), (i, j+1), and (i+1, j+1) for interpolation. For example, the horizontal interpolation calculation is as follows:

[0105]

[0106] in , γ is the pixel value of the intermediate point obtained by interpolation, γ is the weight determined based on the extracted texture features, and P is the pixel value of the four neighboring points.

[0107] The vertical direction can be interpolated in the same way.

[0108] Based on the affine transformation matrix formed by local feature extraction, the third image is subjected to portrait pose correction. This achieves effective correction of non-standard facial poses while preserving the features of important areas such as the eyes and mouth to the maximum extent, resulting in a well-corrected image that is beneficial for improving the accuracy of fatigue detection.

[0109] In another embodiment of the present invention, fatigue detection is performed based on key points of the face in the fourth image, including: determining key points of the eyes, mouth, and head based on the fourth image; determining the aspect ratio of the eyes based on the key points of the eyes; determining the mouth opening degree and mouth corner size based on the key points of the mouth; determining the vertical and horizontal degrees of the head based on the key points of the head; and comparing the changes in the eye length ratio, mouth opening degree, mouth corner size, and vertical and horizontal degrees of the head over a continuous time period with preset thresholds to comprehensively determine the fatigue level as one of the following: alert, moderate fatigue, or severe fatigue.

[0110] In practice, the fatigue detection algorithm may include the following steps:

[0111] 1. Capture images frame by frame after the attitude correction module.

[0112] 2. Capture face regions in images using convolutional neural network models. Specifically, a method combining the MTCNN face detection algorithm with the KCF (Kernelized Correlation Filters) tracking algorithm can be employed. In the initial frame detection, P-Net and R-Net layers are used to obtain high-confidence face candidate boxes. The coordinate information of these candidate boxes is then normalized and used as input to the KCF algorithm. The KCF algorithm quickly locates the high-confidence candidate boxes in new frames, and then the O-Net layer precisely locates the face.

[0113] The KCF algorithm can quickly track high-confidence candidate boxes after the initial frame by skipping the P-Net and R-Net layers. If the tracking is successful, the face is accurately located through the O-Net layer. If the tracking fails, the detection of the P-Net and R-Net layers is repeated.

[0114] 3. Fatigue testing process, such as Figure 6 As shown, the detection methods include blink frequency detection, yawn detection, head tilting during drowsiness detection, and head nodding frequency detection during drowsiness detection. The following explanation uses yawn detection as an example.

[0115] Figure 7 This is a diagram illustrating key points of the mouth, with points 49-68 being the most important. The degree of mouth opening / closing can be calculated using some of these key points, and then judged by the degree of mouth opening / closing and the size of the corners of the mouth. The calculation formula is as follows:

[0116]

[0117] The formula for calculating the angle between the corners of the mouth is:

[0118]

[0119]

[0120]

[0121] Where x and y are the coordinates of key points in the mouth area. The angle of the corners of the mouth when the mouth is open is greater than the angle when a person is speaking normally when yawning.

[0122] In practical implementation, a threshold for mouth opening, a threshold for the corner of the mouth angle, and a time threshold can be set. When MAR and If the yawn width exceeds a threshold and then recovers to below the threshold within a time threshold, it is considered a shallow yawn. If the yawn width recovers after exceeding the time threshold, it is considered a deep yawn. The formula for calculating yawn frequency is:

[0123]

[0124] in, To determine the proportion of frames (Yawn) in the total number of frames (N) over a period of time (e.g., 1 minute), determine if this proportion is greater than a certain threshold, such as 10%.

[0125] Blink frequency detection, head tilting detection during drowsiness detection, and head nodding frequency detection during drowsiness can be achieved using similar methods.

[0126] Blink detection uses the EAR algorithm to calculate the eye's length-to-width ratio (EAR). The calculation formula is as follows:

[0127]

[0128] Among them, p1 and p2 are key points of the eye.

[0129] The key points for the left eye are 37-42, and for the right eye, they are 43-48. The aspect ratios EAR_left and EAR_right for the left and right eyes are calculated separately. A weighted average is then used to obtain a more accurate aspect ratio. The calculation formula is as follows:

[0130] EAR= eye_left + eye_right

[0131] in , These represent the weights for eye_left and eye_right, respectively, with different weights selected based on individual differences. Generally, this is the case... = = 0.5, + = 1.

[0132] The blinking action of the human eye has the following characteristics: a complete blink involves a rapid change in the eye's aspect ratio from approximately 0.25 to 0.1 or less, and then back to approximately 0.25. When the eye is open, the aspect ratio remains around 0.25. Therefore, by observing the fluctuation of the eye's aspect ratio over a short period, one can determine whether the eye is fatigued.

[0133] The PERCLOS test can be used, and its calculation formula is as follows:

[0134]

[0135] Where `closed` represents the number of frames with eyes closed within a certain period, `N` represents the total number of frames in a short period (e.g., 1 minute), and `PERCLOS` represents the fatigue value. A fatigue state is determined when `PERCLOS` exceeds the fatigue threshold. For example, setting `EAR < 0.2` (the eye-closing threshold) indicates closed eyes. When the number of consecutive frames with `EAR < 0.2` exceeds 150, it is considered continuous eye-closing, denoted as `EYE_TRUE`.

[0136] In practical implementation, head fatigue feature detection can be performed by calculating the yaw (left-right) and pitch (up-down) degrees of some key points using the Efficient Perspective-n-Point (EPnP) algorithm. A large, continuous change in the pitch value within a short period is considered drowsy nodding. The formula for calculating the frequency of drowsy nodding is as follows:

[0137]

[0138] Where nod is the number of frames that match the drowsy nodding, and N is the total number of frames in a short period of time, such as less than 1 minute.

[0139] When the number of consecutive frames with yaw or pitch exceeding the threshold exceeds 150, it is considered that the person has not been looking forward for a long time, and is denoted as Nod_TRUE.

[0140] 4. Conduct fatigue testing by combining the three factors. The fatigue testing formula is as follows:

[0141] g = t1 + t2 + t3

[0142] in, These represent the frequencies of blinking, yawning, and drowsy nodding, respectively. i As weight, , g is the fatigue value.

[0143] Fatigue states are categorized into alertness, moderate fatigue, and severe fatigue, corresponding to conditions 2, 3, and 4, respectively. Table 1 provides an example of fatigue state assessment.

[0144]

[0145] First, determine whether condition 1 is met. If it is, then it is severe fatigue. If not, then determine conditions 2-4. (1) If the previous state is conscious, and conditions 2 and 6 are met, remain conscious; if conditions 3 and 5 are met, transition to moderate fatigue; if conditions 1 or (condition 4 and 7) are met, transition to severe fatigue.

[0146] (2) If the previous state is moderate fatigue, and conditions 3 and 6 are met, the moderate fatigue is maintained; if conditions 2 and 5 are met, the state becomes clear; if conditions 1 or (condition 4 and condition 7) are met, the state becomes severe fatigue.

[0147] (3) If the previous state was severe fatigue, condition 7 is met, and the state of severe fatigue is maintained; EYE_TRUE represents continuous eye closure, and Nod_TRUE represents prolonged periods without looking directly ahead. The time and other threshold parameters are not fixed values; they are all reference values.

[0148] The fatigue detection method for portrait enhancement and correction under low-light multi-angle conditions provided in this invention can perform image enhancement processing based on texture features extracted by adaptive texture analysis, thus improving the problems of insufficient low-light enhancement or detail feature extraction. Furthermore, by performing portrait pose correction based on the affine transformation matrix formed by local feature extraction before fatigue detection, it significantly improves the large fatigue detection deviation caused by asymmetric image detail loss due to image acquisition angle. Therefore, performing fatigue detection based on low-light image enhancement and portrait pose correction improves detection accuracy.

[0149] The following is a detailed description of the fatigue detection method for portrait enhancement and correction under low-light multi-angle conditions provided by the embodiments of the present invention through a specific example.

[0150] like Figure 8 As shown, the portrait enhancement and correction fatigue detection method provided by the embodiments of the present invention under low-light multi-angle conditions may include: S401, Obtain the first image; S402, based on the first image and a second image formed using an adaptive texture analysis method; The adaptive texture analysis method encodes features based on the local binary pattern of the window region. S403, based on the first image and the second image, perform image enhancement processing to obtain the third image; S404, Based on the affine transformation matrix formed by local feature extraction, the third image is subjected to portrait pose correction to obtain the fourth image; S405, Based on the fourth image, determine the key points of the eyes, mouth, and head; Specifically, based on the key points of the eyes, the length-to-width ratio of the eyes is determined; based on the key points of the mouth, the opening and closing degree of the mouth and the size of the corners of the mouth are determined; based on the key points of the head, the vertical and horizontal degrees of the head are determined. S406, based on the changes in eye length ratio, jaw opening and closing degree, mouth corner size, and head vertical and horizontal degrees during continuous connection time, compares them with set thresholds to comprehensively determine the fatigue level as one of the following: awake, moderate fatigue, or severe fatigue.

[0151] Secondly, embodiments of the present invention provide a fatigue detection method and apparatus for portrait enhancement and correction under low-light multi-angle conditions, which performs fatigue detection based on low-light image enhancement and portrait posture correction, thereby improving the detection accuracy.

[0152] like Figure 9 As shown, the portrait enhancement and correction fatigue detection device under low-light multi-angle conditions according to an embodiment of the present invention includes: Acquisition unit 51 is used to acquire the first image; The enhancement processing unit 52 is used to perform image enhancement processing based on the first image and the second image formed using an adaptive texture analysis method to obtain a third image; wherein, the adaptive texture analysis method is based on feature encoding of the local binary pattern of the window region; The posture correction unit 53 is used to perform portrait posture correction on the third image based on the affine transformation matrix formed by local feature extraction to obtain the fourth image. The fatigue detection unit 54 is used to perform fatigue detection based on the key points of the face in the fourth image.

[0153] The fatigue detection device for portrait enhancement and correction under low-light multi-angle conditions provided in this invention can perform image enhancement processing based on texture features extracted by adaptive texture analysis methods, improving the problems of insufficient low-light enhancement or detail feature extraction. Furthermore, by performing portrait pose correction based on the affine transformation matrix formed by local feature extraction before fatigue detection, it significantly improves the large fatigue detection deviation caused by asymmetric image detail loss due to image acquisition angle. Therefore, performing fatigue detection based on low-light image enhancement and portrait pose correction improves detection accuracy.

[0154] Thirdly, embodiments of the present invention also provide an electronic device capable of performing fatigue detection based on low-light image enhancement and portrait posture correction, thereby improving detection accuracy.

[0155] like Figure 10 As shown, the electronic device provided in the embodiments of the present invention may include: a housing 61, a processor 62, a memory 63, a circuit board 64, and a power supply circuit 65, wherein the circuit board 64 is disposed inside the space enclosed by the housing 61, and the processor 62 and the memory 63 are disposed on the circuit board 64; the power supply circuit 65 is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory 63 is used to store executable program code; the processor 62 runs a program corresponding to the executable program code by reading the executable program code stored in the memory 63, for executing the portrait enhancement correction fatigue detection method under low light multi-angle conditions provided in any of the foregoing embodiments.

[0156] For details on the specific execution process of the above steps by the processor 62 and the steps further executed by the processor 62 by running executable program code, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0157] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0158] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0159] In particular, the device embodiment is basically similar to the method embodiment, so the description is relatively simple. For relevant details, please refer to the description of the method embodiment.

[0160] For ease of description, the above apparatus is described by dividing it into various functional units / modules. Of course, in implementing this invention, the functions of each unit / module can be implemented in one or more software and / or hardware components.

[0161] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0162] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0163] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting fatigue in portrait enhancement and correction under low-light multi-angle conditions, characterized in that, include: Get the first image; Based on the first image and the second image formed using an adaptive texture analysis method, image enhancement processing is performed to obtain a third image; wherein, the adaptive texture analysis method performs feature encoding based on the local binary pattern of the window region; Based on the affine transformation matrix formed by local feature extraction, the third image is subjected to portrait pose correction to obtain the fourth image; Fatigue detection is performed based on key points of the face in the fourth image; The feature encoding based on the local binary pattern of the window region includes: Set the window size; The center point of the window is determined as the center pixel point; Based on the degree of feature fit between adjacent pixels and the central pixel, the adjacent pixels are divided into two regions: a first feature region and a second feature region. Based on the different regions, the values ​​of the pixels in the two regions are adjusted, and feature encoding is performed according to the adjustment results; The step of adjusting the pixel values ​​in the two regions based on their differences, and performing feature encoding based on the adjustment results, includes: Based on the two adjacent pixels of the first feature region of a pixel in the second feature region, the value of the pixel is differentiated. The average value of each pixel in the first feature region is taken to determine the trend value of the center pixel. Based on the difference between the value of each pixel in the first feature region and the tendency value of the center pixel, the value of each pixel in the first feature region is adjusted. The feature threshold of the center pixel is determined based on the average of the adjusted values ​​of each pixel in the first feature region. The value of a pixel in the first feature region after adjustment and the value of a pixel in the second feature region after differential processing are compared with the feature threshold. If the value is not less than the feature threshold of the center pixel, the feature code of the pixel is assigned a value of 1; otherwise, it is assigned a value of 0. The feature code of the center pixel is determined based on the feature code assignment of each pixel.

2. The method according to claim 1, characterized in that, The differential processing of the values ​​of two adjacent pixels in the first feature region based on a pixel in the second feature region includes: Based on the difference between the value of a pixel in the second feature region and the average value of the pixels in the two adjacent first feature regions, the value of the pixel is differentiated and adjusted.

3. The method according to claim 1, characterized in that, The process of assigning feature codes to each pixel to determine the feature code of the center pixel includes: The feature codes of each pixel in the first feature region and the second feature region are combined in sequence and then added together to determine the feature code of the center pixel.

4. The method according to claim 1, characterized in that, The process of enhancing the image based on the first image and the second image formed using an adaptive texture analysis method to obtain the third image includes: Set the number of scales and the weight corresponding to each scale; For each scale, a feature point in the first image and the corresponding point in the second image are multiplied by a Gaussian blur function to blur the image, and then the reflection component is calculated. The reflection components calculated at different scales are added together with weights to calculate the final enhanced image; Based on the final enhanced image, a third image is obtained.

5. The method according to claim 1, characterized in that, The affine transformation matrix formed based on local feature extraction is used to correct the human pose of the third image, resulting in a fourth image, including: Based on the third image, locate the position of the face; Based on the position of the face, key points of the face are located; the key points of the face include at least one of the following: eyes, mouth, head, and nose tip; local shape and texture features of the key points of the face are extracted using a local binary pattern algorithm; based on the extracted local shape and texture features, an affine transformation matrix is ​​calculated, the affine transformation matrix including translation and midpoint rotation angle; Based on the affine transformation matrix formed by local feature extraction, the third image is subjected to portrait pose correction to obtain the fourth image.

6. The method according to claim 1, characterized in that, The fatigue detection based on key points of the face in the fourth image includes: Based on the fourth image, key points of the eyes, mouth, and head are determined; Based on the aforementioned key eye points, the length-to-width ratio of the eye is determined; Based on the aforementioned key points of the mouth, determine the degree of mouth opening and the size of the corners of the mouth; Based on the aforementioned key points of the head, determine the vertical and horizontal degrees of the head; Based on the changes in the aspect ratio of the eyes, the degree of mouth opening and closing, the size of the corners of the mouth, and the degree of head tilting up and down and left and right over a continuous period of time, and compared with the set thresholds, the fatigue level is comprehensively determined as one of the following: alert, moderate fatigue, or severe fatigue.

7. A fatigue detection device for portrait enhancement and correction under low-light multi-angle conditions, characterized in that, include: The acquisition unit is used to acquire the first image; An enhancement processing unit is used to perform image enhancement processing based on the first image and a second image formed using an adaptive texture analysis method to obtain a third image; wherein, the adaptive texture analysis method is based on feature encoding of the local binary pattern of the window region; The pose correction unit is used to perform portrait pose correction on the third image based on the affine transformation matrix formed by local feature extraction to obtain the fourth image. The fatigue detection unit is used to perform fatigue detection based on key points of the face in the fourth image; The enhancement processing unit is used for: Set the window size; The center point of the window is determined as the center pixel point; Based on the degree of feature fit between adjacent pixels and the central pixel, the adjacent pixels are divided into two regions: a first feature region and a second feature region. Based on the different regions, the values ​​of the pixels in the two regions are adjusted, and feature encoding is performed according to the adjustment results; The enhancement processing unit is specifically used for: Based on the two adjacent pixels of the first feature region of a pixel in the second feature region, the value of the pixel is differentiated. The average value of each pixel in the first feature region is taken to determine the trend value of the center pixel. Based on the difference between the value of each pixel in the first feature region and the tendency value of the center pixel, the value of each pixel in the first feature region is adjusted. The feature threshold of the center pixel is determined based on the average of the adjusted values ​​of each pixel in the first feature region. The value of a pixel in the first feature region after adjustment and the value of a pixel in the second feature region after differential processing are compared with the feature threshold. If the value is not less than the feature threshold of the center pixel, the feature code of the pixel is assigned a value of 1; otherwise, it is assigned a value of 0. The feature code of the center pixel is determined based on the feature code assignment of each pixel.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is used to store computer-readable instructions; and the processor is used to read the computer-readable instructions and implement the portrait enhancement and correction fatigue detection method under low light multi-angle conditions as described in any one of claims 1 to 6.

Images