Fatigue detection method and device for human image enhancement and correction in low-light multi-angle state

Abstract

The embodiment of the invention discloses a portrait enhancement correction fatigue detection method and device in a low-light multi-angle state, relates to the technical field of safe driving, and can perform fatigue detection on the basis of low-light image enhancement and portrait posture correction so as to improve the detection accuracy. The method comprises the following steps: acquiring a first image; performing image enhancement processing based on the first image and a second image formed by using an adaptive texture analysis method to obtain a third image; according to the adaptive texture analysis method, feature coding is carried out based on a local binary pattern of a window region; carrying out portrait posture correction on the third image based on an affine transformation matrix formed by local feature extraction to obtain a fourth image; and performing fatigue detection based on the key points of the human face in the fourth image. The method can be used for human image enhancement and correction fatigue detection in a low-light multi-angle state.

Description

Technical Field

[0001] The present invention relates to the field of safe driving technology, and in particular to a method, device and electronic equipment for detecting fatigue by enhancing and correcting an image under low-light and multi-angle conditions. Background Art

[0002] In recent years, fatigue driving has become a major risk in traffic accidents, especially at night, where the proportion of accidents caused by fatigue driving has been increasing year by year. However, images captured in low-light environments often contain high levels of noise, blurred texture information, and inaccurate color reproduction, making subsequent image processing and fatigue detection difficult.

[0003] Currently, low-light image enhancement techniques include traditional histogram equalization and gamma transform algorithms, which introduce sampling blocks and convolution to refine features and enhance their expressiveness. However, when processing images in low-light conditions, due to insufficient illumination, high noise, and poor image clarity, the enhanced image features suffer from loss of detail, distortion, and excessive noise suppression. This impacts the subsequent fatigue detection process and reduces detection accuracy. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a method, device and electronic device for fatigue detection with portrait enhancement and correction in low-light and multi-angle conditions, which performs fatigue detection based on low-light image enhancement and portrait posture correction, thereby improving detection accuracy.

[0005] In the first aspect, an embodiment of the present invention provides a portrait enhancement, correction and fatigue detection method under low-light and multi-angle conditions, comprising: acquiring a first image; performing image enhancement processing based on the first image and a second image formed using an adaptive texture analysis method to obtain a third image; wherein the adaptive texture analysis method performs feature encoding based on the local binary pattern of the window area; performing portrait posture correction on the third image based on the affine transformation matrix formed by local feature extraction to obtain a fourth image; and performing fatigue detection based on the key points of the face in the fourth image.

[0006] In one embodiment, the feature encoding based on the local binary pattern of the window area includes: setting the size of the window; determining the center point of the window as the central pixel point; dividing the adjacent pixel points into two areas based on the degree of feature fit between the adjacent pixel points and the central pixel point: a first feature area and a second feature area; based on the difference in areas, adjusting the values of the pixel points in the two areas, and performing feature encoding according to the adjustment results.

[0007] In one embodiment, the values of the pixels in the two areas are adjusted based on the area differences, and feature encoding is performed according to the adjustment results, including: performing differential processing on the value of a pixel in the second feature area based on the pixel points of the two adjacent first feature areas; taking the average value of the pixel points in the first feature area to determine the trend value of the central pixel point; adjusting the value of each pixel in the first feature area based on the difference between the value of each pixel in the first feature area and the trend value of the central pixel point; determining the feature threshold of the central pixel point based on the average of the adjusted values of each pixel in the first feature area; comparing the value of a pixel in the first feature area after adjustment and the value of a pixel in the second feature area after differentiation with the feature threshold, if it is not less than the feature threshold of the central pixel point, assigning the feature code of the pixel point to 1, otherwise it is 0; determining the feature code of the central pixel point based on the feature code assignment of each pixel point.

[0008] In one embodiment, the pixel points of the two adjacent first feature areas based on a pixel point of the second feature area perform differential processing on the value of the pixel point, including: based on the difference between the value of a pixel point in the second feature area and the average value of the pixel points of the two adjacent first feature areas, perform differential processing on the value of the pixel point and adjust the value of the pixel point.

[0009] In one embodiment, the feature code assignment based on each pixel point to determine the feature code of the central pixel point includes: combining the feature codes of each pixel point in the first feature area and the second feature area in sequence and adding them together to determine the feature code of the central pixel point.

[0010] In one embodiment, the image enhancement processing is performed based on the first image and the second image formed using the adaptive texture analysis method to obtain a third image, including: setting the number of scales and the weight corresponding to each scale; for each scale, using a Gaussian blur function to multiply a feature point of the first image and the corresponding point of the second image, performing blur processing, and then calculating the reflection component; adding the reflection components calculated at different scales according to the weights to calculate a final enhanced image; and obtaining a third image based on the final enhanced image.

[0011] In one embodiment, the affine transformation matrix formed based on local feature extraction is used to correct the portrait posture of the third image to obtain a fourth image, including: locating the position of the face based on the third image; locating the key points of the face based on the position of the face; the key points of the face include at least one of the following: eyes, mouth, head, and nose tip; local shape and texture feature extraction is performed on the key points of the face through a local binary pattern algorithm; an affine transformation matrix is calculated based on the local shape and texture feature extraction, and the affine transformation matrix includes a translation amount and a midpoint rotation angle; based on the affine transformation matrix formed by local feature extraction, the third image is corrected for portrait posture to obtain a fourth image.

[0012] In one embodiment, the fatigue detection is performed based on the key points of the face in the fourth image, including: determining the key points of the eyes, mouth, and head based on the fourth image; determining the eye length-width ratio based on the eye key points; determining the mouth opening and closing degree and the size of the mouth corners based on the mouth key points; determining the up and down, left and right degrees of the head based on the head key points; based on the changes in the eye length ratio, jaw opening and closing degree and the size of the mouth corners, as well as the up and down, left and right degrees of the head during the continuous connection time, respectively comparing with the set thresholds, and comprehensively determining the fatigue level as one of the following: awake, moderate fatigue, severe fatigue.

[0013] In the second aspect, an embodiment of the present invention also provides a portrait enhancement, correction and fatigue detection device in low-light and multi-angle states, including: an acquisition unit for acquiring a first image; an enhancement processing unit for performing image enhancement processing based on the first image and a second image formed using an adaptive texture analysis method to obtain a third image; wherein the adaptive texture analysis method performs feature encoding based on the local binary pattern of the window area; a posture correction unit for performing portrait posture correction on the third image based on the affine transformation matrix formed by local feature extraction to obtain a fourth image; and a fatigue detection unit for performing fatigue detection based on the key points of the face in the fourth image.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device comprising: a memory and a processor, wherein the memory is used to store computer-readable instructions; the processor is used to read the computer-readable instructions and implement the portrait enhancement and correction fatigue detection method in low-light multi-angle conditions as described in any one of claims 1 to 8.

[0015] The portrait enhancement and correction fatigue detection method, device, and electronic device provided by the embodiments of the present invention in low-light and multi-angle states can perform image enhancement processing on the basis of extracting texture features using an adaptive texture analysis method, thereby improving the problem of insufficient low-light enhancement or detail feature extraction. Moreover, based on the affine transformation matrix formed by local feature extraction, fatigue detection is performed after portrait posture correction, which greatly improves the large deviation in fatigue detection caused by problems such as asymmetric image detail loss due to the image acquisition angle. Therefore, fatigue detection is performed on the basis of low-light image enhancement and portrait posture correction, thereby improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 The schematic diagram of the image enhancement principle of the prior art is provided;

[0018] Figure 2 A flowchart of a method for detecting fatigue in low-light and multi-angle portrait enhancement and correction provided by an embodiment of the present invention;

[0019] Figure 3 Schematic diagram of window division areas in an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of feature coding in an embodiment of the present invention;

[0021] Figure 5 A diagram illustrating an image enhancement principle provided in an embodiment of the present invention;

[0022] Figure 6 A fatigue detection flow chart provided in an embodiment of the present invention;

[0023] Figure 7 A schematic diagram of key points of the mouth provided in an embodiment of the present invention;

[0024] Figure 8 A flowchart of a method for detecting fatigue in low-light and multi-angle portrait enhancement and correction provided by an embodiment of the present invention;

[0025] Figure 9 A schematic structural diagram of a device for detecting fatigue by enhancing and correcting portraits in low-light and multi-angle conditions according to an embodiment of the present invention;

[0026] Figure 10A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0028] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0029] In a first aspect, an embodiment of the present invention provides a method for fatigue detection by portrait enhancement and correction in low-light and multi-angle conditions, which can perform fatigue detection based on low-light image enhancement and portrait posture correction, thereby improving detection accuracy.

[0030] like Figure 2 As shown, an embodiment of the present invention provides a method for detecting fatigue by enhancing and correcting portraits in low-light and multi-angle conditions, comprising:

[0031] S11, acquiring a first image.

[0032] In this step, images for fatigue detection are obtained. The images used for fatigue detection can be taken from captured pictures, or they can be several pictures formed by capturing certain frames from a recorded video. In actual operation, the time of multiple consecutive images in the video can also be recorded. In this way, the changes in the images within a certain time period are calculated by taking multiple consecutive images and the elapsed time, and compared with the set threshold to determine the fatigue level. The images used for fatigue detection can be pictures in different formats, such as: bmp, jpg, png, tif, gif, etc.

[0033] It should be noted that the quality of the image acquired for fatigue detection affects the accuracy of fatigue detection. In order to improve the detection accuracy, the acquired image needs to be processed in the subsequent steps.

[0034] S12, performing image enhancement processing based on the first image and the second image formed using an adaptive texture analysis method to obtain a third image; wherein the adaptive texture analysis method performs feature encoding based on the local binary pattern of the window area.

[0035] After acquiring the first image, this step first processes it using an adaptive texture analysis method. Adaptive texture analysis can extract key image features, such as edges and textures. This information is crucial for subsequent image enhancement and can effectively enhance images in low-light conditions.

[0036] In this step, the adaptive texture analysis method performs feature encoding based on the local binary pattern of the window area. In a specific implementation, the center point of the window can be determined as the central pixel, and the adjacent pixels can be divided into multiple regions based on the degree of feature alignment between the adjacent pixels and the central pixel. Based on the differences in the regions, the values of the pixels within the different regions are adjusted, and feature encoding is performed based on the adjusted results. Using the adaptive texture analysis method, key texture information is extracted from the first image to form a second image.

[0037] To reduce the impact of low-light conditions, we conduct effective image enhancement based on adaptive texture analysis. Low-light image enhancement can be achieved using traditional histogram equalization, gamma transform, or multi-scale Retinex algorithms.

[0038] The present application performs image addition processing based on the first image and the second image formed using an adaptive texture analysis method to obtain a third image.

[0039] S13, performing portrait posture correction on the third image based on the affine transformation matrix formed by local feature extraction to obtain a fourth image.

[0040] In this embodiment of the present invention, the third image is subjected to posture correction based on an affine transformation matrix formed by local feature extraction to produce a fourth image. Non-orthogonal image acquisition is prone to missing details in some feature areas, resulting in reduced detection accuracy. In this step, affine transformation technology based on local feature extraction is used to correct the posture of the enhanced image, facilitating feature point identification for fatigue detection in the next step.

[0041] In a specific implementation, face detection is first performed to quickly and accurately locate the face in the third image. After detecting the face, key points of the face, such as the corners of the eyes, the corners of the mouth, and the tip of the nose, need to be located. The positional changes of these key points can be used to calculate the affine transformation matrix. Finally, using the affine transformation matrix, each pixel in the third image can be transformed into a corrected face image, resulting in the fourth image.

[0042] S14: Perform fatigue detection based on key points of the face in the fourth image.

[0043] In this step, fatigue detection is performed based on key points of the face in the fourth image. In a specific implementation, key points such as the eyes, mouth, and head can be determined first, and the state of these key points or their changes can be used to comprehensively determine whether the person is in a fatigue state or a fatigue level, such as alert, moderate fatigue, or severe fatigue.

[0044] The portrait enhancement and correction fatigue detection method under low-light and multi-angle conditions provided by the embodiment of the present invention can perform image enhancement processing on the basis of extracting texture features using an adaptive texture analysis method, thereby improving the problem of insufficient low-light enhancement or detail feature extraction. Moreover, based on the affine transformation matrix formed by local feature extraction, fatigue detection is performed after portrait posture correction, which greatly improves the large deviation in fatigue detection caused by problems such as asymmetric image detail loss due to the image acquisition angle. Therefore, fatigue detection is performed on the basis of low-light image enhancement and portrait posture correction, thereby improving the detection accuracy.

[0045] Specifically, in one embodiment of the present invention, feature encoding can be performed based on the local binary pattern of the window area in step S12, including: setting the size of the window; determining the center point of the window as the central pixel point; dividing the adjacent pixel points into two areas based on the degree of feature fit between the adjacent pixel points and the central pixel point: a first feature area and a second feature area; based on the difference in areas, adjusting the values of the pixel points in the two areas, and performing feature encoding according to the adjustment results.

[0046] Specifically, before performing feature encoding based on the local binary pattern of the window area, average pooling preprocessing can be performed. By first performing the first step of average pooling on the low-light image, image noise is initially suppressed.

[0047] In practice, the pooling window is typically 2x2 or 3x3, with a stride of 1 or 2. When the stride is 1, the pooling window moves one pixel at a time across the input feature map; when the stride is 2, the pooling window moves two pixels at a time. Within the pooling window, the values of all pixels are summed and then divided by the number of pixels to obtain the average value, which is the pooling result.

[0048] Perform average pooling on the first image. Average pooling reduces the dimensionality of the data by reducing the image resolution, thereby reducing computational effort and storage requirements. By calculating the average value within a region, average pooling can integrate local feature information within the region, helping to extract more global features. Because noise is typically randomly distributed, the average value can smooth out these random fluctuations, providing a preliminary suppression of image noise.

[0049] When encoding features based on the local binary pattern of a window area, the window size must first be set. The window size can be set to either a square or a circle. This invention does not impose any limitations on this. The following description uses a 3x3 square window as an example.

[0050] like Figure 3 As shown, in the 3X3 square window, the center point G of the window tA central pixel is determined, and based on the degree of feature fit between the adjacent pixels and the central pixel, the adjacent pixels are divided into two regions: Region M and Region N. Region M has four pixels: M1, M2, M3, and M4; Region N also has four pixels: N1, N2, N3, and N4.

[0051] It should be noted that if Figure 3 In the example shown, the adjacent pixels that have adjacent edges with the central pixel are grouped as zone M, and the remaining pixels are grouped as zone N. In actual implementation, the division of zone M and zone N is determined based on the relationship between the degree of feature fit between the adjacent pixels and the central pixel in actual conditions. For example, in a circular window, the adjacent pixels that are orthogonal to the central pixel are grouped as zone M, and the remaining pixels are grouped as zone N. In addition, in the subsequent calculations involved in the division of zone M and zone N, the two may be exchanged according to actual conditions, that is, in the current example, zone M is used as the reference point and zone N is used as the adjustment point, while in different actual conditions, zone N is used as the reference point and zone M is used as the adjustment point.

[0052] Based on the different window areas, the values of the pixels within the two areas can be adjusted, and feature encoding can be performed based on the adjustment results. By dividing the adjacent pixels into two areas: a first feature area and a second feature area, based on the degree of feature fit between the adjacent pixels and the central pixel, and adjusting the pixel values based on the different areas, it can solve the problem that the traditional local binary pattern algorithm over-discretely encodes adjacent pixels, lacks neighborhood encoding capabilities, and causes some detailed features to be lost.

[0053] In another embodiment, the values of the pixel points in the two areas are adjusted, and feature encoding is performed according to the adjustment results, including: performing differential processing on the value of a pixel point based on the pixel points of the two adjacent first feature areas of a pixel point in the second feature area; taking the average value of the pixel points in the first feature area to determine the trend value of the central pixel point; adjusting the value of each pixel point in the first feature area based on the difference between the value of each pixel point in the first feature area and the trend value of the central pixel point; determining the feature threshold of the central pixel point based on the average of the adjusted values of each pixel point in the first feature area; comparing the value of a pixel point after adjustment in the first feature area and the value of a pixel point after differentiation in the second feature area with the feature threshold, and if it is not less than the feature threshold of the central pixel point, assigning the feature code of the pixel point to 1, otherwise it is 0; determining the feature code of the central pixel point based on the feature code assignment of each pixel point.

[0054] like Figure 4 As shown, the values of the pixels in the two regions are adjusted, and feature encoding is performed according to the adjustment results. Specifically, the following steps are included:

[0055] S31 . Perform differentiation processing on the value of a pixel point in the second feature area based on the pixel points in two adjacent first feature areas of the pixel point in the second feature area.

[0056] Specifically, the feature differentiation processing of the pixels in the N regions can be performed using the following calculation formula:

[0057]

[0058] S32: averaging the values of the pixels in the first feature area to determine the trend value of the central pixel.

[0059] Specifically, the values of the pixels in the M region are averaged to determine the trend value Avg of the central pixel Gt. The following calculation formula can be used:

[0060]

[0061] S33 . Adjust the value of each pixel in the first feature area based on the difference between the value of each pixel in the first feature area and the trend value of the central pixel.

[0062] Specifically, the difference between the value of each pixel in the M area and the trend value of the central pixel is used to adjust the value of each pixel in the first feature area. The following calculation formula can be used:

[0063]

[0064] S34. Determine a feature threshold of the central pixel based on an average of the adjustment values of the pixels in the first feature area.

[0065] Specifically, the characteristic threshold Ln of the central pixel is determined by taking the average of the adjusted values of the pixels in the M region. The following calculation formula can be used:

[0066]

[0067] S35. Compare the value of a pixel point after adjustment in the first feature area and the value of a pixel point after differential processing in the second feature area with the feature threshold respectively. If they are not less than the feature threshold of the central pixel point, assign the feature code of the pixel point to 1, otherwise it is 0.

[0068] Specifically, the threshold activation function is used for the M area and the N area respectively: if it is not less than the feature threshold of the central pixel, the feature code of the pixel is assigned to 1, otherwise it is 0. The threshold activation function of the M area and the N area can be calculated using the following formula:

[0069]

[0070]

[0071] S36. Determine the feature code of the central pixel based on the feature code assignment of each pixel.

[0072] Specifically, the feature code of each pixel point can be assigned according to different areas in the window to determine the feature code LBP of the central pixel point. The feature codes of each pixel point in the first feature area and the second feature area can be combined in sequence and then connected to determine the feature code of the central pixel point. Alternatively, the following calculation formula can be used:

[0073]

[0074] Through the above encoding process, the discrete features in traditional feature extraction are continuous, the light and dark relationship between the pixel points and the surrounding neighborhood is retained, and the texture feature extraction of the face in low-light environments is greatly restored with good robustness.

[0075] In another embodiment of the present invention, the value of a pixel point in the second feature area is differentiated based on the pixel points of the two adjacent first feature areas, including: based on the difference between the value of a pixel point in the second feature area and the average value of the pixel points of the two adjacent first feature areas, the value of the pixel point is differentiated and the value of the pixel point is adjusted.

[0076] In another embodiment of the present invention, the feature code of the central pixel point is determined based on the feature code assignment of each pixel point, including: combining the feature codes of each pixel point in the first feature area and the second feature area in sequence and adding them together to determine the feature code of the central pixel point.

[0077] In another embodiment of the present invention, image enhancement processing is performed based on the first image and the second image formed using an adaptive texture analysis method to obtain a third image, including: setting the number of scales and the weight corresponding to each scale; for each scale, using a Gaussian blur function to multiply a feature point of the first image and the corresponding point of the second image, performing blur processing, and then calculating the reflection component; adding the reflection components calculated at different scales according to the weights to calculate the final enhanced image; and obtaining the third image based on the final enhanced image.

[0078] In a specific implementation, multi-scale Retinex processing is performed on any feature point I(x, y) in the first image and the corresponding feature point F(x, y) in the second image. Through multi-scale Retinex processing, the illumination component L and the reflection component R of the image can be separated at different scales. Figure 1 As shown in the figure, the basic idea of Retinex theory is that the intensity of illumination determines the dynamic range of all pixels in the original image, while the inherent properties of the original image are determined by the reflection coefficient of the object itself. That is, it is assumed that the original image is the product of the reflection image and the illumination image. Figure 5 As shown, the present invention performs multi-scale Retinex processing based on feature texture extraction.

[0079] Specifically, for each scale k, the Gaussian blur function G is used k (x,y) pairs Perform blur processing and then calculate the reflection component. The mathematical expression can be:

[0080] R k ( x , y ) = log [ I ( x , y ) ⋅ F ( x , y ) ] − log [ G k ( x , y ) ⋅ I ( x , y ) ⋅ F ( x , y ) ]

[0081] Among them, R k (x,y) represents the reflection component calculated at the kth scale.

[0082] Finally, the reflection components R calculated at different scales are k (x,y) by weight ω k Add together to get the final enhanced image R MFSR ( x,y ) The mathematical expression that can be used is:

[0083]

[0084] Where N represents the number of scales, ω K represents the weight of the k-th scale.

[0085] Image enhancement processing is performed based on the first image and a second image formed using an adaptive texture analysis method. This combines the advantages of multi-scale Retinex and feature extraction to effectively enhance images in low-light conditions. Through appropriate feature extraction, multi-scale Retinex processing, and reflectance component fusion, fine-grained image enhancement is achieved, improving both visual quality and recognition performance.

[0086] In another embodiment of the present invention, the third image is corrected for portrait posture based on the affine transformation matrix formed by local feature extraction to obtain a fourth image, including: locating the position of the face based on the third image; locating the key points of the face based on the position of the face; the key points of the face include at least one of the following: eyes, mouth, head, and nose tip; performing local shape and texture feature extraction on the key points of the face through a local binary pattern algorithm; calculating an affine transformation matrix based on the local shape and texture feature extraction, the affine transformation matrix including a translation amount and a midpoint rotation angle; performing portrait posture correction on the third image based on the affine transformation matrix formed by local feature extraction to obtain a fourth image.

[0087] In specific implementations, quickly and accurately locating the face based on the third image is the first step in portrait pose correction. After locating the face, key facial points, such as the eyes, mouth, head, and nose tip, are located. These key points are used in the subsequent calculation of the affine transformation matrix. In specific implementations, the MTCNN neural network can be used to detect five key facial regions, such as the eyes and mouth.

[0088] First, the LBP (Local Binary Patterns) algorithm is used to extract the local shape and texture features of the detected eye and mouth areas. This method preserves the shape and texture features of important areas such as the eyes and mouth. The LBP algorithm formula is:

[0089]

[0090] in is the center pixel position, is the gray value of the center pixel, is the gray value of the neighborhood pixel, (x) is a binary function (0, 1).

[0091] Next, calculate the affine transformation matrix.

[0092] Set R after low light enhancement MFSR A point in (x, y) is P(x, y), and the corresponding point after affine transformation is P′(x′, y′). The general form of affine transformation can be expressed as:

[0093] [ x ′ y ′ 1 ] = [ a b c d e f 0 0 1 ] [ x y 1 ]

[0094] Among them, the matrix [ a b d e ] Represents the linear transformation part, including rotation and scaling; vector [ c f ] Represents the translation part.

[0095] The image is rotated by an angle θ around the midpoint of the line connecting the two eyes, and the rotation matrix can be expressed as:

[0096] R ( θ ) = [ cos θ − sin θ sin θ cos θ ]

[0097] Taking into account the translation operation, the complete affine transformation matrix can be expressed as:

[0098] A = [ cos θ − sin θ t x sin θ cos θ t y 0 0 1 ]

[0099] Among them, t x and t y Indicates the amount of translation.

[0100] After obtaining the affine transformation matrix A, it can be applied to each pixel in the original image to obtain the corrected face image. Specifically, for a point P (x, y) in the original image, its corrected coordinates P′ (x′, y′) can be calculated using the following formula:

[0101] [ x ′ y ′ 1 ] = A [ x y 1 ]

[0102] Right now:

[0103]

[0104] Finally, after determining the affine transformation matrix, bilinear interpolation is performed. After the affine transformation, bilinear texture feature weight interpolation is performed on important areas such as the eyes and mouth. The coordinates to be interpolated (x, y) are determined, and four integer coordinate points are found around them: (i, j), (i+1, j+1), (i, j+1), (i+1, j+1) for interpolation. For example, the horizontal interpolation calculation is as follows:

[0105]

[0106] in , is the pixel value of the middle point obtained by interpolation, γ is the weight determined according to the extracted texture features, and P is the pixel value of the four adjacent points.

[0107] The vertical direction can be interpolated in the same way.

[0108] Based on the affine transformation matrix formed by local feature extraction, the portrait posture correction is performed on the third image, which effectively corrects the non-standard posture of the face while maximally retaining the features of important areas such as the eyes and mouth, so that the corrected image has a good effect, which is conducive to improving the accuracy of fatigue detection.

[0109] In another embodiment of the present invention, fatigue detection is performed based on the key points of the face in the fourth image, including: determining the key points of the eyes, mouth, and head based on the fourth image; determining the eye length-width ratio based on the eye key points; determining the mouth opening and closing degree and the size of the mouth corners based on the mouth key points; determining the up and down, left and right degrees of the head based on the head key points; based on the changes in the eye length ratio, jaw opening and closing degree and the size of the mouth corners, as well as the up and down, left and right degrees of the head during the continuous connection time, respectively comparing with the set thresholds, and comprehensively determining the fatigue level as one of the following: awake, moderate fatigue, severe fatigue.

[0110] In a specific implementation, the fatigue detection algorithm may include the following steps:

[0111] 1. Capture the image after the posture correction module frame by frame.

[0112] 2. Use a convolutional neural network model to capture the face region in the image. Specifically, a method that combines the MTCNN face detection algorithm with the KCF (Kernelized Correlation Filters) tracking algorithm can be used. In the initial frame, detection is performed through the P-Net and R-Net layers to obtain high-confidence face candidate frames. The coordinate information of these candidate frames is then normalized and fed as input to the KCF algorithm. The KCF algorithm quickly locates the high-confidence candidate frames in the new frame, and then accurately locates the face through the O-Net layer.

[0113] The KCF algorithm can skip the P-Net and R-Net layers after the initial frame to quickly track the candidate frame with high confidence. If the tracking is successful, the O-Net layer is used to accurately locate the face. If the tracking fails, the P-Net and R-Net layers are re-detected.

[0114] 3. Fatigue detection process, such as Figure 6 As shown in the figure, it specifically includes blink frequency detection, yawn detection, head tilt detection when dozing off, and head nodding frequency detection when dozing off. The following uses yawn detection as an example for explanation.

[0115] Figure 7 The diagram below shows the key points of the mouth, where the key points are 49-68. Yawning can be determined by calculating the degree of mouth opening and closing using some key points of the mouth, and then judging by the degree of mouth opening and closing and the size of the mouth corners. The calculation formula is:

[0116]

[0117] The formula for calculating the mouth angle is:

[0118]

[0119]

[0120]

[0121] Where x, y are the coordinates of the key points of the mouth, It is the angle at the corners of the mouth when it is open. When a person yawns normally, the degree of opening and closing of the mouth and the angle of the corners of the mouth are greater than when speaking normally.

[0122] In the specific implementation, a mouth opening threshold, a mouth angle threshold and a time threshold can be set. After the opening is greater than the threshold, it is considered a shallow yawn if it returns to less than the threshold within the time threshold. When the opening is restored after exceeding the time threshold, it is considered a deep yawn. The formula for calculating the yawn frequency is:

[0123]

[0124] in, The ratio of the number of yawning frames (Yawn) to the total number of frames (N) in a period of time (for example, 1 minute) is used to determine whether the ratio is greater than a threshold. For example, the threshold can be set to 10%.

[0125] Blink frequency detection, head tilt detection when dozing off, and head nodding frequency detection when dozing off can be achieved using similar methods.

[0126] Blink detection uses the EAR algorithm to calculate the eye aspect ratio EAR. The calculation formula is:

[0127]

[0128] Among them, p1, p2, etc. are key points of the eyes.

[0129] The key points of the left eye are 37-42, and the key points of the right eye are 43-48. The aspect ratios of the left eye and right eye are calculated as EAR_left and EAR_right respectively. A more accurate aspect ratio is obtained by weighted averaging. The calculation formula is as follows:

[0130] EAR= eye_left + eye_right

[0131] in 、 They are the weights of eye_left and eye_right respectively. Different weights are selected according to individual differences. Usually = = 0.5, + = 1.

[0132] The human eye blink has the following characteristics: a complete blink involves a rapid change in the eye's aspect ratio from approximately 0.25 to 0.1 (or less) to approximately 0.25. While open, the ratio remains around 0.25. Therefore, fatigue can be determined by observing the fluctuation of the EAR over a short period of time.

[0133] The PERCLOS judgment method can be used, and its calculation formula is:

[0134]

[0135] Where closed is the number of frames with eyes closed during a period, N is the total number of frames in a short period (e.g., 1 minute), and PERCLOS is the fatigue value. When PERCLOS exceeds the fatigue threshold, fatigue is considered. For example, setting EAR < 0.2 (the eye closure threshold) indicates eye closure. If the number of consecutive frames with EAR < 0.2 exceeds 150, it is considered continuous eye closure and is recorded as EYE_TRUE.

[0136] In practice, head fatigue feature detection can use the EPnP (Efficient Perspective-n-Point) algorithm to calculate the Yaw (left and right) and Pitch (up and down) degrees using certain key points. Continuous and large changes in the Pitch value within a short period of time are considered sleepy nodding. The formula for calculating sleepy nodding frequency is:

[0137]

[0138] Where nod is the number of frames that match the nodding pattern, and N is the total number of frames in a short period of time, for example, less than 1 minute.

[0139] When the number of consecutive frames with Yaw or Pitch greater than the threshold exceeds 150 frames, it is considered that the eyes have not been looking forward for a long time and is recorded as Nod_TRUE.

[0140] 4. Combine the three factors to perform fatigue detection. The fatigue detection formula is as follows:

[0141] g = t1 + t2 + t3

[0142] in, are the frequencies of blinking, yawning, and nodding while dozing off, t i is the weight, , g is the fatigue value.

[0143] Fatigue states are divided into awake, moderately fatigued, and severely fatigued, corresponding to conditions 2, 3, and 4, respectively. Table 1 shows an example of fatigue state judgment.

[0144]

[0145] First, determine whether condition 1 is met. If so, it is severe fatigue. If not, determine conditions 2-4. (1) If the previous state is awake, and condition 2 and condition 6 are met, then the state remains awake; if condition 3 and condition 5 are met, then the state changes to moderate fatigue; if condition 1 or (condition 4 and condition 7) are met, then the state changes to severe fatigue.

[0146] (2) If the previous state is moderate fatigue, and condition 3 and condition 6 are met, the state remains moderately fatigued; if condition 2 and condition 5 are met, the state becomes awake; if condition 1 or (condition 4 and condition 7) are met, the state becomes severely fatigued.

[0147] (3) If the previous state is severe fatigue, condition 7 is met and severe fatigue is maintained; EYE_TRUE indicates continuous eye closure, and NOD_TRUE indicates a prolonged period of not looking straight ahead. Threshold parameters such as time are not fixed and are provided as reference values.

[0148] The portrait enhancement and correction fatigue detection method under low-light and multi-angle conditions provided by the embodiment of the present invention can perform image enhancement processing on the basis of extracting texture features using an adaptive texture analysis method, thereby improving the problem of insufficient low-light enhancement or detail feature extraction. Moreover, based on the affine transformation matrix formed by local feature extraction, fatigue detection is performed after portrait posture correction, which greatly improves the large deviation in fatigue detection caused by problems such as asymmetric image detail loss due to the image acquisition angle. Therefore, fatigue detection is performed on the basis of low-light image enhancement and portrait posture correction, thereby improving the detection accuracy.

[0149] The following describes in detail the method for detecting fatigue in low-light and multi-angle portrait enhancement correction provided by an embodiment of the present invention through a specific embodiment.

[0150] like Figure 8 As shown, the method for detecting fatigue by enhancing and correcting portrait fatigue in low-light and multi-angle conditions provided by an embodiment of the present invention may include: S401, acquiring a first image; S402, forming a second image based on the first image and using an adaptive texture analysis method; Wherein, the adaptive texture analysis method performs feature encoding based on the local binary pattern of the window area; S403, performing image enhancement processing based on the first image and the second image to obtain a third image; S404, performing portrait posture correction on the third image based on the affine transformation matrix formed by local feature extraction to obtain a fourth image; S405, determining key points of eyes, mouth, and head based on the fourth image; The eye length-width ratio is determined based on the eye key points; the mouth opening and closing degree and the size of the mouth corners are determined based on the mouth key points; and the head up and down and left and right degrees are determined based on the head key points. S406, based on the changes in the eye length ratio, jaw opening and closing degree, mouth corner size, and head up and down, left and right degrees during the continuous connection time, respectively, compared with the set thresholds, the fatigue level is comprehensively determined as one of the following: awake, moderate fatigue, severe fatigue.

[0151] In a second aspect, an embodiment of the present invention provides a method and device for portrait enhancement and correction fatigue detection in low-light and multi-angle conditions, which performs fatigue detection based on low-light image enhancement and portrait posture correction, thereby improving detection accuracy.

[0152] like Figure 9 As shown, the embodiment of the present invention provides a device for detecting fatigue by enhancing and correcting portraits in low-light and multi-angle conditions, including: An acquisition unit 51 is configured to acquire a first image; an enhancement processing unit 52 configured to perform image enhancement processing based on the first image and a second image formed using an adaptive texture analysis method to obtain a third image; wherein the adaptive texture analysis method performs feature encoding based on a local binary pattern of a window area; a posture correction unit 53 for performing portrait posture correction on the third image based on an affine transformation matrix formed by extracting local features to obtain a fourth image; The fatigue detection unit 54 is configured to perform fatigue detection based on key points of the face in the fourth image.

[0153] The portrait enhancement and correction fatigue detection device provided by the embodiment of the present invention in low-light multi-angle conditions can perform image enhancement processing on the basis of extracting texture features using an adaptive texture analysis method, thereby improving the problem of insufficient low-light enhancement or detail feature extraction. Moreover, based on the affine transformation matrix formed by local feature extraction, fatigue detection is performed after portrait posture correction, which greatly improves the large deviation in fatigue detection caused by problems such as asymmetric image detail loss due to the image acquisition angle. Therefore, fatigue detection is performed on the basis of low-light image enhancement and portrait posture correction, thereby improving the detection accuracy.

[0154] In a third aspect, an embodiment of the present invention further provides an electronic device capable of performing fatigue detection based on low-light image enhancement and portrait posture correction, thereby improving detection accuracy.

[0155] like Figure 10 As shown, the electronic device provided by an embodiment of the present invention may include: a shell 61, a processor 62, a memory 63, a circuit board 64 and a power supply circuit 65, wherein the circuit board 64 is placed inside the space enclosed by the shell 61, and the processor 62 and the memory 63 are arranged on the circuit board 64; the power supply circuit 65 is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory 63 is used to store executable program code; the processor 62 runs the program corresponding to the executable program code by reading the executable program code stored in the memory 63, so as to execute the portrait enhancement correction fatigue detection method in low-light multi-angle state provided in any of the aforementioned embodiments.

[0156] The specific execution process of the above steps by the processor 62 and the steps further executed by the processor 62 by running the executable program code can be found in the description of the above embodiment and will not be repeated here.

[0157] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0158] Each embodiment in this specification is described in a related manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0159] In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0160] For the convenience of description, the above device is described as being divided into various units / modules based on their functions. Of course, when implementing the present invention, the functions of each unit / module can be implemented in the same or more than two software and / or hardware.

[0161] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0162] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

[0163] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for detecting fatigue by enhancing and correcting portraits in low-light and multi-angle conditions, characterized in that: include: acquiring a first image; Performing image enhancement processing based on the first image and a second image formed using an adaptive texture analysis method to obtain a third image; wherein the adaptive texture analysis method performs feature encoding based on a local binary pattern of a window area; Performing portrait posture correction on the third image based on an affine transformation matrix formed by local feature extraction to obtain a fourth image; Fatigue detection is performed based on key points of the face in the fourth image.

2. The method according to claim 1, characterized in that The feature encoding based on the local binary pattern of the window area includes: Set the size of the window; Determine the center point of the window as the center pixel point; Based on the degree of feature fit between the adjacent pixels and the central pixel, the adjacent pixels are divided into two regions: a first feature region and a second feature region; Based on the difference in regions, the values of the pixels in the two regions are adjusted, and feature encoding is performed according to the adjustment results.

3. The method according to claim 2, characterized in that The adjusting the values of the pixels in the two regions based on the difference in regions, and performing feature coding according to the adjustment results, includes: performing differential processing on a value of a pixel point in the second feature area based on two adjacent pixel points in the first feature area; averaging the values of the pixels in the first feature area to determine the trend value of the central pixel; Adjusting the value of each pixel in the first feature area based on the difference between the value of each pixel in the first feature area and the trend value of the central pixel; determining a characteristic threshold of the central pixel based on an average of the adjusted values of the pixels in the first characteristic area; Comparing the value of a pixel point after adjustment in the first feature area and the value of a pixel point after differentiation processing in the second feature area with the feature threshold, respectively, if the value is not less than the feature threshold of the central pixel point, assigning the feature code of the pixel point to 1, otherwise to 0; Based on the feature code assignment of each pixel point, the feature code of the central pixel point is determined.

4. The method according to claim 3, characterized in that The performing differential processing on the value of a pixel point of two adjacent pixels of the first feature area based on a pixel point of the second feature area includes: Based on the difference between the value of a pixel point in the second feature area and the average value of the pixels in two adjacent first feature areas, differentiation processing is performed on the value of the pixel point to adjust the value of the pixel point.

5. The method according to claim 3, characterized in that The determining the feature code of the central pixel based on the feature code assignment of each pixel point includes: The feature codes of the pixels in the first feature area and the second feature area are sequentially combined and then added to determine the feature code of the central pixel.

6. The method according to claim 1, characterized in that The step of performing image enhancement processing based on the first image and the second image formed using the adaptive texture analysis method to obtain a third image includes: Set the number of scales and the weight corresponding to each scale; For each scale, a Gaussian blur function is used to multiply a feature point of the first image and a corresponding point of the second image, and then blur the result, and then calculate the reflection component; Adding the reflection components calculated at different scales according to weights to calculate a final enhanced image; A third image is obtained based on the final enhanced image.

7. The method according to claim 1, characterized in that The affine transformation matrix formed based on the local feature extraction is used to perform portrait posture correction on the third image to obtain a fourth image, including: locating a position of a human face based on the third image; Based on the position of the face, locate key points of the face; the key points of the face include at least one of the following: eyes, mouth, head, and nose tip; extract local shape and texture features of the key points of the face using a local binary pattern algorithm; calculate an affine transformation matrix based on the local shape and texture feature extraction, the affine transformation matrix including a translation amount and a midpoint rotation angle; Based on the affine transformation matrix formed by local feature extraction, portrait posture correction is performed on the third image to obtain a fourth image.

8. The method according to claim 1, characterized in that The performing fatigue detection based on key points of the face in the fourth image includes: Based on the fourth image, determining key points of eyes, mouth, and head; Determining the eye aspect ratio based on the eye key points; Determine the degree of mouth opening and closing and the size of mouth corners based on the mouth key points; Based on the key points of the head, determine the up and down and left and right degrees of the head; Based on the changes in the eye length ratio, jaw opening and closing degree, mouth corner size, and head up and down, left and right degrees during the continuous connection time, they are compared with the set thresholds and the fatigue level is comprehensively determined as one of the following: awake, moderately tired, or severely tired.

9. A device for detecting fatigue by enhancing and correcting portraits in low-light and multi-angle conditions, characterized in that: include: an acquisition unit, configured to acquire a first image; an enhancement processing unit, configured to perform image enhancement processing based on the first image and a second image formed using an adaptive texture analysis method to obtain a third image; wherein the adaptive texture analysis method performs feature encoding based on a local binary pattern of a window area; a posture correction unit, configured to perform portrait posture correction on the third image based on an affine transformation matrix formed by extracting local features, to obtain a fourth image; A fatigue detection unit is used to perform fatigue detection based on key points of the face in the fourth image.

10. An electronic device, characterized in that: The electronic device includes: a memory and a processor, wherein the memory is used to store computer-readable instructions; the processor is used to read the computer-readable instructions and implement the portrait enhancement and correction fatigue detection method in low-light multi-angle conditions as described in any one of claims 1 to 8.

Images