Methods, devices, equipment, and storage media for detecting and warning of driver fatigue.

Abstract

This disclosure provides a method, device, equipment, and storage medium for detecting and warning of driver fatigue. The method includes: reconstructing an image to be detected using a low-rank approximation method to obtain a reconstructed image to be detected; extracting facial features from the reconstructed image to be detected, and calculating the current eye aspect ratio, current mouth aspect ratio, and current head pitch angle using the facial features; determining the magnitude of a dynamic deviation operator based on the degree of change between the current eye aspect ratio and the original eye aspect ratio, and the degree of change between the current mouth aspect ratio and the original mouth aspect ratio; combining the values ​​of the current eye aspect ratio and the current mouth aspect ratio using the dynamic deviation operator to obtain a dynamic weighted sum of eye and mouth; determining the driver's pitch angle state, and if the current head pitch angle is greater than a pitch angle threshold, determining driver fatigue; otherwise, determining the driver's eye and mouth opening / closing state, and if the dynamic weighted sum of eye and mouth is greater than a combined eye and mouth threshold, determining driver fatigue.

Description

Technical Field

[0001] This document relates to the field of computer vision technology, and in particular to a method, device, equipment, and storage medium for detecting and warning of driver fatigue in automobiles. Background Technology

[0002] Fatigue driving refers to the phenomenon that drivers experience inattention, fatigue, drowsiness, and operational errors after driving for a long time, which can easily lead to traffic accidents. Therefore, fatigue driving detection has become a crucial task.

[0003] In related technologies, traditional methods mainly utilize smart sensors to detect the driver's physiological indicators, such as heart rate and skin conductance. This requires the driver to wear multiple sensors to assist in data collection, resulting in costly and inefficient data acquisition devices. In recent years, deep learning has also been gradually applied to fatigue driving detection in automobiles, using deep learning methods to detect facial expressions or eye movement frequency. However, these deep learning methods are too simplistic in their recognition approach, lacking dynamic balance between parameters, and suffer from frame loss during recognition, making them prone to misjudgments.

[0004] Based on the above analysis of the development status of this technology field, existing technologies lack solutions for balancing the correlation between parameters. Summary of the Invention

[0005] The purpose of this invention is to provide a method, device, equipment, and storage medium for detecting and warning of driver fatigue in automobiles, in order to solve the aforementioned problems in the prior art.

[0006] According to a first aspect of the present disclosure, a method for detecting and warning of driver fatigue in automobiles is provided, comprising:

[0007] The low-rank approximation method is used to reconstruct the image to be detected, and the reconstructed image to be detected is obtained. Facial features are extracted from the reconstructed image to be detected, and the current aspect ratio of the eyes, the current aspect ratio of the mouth, and the current head pitch angle are calculated based on the facial features.

[0008] The size of the dynamic deviation operator is determined based on the degree of change between the current eye aspect ratio and the original eye aspect ratio, and the degree of change between the current mouth aspect ratio and the original mouth aspect ratio.

[0009] The dynamic bias operator is used to combine the current eye aspect ratio and the current mouth aspect ratio to obtain a dynamic weighted sum of the eyes and mouth.

[0010] The system determines the driver's pitch angle. If the current head pitch angle is greater than the pitch angle threshold, the driver is considered to be driving while fatigued. Otherwise, the system determines the driver's eye and mouth opening / closing status. If the dynamic weighted sum of the eye and mouth opening / closing is greater than the combined eye and mouth opening / closing threshold, the driver is considered to be driving while fatigued.

[0011] According to a second aspect of the present disclosure, a vehicle fatigue driving detection and early warning device is provided, comprising:

[0012] The image reconstruction and feature extraction module is used to reconstruct the image to be detected using the low-rank approximation method to obtain the reconstructed image to be detected; extract facial features from the reconstructed image to be detected, and calculate the current eye aspect ratio, current mouth aspect ratio, and current head pitch angle through the facial features;

[0013] The deviation operator size determination module is used to determine the size of the dynamic deviation operator based on the degree of change between the current eye aspect ratio and the original eye aspect ratio, and the degree of change between the current mouth aspect ratio and the original mouth aspect ratio.

[0014] The dynamic weighted sum calculation module for eyes and mouth is used to combine the current eye aspect ratio and the current mouth aspect ratio using a dynamic deviation operator to obtain the dynamic weighted sum of eyes and mouth.

[0015] The fatigue driving detection module is used to determine the driver's pitch angle. If the current head pitch angle is greater than the pitch angle threshold, it is determined to be fatigue driving; otherwise, it determines the driver's eye and mouth opening and closing status. If the dynamic weighted sum of the eye and mouth is greater than the combined eye and mouth threshold, it is determined to be fatigue driving.

[0016] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0017] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the method for detecting and warning of vehicle fatigue driving disclosed in the first aspect are implemented when the computer program is executed by the processor.

[0018] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which an information transmission implementation program is stored, wherein when the program is executed by a processor, it implements the steps of the vehicle fatigue driving detection and early warning method of the first aspect of the present disclosure.

[0019] The technical solution provided by the embodiments of the present invention has the following beneficial effects: using the low-rank approximation method to reconstruct the image reduces the frame loss phenomenon in the recognition process and improves the sensitivity of visual recognition and detection; using the dynamic bias operator balances the correlation between the aspect ratio of the eyes and the aspect ratio of the mouth, thereby improving the recognition and detection effect of car fatigue driving.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a method for detecting and warning of driver fatigue in automobiles according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of low-rank approximation reconstruction according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of facial feature extraction according to an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram illustrating the calculation of the head pitch angle according to an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of fatigue driving detection according to an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of a vehicle fatigue driving detection and early warning device according to an embodiment of the present invention;

[0028] Figure 7 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0030] Method Implementation Examples

[0031] According to an embodiment of the present invention, a method for detecting and warning of driver fatigue in automobiles is provided. Figure 1 This is a flowchart of a method for detecting and warning of driver fatigue in automobiles according to an embodiment of the present invention, as follows: Figure 1 As shown, the method for detecting and warning of vehicle fatigue driving according to an embodiment of the present invention specifically includes:

[0032] In step S110, the low-rank approximation method is used to reconstruct the image to be detected, resulting in a reconstructed image. Facial features are extracted from the reconstructed image, and the current eye aspect ratio, mouth aspect ratio, and head pitch angle are calculated using these features. Specifically, this includes:

[0033] The T-SVD low-rank approximation method is used to decompose the third-order tensor contained in the image to be detected into two orthogonal matrices and one diagonal matrix. Formula 1 is used to decompose the third-order tensor of the image to be detected:

[0034] A = U * S * V T Formula 1;

[0035] In the formula, A represents the third-order tensor of the image to be detected, S represents the diagonal matrix, and U and V represent orthogonal matrices. Larger eigenvalues ​​in the diagonal matrix S are retained; in this embodiment, 1 / 5 to 1 / 10 of the image's rank is retained. The retained eigenvalues ​​are multiplied by the corresponding eigenvectors, i.e., a low-rank approximation, to obtain the reconstructed image to be detected. The reconstructed image is an approximate restoration of the image to be detected. Figure 2 This is a schematic diagram of low-rank approximation reconstruction according to an embodiment of the present invention, as shown below. Figure 2 As shown, the process of low-rank approximation reconstructing the decomposed tensor is illustrated.

[0036] The aspect ratio of the eyes is calculated by selecting multiple eye feature points from the facial features, and the aspect ratio of the mouth is calculated by selecting multiple mouth feature points from the facial features. Figure 3 This is a schematic diagram of facial feature extraction according to an embodiment of the present invention, as shown below. Figure 3 As shown, the facial features extracted in this embodiment are used. Formula 2 is used to calculate the current aspect ratio of the eyes, and Formula 3 is used to calculate the current aspect ratio of the mouth.

[0037]

[0038]

[0039] In the formula, EAR represents the aspect ratio of the eyes, i.e., the degree of eye opening and closing; MAR represents the aspect ratio of the mouth, i.e., the degree of mouth opening and closing; and P represents... Figure 3 The feature points on the P-axis represent the coordinates of the specific feature points.

[0040] The head pitch angle is obtained by calculating the Euler angles in the facial features through quaternion construction. The quaternion is constructed using Formula 4, and the head pitch angle is obtained using Formula 5.

[0041]

[0042] pitch=arcsin[2(wy-zx)] Formula 5;

[0043] In the formula, 'a' represents the angle of rotation about the axis of rotation; 'w' represents the real part of the quaternion, i.e., the angle of rotation about the origin; and 'cosθ' represents the angle of rotation about the origin. x cosθ y cosθ z These represent the components of the rotation axis in the x, y, and z axes, respectively; x, y, and z represent the imaginary components of the quaternion in the x, y, and z axes, i.e., the angles of rotation around the x, y, and z axes; pitch represents the head pitch angle, i.e., the rotation angle around the y-axis. Figure 4 This is a schematic diagram illustrating the calculation of the head pitch angle according to an embodiment of the present invention, as shown below. Figure 4 As shown, a schematic diagram of spatial angles is presented in the figure. Let ψ represent the rotation angle along the x-axis, ψ represent the rotation angle along the z-axis, and N(y′) represent the head pitch angle. Among these, the rotation angle along the y-axis is most relevant to drowsy driving behavior, so the head pitch angle is chosen to represent the driving posture.

[0044] In step S120, the magnitude of the dynamic deviation operator is determined based on the degree of change between the current eye aspect ratio and the original eye aspect ratio, and the degree of change between the current mouth aspect ratio and the original mouth aspect ratio. Specifically, this includes:

[0045] Obtain the pre-collected original aspect ratios of the driver's eyes and mouth, and set critical values ​​for both. Calculate the eye aspect ratio deviation by comparing the current eye aspect ratio with the critical value, and calculate the mouth aspect ratio deviation by comparing the current mouth aspect ratio with the critical value. Determine the value of the dynamic deviation operator based on these values; the dynamic deviation operator's value range is [0,1]. Use Formula 6 to calculate the eye aspect ratio deviation, and Formula 7 to calculate the mouth aspect ratio deviation.

[0046]

[0047]

[0048] In the formula, F(EAR) represents the deviation value of the eye aspect ratio, F(MAR) represents the deviation value of the mouth aspect ratio, and EAR critical The aspect ratio of the eye is the critical value, MAR. critical This represents the critical value for the mouth's aspect ratio. If both the calculated eye aspect ratio deviation metric F(EAR) and mouth aspect ratio deviation metric F(MAR) are 0, the dynamic deviation operator α is set to 0.5; otherwise, Formula 8 is used to calculate the dynamic deviation operator α:

[0049] α=F(EAR) / (F(EAR)+F(MAR)) Formula 8;

[0050] In step S130, the dynamic deviation operator combines the current eye aspect ratio and the current mouth aspect ratio to obtain a dynamic weighted sum of the eyes and mouth. The dynamic weighted sum of the eyes and mouth is obtained using Formula 9:

[0051] Π(α)=(EAR) α *(MAR) 1-α Formula 9;

[0052] In the formula, ∏(α) represents the dynamic weighted sum of the eyes and mouth, EAR represents the current aspect ratio of the eyes, MAR represents the current aspect ratio of the mouth, and α represents the dynamic deviation operator.

[0053] In step S140, the driver's pitch angle is determined. If the current head pitch angle is greater than the pitch angle threshold, it is determined to be fatigued driving and an alarm is issued. Otherwise, the driver's eye and mouth opening and closing status is determined. If the dynamic weighted sum of the eyes and mouth is greater than the combined eye and mouth threshold, it is determined to be fatigued driving and an alarm is issued.

[0054] The method further includes:

[0055] From the image sequence previously processed by the low-rank approximation method, the pitch angle of each image is obtained and the average value is calculated to obtain the pitch angle threshold. From the image sequence previously processed by the low-rank approximation method, the aspect ratio of the eyes and the aspect ratio of the mouth of each image are obtained, and the average value is calculated to obtain the aspect ratio thresholds of the eyes and the mouth. The aspect ratio thresholds of the eyes and the aspect ratio thresholds of the mouth are fused to obtain the composite threshold of the eyes and mouth.

[0056] The pitch angle threshold, eye aspect ratio threshold, and eye aspect ratio threshold are pre-obtained based on historical experience. Specifically, this involves collecting images of each frame over a period of time to obtain the image sequence F1, F2, F3, ..., F num The T-SVD low-rank approximation method is used to decompose the third-order tensor contained in each image in the image sequence into two orthogonal matrices and a diagonal matrix, preserving 1 / 5 to 1 / 10 of the rank of the image. The preserved eigenvalues ​​are multiplied by the corresponding eigenvectors, i.e., low-rank approximation, to obtain the reconstructed sequence F1′, F2′, F3′, ..., F num ′;

[0057] Facial features are extracted from the reconstructed image sequence. The aspect ratios of the eyes and mouth, as well as the head pitch angle, are calculated for each image in the reconstructed sequence. The eye aspect ratio threshold is calculated using Formula 10, the mouth aspect ratio threshold is calculated using Formula 11, and the pitch angle threshold is calculated using Formula 12.

[0058]

[0059]

[0060]

[0061] In the formula, i = 1, 2, ..., num, V EAR The eye aspect ratio threshold, EAR i V represents the aspect ratio of the eyes in the i-th image of the reconstructed image sequence; MAR MAR represents the mouth aspect ratio threshold. i V represents the aspect ratio of the mouth in the i-th image of the reconstructed image sequence; pitch Indicates the pitch angle threshold, pitch i This represents the head pitch angle in the i-th image of the reconstructed image sequence; each threshold is obtained by calculating the average of each index in the reconstructed sequence. The eye and mouth synthesis thresholds are calculated using Formula 13:

[0062] V mix =max{V EAR V MAR}+1 Formula 13;

[0063] In the formula, V mix This represents the threshold for combining the eyes and mouth.

[0064] During detection, the pitch angle threshold V pitch Used to compare with the current head pitch angle to determine the driver's pitch angle status; eye and mouth synthesis threshold V mix It is used to compare with the dynamic weighted sum of eyes and mouth and Π(α) to determine the opening and closing state of the driver's eyes and mouth. Figure 5 This is a schematic diagram of fatigue driving detection according to an embodiment of the present invention, as shown below. Figure 5 As shown, this demonstrates the use of pitch angle threshold V pitch Combined with the threshold V of eyes and mouth mix The process of determining current fatigue driving.

[0065] In summary, to address the existing problems, this invention proposes a method for detecting and warning of driver fatigue in automobiles. It uses a low-rank approximation method to reconstruct images, reducing frame loss during recognition and improving the sensitivity of visual recognition and detection. A dynamic bias operator is used to balance the correlation between the aspect ratios of the eyes and mouth, considering the mutual influence of changes in these ratios, thus enhancing the detection effect of driver fatigue. Overall, it achieves good recognition and detection results without using sensor devices, effectively reducing hardware costs.

[0066] Device Examples

[0067] According to an embodiment of the present invention, a device for detecting and warning of driver fatigue in automobiles is provided. Figure 6 This is a schematic diagram of a vehicle fatigue driving detection and warning device according to an embodiment of the present invention, as shown below. Figure 6 As shown, the vehicle fatigue driving detection and early warning device according to an embodiment of the present invention specifically includes:

[0068] The image reconstruction and feature extraction module 60 is used to reconstruct the image to be detected using a low-rank approximation method, obtaining the reconstructed image; extracting facial features from the reconstructed image, and calculating the current eye aspect ratio, current mouth aspect ratio, and current head pitch angle based on the facial features. Specifically, the image reconstruction and feature extraction module 60 is used for:

[0069] The T-SVD low-rank approximation method is used to decompose the third-order tensor contained in the image to be detected into two orthogonal matrices and a diagonal matrix. The larger eigenvalues ​​in the diagonal matrix are retained, and the retained eigenvalues ​​are multiplied by the corresponding eigenvectors to obtain the reconstructed image to be detected.

[0070] The current eye aspect ratio is calculated by selecting multiple eye feature points from facial features.

[0071] The current mouth aspect ratio is calculated by selecting multiple mouth feature points from the facial features.

[0072] The Euler angles in the facial features are calculated by constructing quaternions, and the current head pitch angle is obtained from the Euler angles.

[0073] The dynamic deviation operator determination module 62 is used to determine the magnitude of the dynamic deviation operator based on the degree of change between the current eye aspect ratio and the original eye aspect ratio, and the degree of change between the current mouth aspect ratio and the original mouth aspect ratio. Specifically, the dynamic deviation operator determination module 62 is used for:

[0074] Obtain the pre-collected original aspect ratio of the driver's eyes and mouth, and set the critical values ​​for the aspect ratio of the eyes and mouth. Calculate the aspect ratio deviation metric by comparing the current aspect ratio of the eyes with the critical value of the eyes, and calculate the aspect ratio deviation metric by comparing the current aspect ratio of the mouth with the critical value of the mouth. Determine the value of the dynamic deviation operator based on the aspect ratio deviation metric and the mouth aspect ratio deviation metric. The value range of the dynamic deviation operator is [0,1].

[0075] The dynamic weighted sum calculation module 64 for eyes and mouth is used to combine the current aspect ratio of the eyes and the current aspect ratio of the mouth using a dynamic deviation operator to obtain the dynamic weighted sum of eyes and mouth.

[0076] The fatigue driving detection module 66 is used to determine the driver's pitch angle status. If the current head pitch angle is greater than the pitch angle threshold, it is determined to be fatigue driving and an alarm is issued. Otherwise, it determines the driver's eye and mouth opening and closing status. If the dynamic weighted sum of the eyes and mouth is greater than the combined eye and mouth threshold, it is determined to be fatigue driving and an alarm is issued.

[0077] The device further includes:

[0078] The pitch angle threshold calculation module 68 is used to obtain the pitch angle of each image from the image sequence that has been processed by the low-rank approximation method and calculate the average value to obtain the pitch angle threshold.

[0079] The mouth aspect ratio threshold fusion module 610 is used to obtain the eye aspect ratio and mouth aspect ratio of each image from the image sequence that has been processed by the low-rank approximation method, calculate the average value to obtain the eye aspect ratio threshold and mouth aspect ratio threshold respectively, and fuse the eye aspect ratio threshold and mouth aspect ratio threshold to obtain the eye and mouth composite threshold.

[0080] In summary, addressing the existing problems, this invention provides a vehicle fatigue driving detection and warning device. It uses a low-rank approximation method to reconstruct images, reducing frame loss during recognition and improving the sensitivity of visual recognition and detection. A dynamic bias operator is used to balance the correlation between the aspect ratios of the eyes and mouth, considering the mutual influence of changes in these ratios, thus enhancing the recognition and detection effect of vehicle fatigue driving. Overall, it achieves good recognition and detection results without using sensor devices, effectively reducing hardware costs.

[0081] Electronic device examples

[0082] Figure 7 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device 700 may include at least one processor 710 and a memory 720. The processor 710 can execute instructions stored in the memory 720. The processor 710 is communicatively connected to the memory 720 via a data bus. In addition to the memory 720, the processor 710 can also be communicatively connected to an input device 730, an output device 740, and a communication device 750 via the data bus.

[0083] The processor 710 can be any conventional processor, such as a commercially available CPU. The processor may also include, for example, a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a System on Chip (SOC), an Application Specific Integrated Circuit (ASIC), or a combination thereof.

[0084] The memory 720 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0085] In this embodiment of the present disclosure, the memory 720 stores executable instructions, and the processor 710 can read the executable instructions from the memory 720 and execute the instructions to implement all or part of the steps of the vehicle fatigue driving detection and warning method in any of the above exemplary embodiments.

[0086] Computer-readable storage medium embodiments

[0087] In addition to the methods and apparatus described above, exemplary embodiments of this disclosure may also be a computer program product or a computer-readable storage medium storing the computer program product, the computer product including computer program instructions that can be executed by a processor to implement all or part of the steps described in any of the methods in the exemplary embodiments described above.

[0088] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. Programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0089] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, or any suitable combination thereof.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting and warning of fatigue driving of an automobile, characterized in that, The method comprises the following steps: reconstructing the to-be-detected image using a low-rank approximation method to obtain a reconstructed to-be-detected image; extracting a facial feature in the reconstructed to-be-detected image, and calculating a current eye aspect ratio, a current mouth aspect ratio and a current head pitch angle through the facial feature, specifically comprising: using a T-SVD low-rank approximation method to decompose a third-order tensor contained in the to-be-detected image into two orthogonal matrices and a diagonal matrix, retaining larger eigenvalues in the diagonal matrix, wherein the larger eigenvalues represent a rank of 1 / 5-1 / 10 of the retained image, and multiplying the retained eigenvalues with corresponding eigenvectors to obtain the reconstructed to-be-detected image; determining the size of a dynamic bias operator according to the degree of change of the current eye aspect ratio and the original eye aspect ratio, and the degree of change of the current mouth aspect ratio and the original mouth aspect ratio, specifically comprising: obtaining a pre-acquired original eye aspect ratio and an original mouth aspect ratio of a driver, setting an eye aspect ratio threshold value and a mouth aspect ratio threshold value, calculating an eye aspect ratio deviation metric value by comparing the size of the current eye aspect ratio and the eye aspect ratio threshold value, calculating a mouth aspect ratio deviation metric value by comparing the size of the current mouth aspect ratio and the mouth aspect ratio threshold value, and determining the value of the dynamic bias operator according to the eye aspect ratio deviation metric value and the mouth aspect ratio deviation metric value; wherein the value range of the dynamic bias operator is [0, 1], the eye aspect ratio deviation metric value is calculated using formula 1, and the mouth aspect ratio deviation metric value is calculated using formula 2: Formula 1 ; Formula 2; wherein represents the eye aspect ratio deviation metric value, represents the mouth aspect ratio deviation metric value, represents the eye aspect ratio critical value, represents the mouth aspect ratio critical value; if the calculated eye aspect ratio deviation metric value and the mouth aspect ratio deviation metric value are both 0, the dynamic deviation operator α takes the value 0.5, otherwise the dynamic deviation operator α is calculated using equation 3: Equation 3; combining the values of the current eye aspect ratio and the current mouth aspect ratio using the dynamic bias operator to obtain an eye-mouth dynamic weighted sum, and obtaining the eye-mouth dynamic weighted sum using formula 4: Formula 4; wherein, denotes the eye mouth dynamic weighting sum, denotes the current eye aspect ratio, denotes the current mouth aspect ratio, α denotes the dynamic bias operator; judging the pitch angle state of driving, if the current head pitch angle is greater than a pitch angle threshold value, it is determined that the driver is fatigued; otherwise, judging the eye-mouth opening state of driving, if the eye-mouth dynamic weighted sum is greater than an eye-mouth synthesis threshold value, it is determined that the driver is fatigued.

2. The method of claim 1, wherein, The method further comprises: calculating the current eye aspect ratio by selecting a plurality of eye feature points in the facial feature; calculating the current mouth aspect ratio by selecting a plurality of mouth feature points in the facial feature; calculating the current head pitch angle by constructing a quaternion to calculate the Euler angle in the facial feature.

3. The method of claim 1, wherein, The method further comprises: obtaining the pitch angle threshold value from the average value of the pitch angle of each image in a sequence of images processed by the low-rank approximation method.

4. The method of claim 1, wherein, The method further comprises: obtaining the eye aspect ratio threshold value and the mouth aspect ratio threshold value from the average value of the eye aspect ratio and the mouth aspect ratio of each image in a sequence of images processed by the low-rank approximation method, respectively, and fusing the eye aspect ratio threshold value and the mouth aspect ratio threshold value to obtain the eye-mouth synthesis threshold value.

5. A device for detecting and warning of fatigue driving of an automobile, characterized in that, The method comprises the following steps: The image reconstruction and feature extraction module is configured to reconstruct the image to be detected using a low-rank approximation method to obtain a reconstructed image to be detected, extract facial features in the reconstructed image to be detected, and calculate a current eye aspect ratio, a current mouth aspect ratio, and a current head pitch angle based on the facial features, and specifically configured to: The T-SVD low-rank approximation method is used to decompose a third-order tensor contained in the image to be detected into two orthogonal matrices and a diagonal matrix, and the larger eigenvalues in the diagonal matrix are retained, wherein the larger eigenvalues represent a rank of 1 / 5-1 / 10 of the retained image, and the retained eigenvalues are multiplied by corresponding eigenvectors to obtain the reconstructed image to be detected. The dynamic bias operator determination module is configured to determine the size of the dynamic bias operator based on the degree of change of the current eye aspect ratio and the original eye aspect ratio and the degree of change of the current mouth aspect ratio and the original mouth aspect ratio, and specifically configured to: The original eye aspect ratio and the original mouth aspect ratio of the driver are obtained in advance, and the eye aspect ratio threshold and the mouth aspect ratio threshold are set. The eye aspect ratio deviation measurement value is calculated by comparing the size of the current eye aspect ratio and the eye aspect ratio threshold, and the mouth aspect ratio deviation measurement value is calculated by comparing the size of the current mouth aspect ratio and the mouth aspect ratio threshold. The value of the dynamic bias operator is determined according to the eye aspect ratio deviation measurement value and the mouth aspect ratio deviation measurement value. The value of the dynamic bias operator ranges from 0 to 1. The eye aspect ratio deviation measurement value is calculated using Formula 1, and the mouth aspect ratio deviation measurement value is calculated using Formula 2: Formula 1: Formula 2: wherein represents an eye aspect ratio deviation metric value, represents a mouth aspect ratio deviation metric value, represents an eye aspect ratio critical value, represents a mouth aspect ratio critical value; if the calculated eye aspect ratio deviation metric value and the mouth aspect ratio deviation metric value are both 0, the dynamic deviation operator α takes the value 0.5, otherwise the dynamic deviation operator α is calculated using equation 3: Formula 3: The eye and mouth dynamic weighted sum calculation module is configured to combine the values of the current eye aspect ratio and the current mouth aspect ratio using the dynamic bias operator to obtain an eye and mouth dynamic weighted sum, and the eye and mouth dynamic weighted sum is obtained using Formula 4: Formula 4: wherein, denotes the eye mouth dynamic weighting sum, denotes the current eye aspect ratio, denotes the current mouth aspect ratio, α denotes the dynamic bias operator; The fatigue driving detection module is configured to judge the pitch angle state of driving. If the current head pitch angle is greater than the pitch angle threshold, it is determined that the driver is fatigued. Otherwise, the eye and mouth opening state of the driver is judged. If the eye and mouth dynamic weighted sum is greater than the eye and mouth synthesis threshold, it is determined that the driver is fatigued.

6. The apparatus of claim 5, wherein, The device further comprises: The pitch angle threshold calculation module is configured to obtain the pitch angle of each image from a sequence of images processed by the low-rank approximation method in the past and calculate the average value to obtain the pitch angle threshold. The mouth aspect ratio threshold fusion module is configured to obtain the eye aspect ratio and the mouth aspect ratio of each image from a sequence of images processed by the low-rank approximation method in the past, calculate the average value of the eye aspect ratio and the mouth aspect ratio respectively to obtain the eye aspect ratio threshold and the mouth aspect ratio threshold, and fuse the eye aspect ratio threshold and the mouth aspect ratio threshold to obtain the eye and mouth synthesis threshold.

7. An electronic device, comprising: It comprises: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is executed by the processor to implement the steps of the method for detecting and warning of fatigue driving of a vehicle according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores an information transmission implementation program, and the program is executed by the processor to implement the steps of the automobile fatigue driving detection and warning method in any one of claims 1 to 4.

Images