Fatigue state recognition method, system, and storage medium based on deep learning

Abstract

The present invention provides a fatigue state recognition method, system, storage medium and electronic device based on deep learning, the method includes: S1, obtaining video stream data, and obtaining a face picture from the video stream data; S2, performing face detection and face key point detection on the face picture; S3, obtaining facial feature information of the face picture through a face alignment model; S4, performing gaze estimation, head posture estimation and mouth estimation on the facial feature information, and obtaining gaze estimation information, head posture estimation information and mouth estimation information respectively; S5, combining the gaze estimation information, head posture estimation information and mouth estimation information to obtain a comprehensive evaluation value for fatigue state recognition; S6, comparing the comprehensive evaluation value with a set value to determine whether the person corresponding to the face picture is in a fatigue state. According to the method of the embodiment of the present invention, the fatigue state of the person is detected in real time, and the accuracy of recognition is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and image processing, and more specifically, to a fatigue state recognition method based on deep learning, a fatigue state recognition system based on deep learning, a computer storage medium and an electronic device. Background Art

[0002] In recent years, due to the rapid development of the Internet and artificial intelligence technology, intelligent monitoring and intelligent personnel management have received widespread attention from the public. In some enterprises involved in production safety and public security departments with militarized management, full coverage of video surveillance has been achieved. Some on-duty personnel are difficult to ensure that they are always in a normal working state. For some special security positions, they are prone to job fatigue. Neglecting their duties will bring huge safety hazards. Therefore, the behavior analysis of on-duty personnel plays an increasingly important role in intelligent management.

[0003] At present, there are two main methods for identifying fatigue behavior of on-the-job personnel. One is based on human body detection, which uses full-body images of the human body to model and analyze human posture features. Human posture is easily affected by occlusion. Once the human body is occluded, fatigue detection cannot be performed.

[0004] When the human body is not obscured, human body detection generally uses a large-scale target detection algorithm network. The human body detection area is the whole body or half of the body. The human body image still needs to be segmented after detection. Therefore, the recognition method based on human body detection is generally time-consuming, and the real-time detection performance of the recognition system is insufficient.

[0005] The other is a method based on face detection. The face detection method based on traditional algorithms does not obtain the deep feature information of the sleeping personnel while they are on duty, and its accuracy is not high.

[0006] The existing fatigue status identification methods have a single technical evaluation index. Fatigue detection of on-the-job personnel is carried out by only identifying one index, which is easy to cause misjudgment of the working status of on-the-job personnel. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a fatigue state recognition method based on deep learning, a fatigue state recognition system based on deep learning, a computer storage medium and an electronic device, which can detect the fatigue state of personnel in real time and effectively improve the recognition accuracy.

[0008] According to the first aspect of the present invention, the fatigue state recognition method based on deep learning includes: S1, obtaining video stream data, and obtaining a face picture from the video stream data; S2, performing face detection and face key point detection on the face picture; S3, obtaining facial feature information of the face picture through a face alignment model; S4, performing gaze estimation, head posture estimation and mouth estimation on the facial feature information, and obtaining gaze estimation information, head posture estimation information and mouth estimation information respectively; S5, combining the gaze estimation information, the head posture estimation information and the mouth estimation information to obtain a comprehensive evaluation value for fatigue state recognition; S6, comparing the comprehensive evaluation value with a set value to determine whether the person corresponding to the face picture is in a fatigue state.

[0009] According to the deep learning-based fatigue status recognition method of an embodiment of the present invention, by combining deep neural networks with traditional image processing technology, an evaluation system for fatigue evaluation is set up through multiple indicators such as estimation information, head posture estimation information and mouth estimation information. It can detect the fatigue status of personnel in real time and effectively improve the recognition accuracy.

[0010] According to one embodiment of the present invention, step S2 includes:

[0011] S21, preprocessing the face image; S22, marking multiple key points and face frames of the face in the face image to obtain marked data; S23, forming the marked data into a training set and training to obtain a face key point detection model; S24, performing face detection and face key point detection on the face image using the face key point detection model.

[0012] According to one embodiment of the present invention, in step S4, the gaze estimation comprises the following steps:

[0013] S41. Obtaining face information from the facial information features; S42. Preliminarily determining the position of human eyes based on the face information; S43. Using the SAGE algorithm to re-locate the eyes and determine the position of the eyes; S44. Performing eye gaze estimation analysis; S45. Defining an attention concentration area within the screen area. If the line of sight of the human eyes exceeds the attention concentration area within a preset time, it is determined that the person corresponding to the face information is in a fatigue state.

[0014] According to one embodiment of the present invention, in step S44, an improved SAGE algorithm is used to perform eye gaze estimation analysis, wherein the improved SAGE algorithm includes the following steps: S441, performing face detection to obtain face key point information; S442, preliminarily obtaining the eye position based on the face key point information; S443, using the position coordinates of the four corners of the eye to accurately locate the eye position; S444, using the SAGE algorithm to perform line of sight estimation to obtain the eye gaze point, and identify the eye gaze concentration within the set area.

[0015] According to one embodiment of the present invention, in step S4, the head posture estimation includes the following steps:

[0016] S41', obtain face information from the facial information features; S42', detect and locate the head position according to the face information; S43', use FSA-Net algorithm to obtain the head posture, and obtain the head posture Euler angles yaw, pitch, roll; S44', analyze the head posture; S45', set parameter thresholds, if the angle changes in the Pitch and Roll directions within a preset time exceed a preset time ratio, it is determined that the person corresponding to the face information is in a fatigue state.

[0017] According to one embodiment of the present invention, step S43' includes:

[0018] S431', extract facial features according to the face information; S432', fuse features through a convolutional neural network to generate a feature map and a feature vector; S433', generate a parameter set through the feature vector; S434', convert the parameter set into a head posture angle to obtain the yaw, pitch, and roll angles.

[0019] According to one embodiment of the present invention, the mouth estimation includes: S51, making a preliminary judgment on the mouth opening size in the facial information according to a set first threshold; if the first threshold is greater than a first set threshold, it is judged to be an open mouth state, and the duration of the open mouth state within a preset time is counted; if the duration exceeds a preset time ratio, it is judged that the person corresponding to the facial information is in a fatigue state; S52, making a secondary judgment on a second threshold that is less than the first threshold; if the second threshold is greater than a second set threshold, and the open mouth continues for more than a preset number of frames in the video stream, it is judged that the person corresponding to the facial information is in a fatigue state.

[0020] A fatigue state recognition system based on deep learning according to an embodiment of the second aspect of the present invention includes:

[0021] A data acquisition module, wherein the data acquisition module acquires video stream data and acquires a face image from the video stream data; a face detection module, wherein the face detection module performs face detection and face key point detection on the face image; a face alignment module, wherein the face alignment module acquires facial feature information of the face image through a face alignment model; an information estimation module, wherein the information estimation module performs gaze estimation, head posture estimation and mouth estimation on the facial feature information to obtain gaze estimation information, head posture estimation information and mouth estimation information respectively; a comprehensive evaluation module, wherein the comprehensive evaluation module combines the gaze estimation information, the head posture estimation information and the mouth estimation information to obtain a comprehensive evaluation value for fatigue state recognition; and a judgment module, wherein the judgment module compares the comprehensive evaluation value with a set value to determine whether the person corresponding to the face image is in a fatigue state.

[0022] In a third aspect, an embodiment of the present invention provides a computer storage medium, comprising one or more computer instructions, which implement the method described in the above embodiment when executed.

[0023] An electronic device according to an embodiment of the fourth aspect of the present invention includes a memory and a processor, wherein the memory is used to store one or more computer instructions; the processor is used to call and execute the one or more computer instructions, thereby implementing a method as described in any of the above embodiments.

[0024] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0026] Figure 1 is a flowchart of a fatigue state recognition method based on deep learning according to an embodiment of the present invention;

[0027] Figure 2 A gaze estimation graph in a fatigue state recognition method based on deep learning according to an embodiment of the present invention;

[0028] Figure 3 A schematic diagram of a fatigue state recognition system based on deep learning according to an embodiment of the present invention;

[0029] Figure 4 Schematic diagram of an electronic device according to an embodiment of the present invention.

[0030] Reference numerals:

[0031] Fatigue state recognition system based on deep learning 100;

[0032] Data acquisition module 10; face detection module 20; face alignment module 30;

[0033] Information estimation module 40; comprehensive evaluation module 50; judgment module 60;

[0034] Electronic device 300;

[0035] Memory 310; operating system 311; application program 312;

[0036] Processor 320 ; network interface 330 ; input device 340 ; hard disk 350 ; display device 360 ​​. DETAILED DESCRIPTION

[0037] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0038] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0039] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0040] The following first describes in detail the fatigue state recognition method based on deep learning according to an embodiment of the present invention with reference to the accompanying drawings.

[0041] like Figure 1 As shown, the fatigue state recognition method based on deep learning according to an embodiment of the present invention includes the following steps:

[0042] S1, obtain video stream data, and obtain face images from the video stream data;

[0043] S2, performing face detection and face key point detection on the face image;

[0044] S3, obtaining facial feature information of the face image through a face alignment model;

[0045] S4, performing gaze estimation, head posture estimation and mouth estimation on the facial feature information to obtain gaze estimation information, head posture estimation information and mouth estimation information respectively;

[0046] S5, combining the gaze estimation information, the head posture estimation information and the mouth estimation information to obtain a comprehensive evaluation value for fatigue state recognition;

[0047] S6. Compare the comprehensive evaluation value with the set value to determine whether the person corresponding to the face image is in a fatigue state.

[0048] First of all, it should be noted that the fatigue state recognition method based on deep learning according to the embodiment of the present invention can be used for some special safety positions, which are prone to fatigue at work and neglect of duty will bring huge safety hazards. For example: positions such as drivers and safety managers. The fatigue state recognition method based on deep learning according to the embodiment of the present invention can perform real-time fatigue detection on special safety positions such as drivers and safety managers to reduce safety hazards caused by fatigue work.

[0049] In other words, according to the fatigue state recognition method based on deep learning in the embodiment of the present invention, first, video stream data is acquired through a monitoring device or a camera device, and a face image is acquired from the video stream data, and the video stream data is decoded to generate image data including the face image. The method of obtaining the image data from the video stream data is easy to think of and can be implemented by those skilled in the art, and will not be elaborated here;

[0050] Next, face detection and face key point detection are performed on the face image. Face detection mainly provides head posture information, while face key point detection mainly provides eye position information.

[0051] Through the face alignment model, the facial feature information of the face image is obtained, including the feature information of the eyebrows, eyes, nose, mouth, face and other contours.

[0052] Fatigue estimation is performed on the acquired head posture information and facial feature information. It can be divided into: gaze estimation, that is, estimating the degree of human eye concentration. It can be estimated based on whether the human eye sight exceeds the concentration area for a certain period of time, and obtain information on whether the human eye sight exceeds the concentration area and the duration information; head posture estimation is fatigue estimation based on head posture, which can be estimated based on whether the nodding is frequent and the duration of the nodding amplitude, and obtain the nodding frequency and nodding amplitude information and duration information; mouth estimation is fatigue estimation based on mouth information, which can be estimated based on the duration of the mouth opening and closing degree. The continuous opening of the mouth indicates yawning, which indicates fatigue, and obtains the opening and closing degree information and duration information of the mouth.

[0053] The gaze estimation information, the head posture estimation information and the mouth estimation information are combined to obtain a comprehensive evaluation value for fatigue state recognition. Different weights can be assigned to the three types of information for weighted summation to obtain a comprehensive evaluation value for fatigue state recognition.

[0054] Finally, the comprehensive evaluation value is compared with the set value to determine whether the person corresponding to the face image is in a fatigue state.

[0055] Therefore, according to the fatigue status recognition method based on deep learning of an embodiment of the present invention, by combining deep neural networks with traditional image processing technology, an evaluation system for fatigue evaluation is set up through multiple indicators such as estimation information, head posture estimation information and mouth estimation information. It can detect the fatigue status of personnel in real time and effectively improve the recognition accuracy.

[0056] In some specific embodiments of the present invention, step S2 includes:

[0057] S21, preprocessing the face image;

[0058] S22, marking multiple key points and face frames of the face in the face image to obtain marking data;

[0059] S23, forming a training set with the labeled data and performing training to obtain a facial key point detection model;

[0060] S24. Performing face detection and face key point detection on the face image through a face key point detection model.

[0061] The face image can be a sequence of images generated from a video stream or a picture captured by a camera, wherein the image contains face information. Preprocessing the face image is an important step in the face detection process. Due to the different input image acquisition environments, in order to ensure the consistency of face size, position and face image quality in the face image, the image must be preprocessed.

[0062] The facial image data is preprocessed, which mainly includes data screening, image normalization, noise filtering and image enhancement, a total of four image preprocessing operations, so that the subsequent facial image feature extraction process can be more accurate and efficient.

[0063] In one embodiment of the present case, first, the face images are preprocessed, and the mean and variance values ​​of each image sequence are calculated respectively. For each frame image in each image sequence, a standardized preprocessing is performed by first subtracting the corresponding mean and then dividing by the corresponding variance to obtain a processed image sequence, and then a plurality of face samples can be generated according to each processed image sequence.

[0064] In one embodiment of the present case, the face samples can be divided into training samples and test samples, and the network model parameters of the convolutional neural network can be trained through the training samples, and the model performance obtained by the training can be verified through the test samples. When the model performance meets the set requirements, the network model parameters of the convolutional neural network will be re-trained or continued to be trained through the training samples until the model performance of the trained human shape detection model meets the set requirements.

[0065] In this embodiment, any convolutional neural network structure can be selected, such as ResNet network structure, VGG network structure, Densenet network structure, etc., which is not limited here.

[0066] The face key point detection model can be obtained by training with labeled samples, so that the face detection model can accurately extract effective facial feature information of the image sequence, including eyebrows, eyes, nose, mouth, face, etc.

[0067] The facial key point detection model can effectively obtain facial feature information, which can be used for subsequent fatigue estimation.

[0068] According to one embodiment of the present invention, in step S4, the modeling process of gaze estimation and head posture estimation is also as described in S21-S24, which will not be described in detail here.

[0069] According to one embodiment of the present invention, in step S4, the gaze estimation comprises the following steps:

[0070] S41, obtaining face information from facial information features;

[0071] S42, preliminarily determining the eye position of a person based on the face information;

[0072] S43, using SAGE algorithm to locate the human eye for the second time and determine the position of the human eye;

[0073] S44, performing human eye gaze estimation analysis;

[0074] S45. In the screen area, an attention concentration area is defined. If the line of sight of the human eye exceeds the attention concentration area within a preset time, it is determined that the person corresponding to the facial information is in a fatigue state.

[0075] Gaze estimation, that is, estimating the degree of human eye concentration, can be estimated based on whether the human eye gaze exceeds the attention concentration area for a certain duration;

[0076] In one embodiment of the present invention, Figure 2 As shown, the human eye gaze concentration area is preset. Since the monitoring video output monitors are different, the gaze concentration area is specified to be the size of the monitor screen. The output coordinates of the human eye gaze point are marked as (x, y). If the real-time coordinates are always within the specified area, it is recorded as normal; if the line of sight jumps out of the specified area, the state duration is preset within 10 seconds. If the duration is >40%, it is determined that the person corresponding to the facial information is in a fatigue state.

[0077] Preferably, if Figure 2 As shown, in step S44, the improved SAGE algorithm is used to perform eye gaze estimation analysis, wherein the improved SAGE algorithm includes the following steps:

[0078] S441, performing face detection to obtain key point information of the face;

[0079] S442, preliminarily obtaining the eye position according to the facial key point information;

[0080] S443, using the position coordinates of the four corners of the human eye to accurately locate the position of the human eye;

[0081] S444, using the SAGE algorithm to estimate the sight line, obtain the human eye gaze point, and identify the human eye gaze concentration in the set area. The improved algorithm effectively improves the accuracy of identifying the human eye concentration.

[0082] According to one embodiment of the present invention, in step S4, the head posture estimation includes the following steps:

[0083] S41', obtaining face information from facial information features;

[0084] S42', detecting and locating the head position according to the face information;

[0085] S43', using FSA-Net algorithm to obtain head posture, and obtain the head posture Euler angles yaw, pitch, roll;

[0086] S44', analyzing the head posture;

[0087] S45', setting parameter thresholds, if the angle change in the Pitch and Roll directions within the preset time exceeds the preset time ratio, it is determined that the person corresponding to the face information is in a fatigue state.

[0088] Head posture estimation is to obtain the posture angle of the head through the facial image, that is, to calculate the angles in the three directions of Pitch, Yaw and Roll. The academic names are pitch angle, yaw angle and roll angle, which respectively represent the up and down deflection, left and right deflection angle and roll angle of the head. Fatigue estimation can be performed through head posture, and it can be estimated by whether the head nods frequently and the amplitude of the nod, that is, Pitch and Roll;

[0089] Preferably, the parameter threshold is set to 0.32. Within a preset 10 seconds, when the time proportion of |Pitch|≥25° or |Roll|≥25° exceeds 30%, it is determined that the person corresponding to the facial information is in a fatigue state.

[0090] In some specific embodiments of the present invention, step S43' includes:

[0091] S431', extracting facial features according to the face information;

[0092] S432', fusing features through a convolutional neural network, generating a feature map, and generating a feature vector;

[0093] S433', generating a parameter set through a feature vector;

[0094] S434', convert the parameter set into head posture angles to obtain yaw, pitch, and roll angles.

[0095] This deep learning algorithm effectively reduces the number of parameters and computational complexity, significantly improves data acquisition efficiency, and can perform real-time recognition on the device.

[0096] According to one embodiment of the present invention, the mouth estimation includes:

[0097] S51, preliminarily judging the mouth opening degree in the face information according to a set first threshold value, if the first threshold value is greater than a first set threshold value, judging that the mouth is in an open state, counting the duration of the open mouth state within a preset time, if the duration exceeds a preset time ratio, judging that the person corresponding to the face information is in a fatigue state;

[0098] S52. Perform a secondary judgment on a second threshold value that is smaller than the first threshold value. If the second threshold value is larger than a second set threshold value, and the mouth is opened continuously for more than a preset number of frames in the video stream, it is determined that the person corresponding to the facial information is in a fatigue state.

[0099] In fatigue judgment scenarios, a continuously open mouth generally represents yawning, which means being in a fatigue state. Fatigue can be estimated based on the duration of the mouth opening and closing.

[0100] According to an embodiment of the present invention, the first threshold is set to 0.5, and the second threshold is set to 0.42. First judgment: When the first threshold is greater than the first threshold of 0.5, the mouth is considered to be in a wide-open state, and the duration of the wide-open state within 5 consecutive seconds is counted. If the duration is greater than 20%, it is determined to be a yawning state, that is, it is determined that the person corresponding to the face information is in a fatigue state.

[0101] A secondary judgment is performed on people with smaller mouth shapes. When the second threshold is greater than the second set threshold 0.42, the mouth is considered to be in an open state. The duration of the open state is counted for 5 consecutive seconds. If the duration is >20%, it is judged to be a yawning state, that is, the person corresponding to the facial information is judged to be in a fatigued state.

[0102] Secondary judgment based on the size of the person's mouth can effectively improve the accuracy of fatigue status recognition.

[0103] In summary, according to the deep learning-based fatigue status recognition method of an embodiment of the present invention, by combining deep neural networks and traditional image processing technology, an evaluation system for fatigue evaluation based on multiple indicators such as estimation information, head posture estimation information and mouth estimation information and multiple models are set up to detect the fatigue status of personnel in real time, the detection efficiency is significantly improved, and the recognition accuracy is effectively improved.

[0104] The embodiment of the present invention also provides a fatigue state recognition system 100 based on deep learning, including: a data acquisition module 10, a face detection module 20, a face alignment module 30, an information estimation module 40, a comprehensive evaluation module 50 and a judgment module 60.

[0105] Specifically, the data acquisition module 10 acquires video stream data and acquires a face image from the video stream data, the face detection module 20 performs face detection and face key point detection on the face image, the face alignment module 30 acquires facial feature information of the face image through a face alignment model, the information estimation module 40 performs gaze estimation, head posture estimation and mouth estimation on the facial feature information, and obtains gaze estimation information, head posture estimation information and mouth estimation information respectively, the comprehensive evaluation module 50 combines the gaze estimation information, the head posture estimation information and the mouth estimation information to obtain a comprehensive evaluation value for fatigue state recognition, and the judgment module 60 compares the comprehensive evaluation value with the set value to determine whether the person corresponding to the face image is in a fatigue state.

[0106] The functions of each module of the deep learning-based fatigue state recognition system according to an embodiment of the present invention have been described in detail in the above embodiment, so they will not be repeated here.

[0107] In summary, the fatigue status recognition system based on deep learning according to the embodiment of the present invention combines deep neural networks with traditional image processing technology through a data acquisition preprocessing module 10, a face detection module 20, a face alignment module 30, an information estimation module 40, a comprehensive evaluation module 50 and a judgment module 60. The information estimation module 40 estimates eye information, head posture information and mouth information to perform fatigue evaluation on multiple indicators. The fatigue status of a person can be detected in real time, the detection efficiency is significantly improved, and the recognition accuracy is effectively improved.

[0108] In addition, the present invention also provides a computer storage medium, which includes one or more computer instructions, and the one or more computer instructions implement any of the above-mentioned deep learning-based fatigue state identification methods when executed.

[0109] That is to say, the computer storage medium stores a computer program, and when the computer program is executed by the processor, the processor executes any of the above-mentioned fatigue state identification methods based on deep learning.

[0110] like Figure 4 As shown, an embodiment of the present invention provides an electronic device 300, including a memory 310 and a processor 320, wherein the memory 310 is used to store one or more computer instructions, and the processor 320 is used to call and execute the one or more computer instructions, so as to implement any of the methods described above.

[0111] That is, the electronic device 300 includes: a processor 320 and a memory 310, wherein the memory 310 stores computer program instructions, wherein when the computer program instructions are executed by the processor, the processor 320 executes any of the above methods.

[0112] Furthermore, if Figure 4 As shown, the electronic device 300 further includes a network interface 330 , an input device 340 , a hard disk 350 , and a display device 360 ​​.

[0113] The above-mentioned interfaces and devices can be interconnected through a bus architecture. The bus architecture can be a bus and bridge that can include any number of interconnected buses. Specifically, one or more central processing units (CPUs) represented by processor 320 and various circuits of one or more memories represented by memory 310 are connected together. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits together. It can be understood that the bus architecture is used to achieve connection and communication between these components. In addition to the data bus, the bus architecture also includes a power bus, a control bus, and a status signal bus, which are all well known in the art, so they will not be described in detail herein.

[0114] The network interface 330 can be connected to a network (such as the Internet, a local area network, etc.), obtain relevant data from the network, and save it in the hard disk 350.

[0115] The input device 340 can receive various instructions input by the operator and send them to the processor 320 for execution. The input device 340 can include a keyboard or a pointing device (for example, a mouse, a trackball, a touch pad or a touch screen).

[0116] The display device 360 ​​can display the result obtained by the processor 320 executing the instruction.

[0117] The memory 310 is used to store programs and data necessary for the operation of the operating system, as well as data such as intermediate results during the calculation process of the processor 320.

[0118] It is understood that the memory 310 in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. The memory 310 of the apparatus and method described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0119] In some implementations, the memory 310 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system 311 and application programs 312 .

[0120] The operating system 311 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 312 includes various application programs, such as a browser, etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 312.

[0121] The method disclosed in the above embodiment of the present invention can be applied to the processor 320, or implemented by the processor 320. The processor 320 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 320 or the instruction in the form of software. The above processor 320 can be a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be combined to execute. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 310, and the processor 320 reads the information in the memory 310 and completes the steps of the above method in combination with its hardware.

[0122] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0123] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0124] Specifically, the processor 320 is further configured to read the computer program and execute any of the above methods.

[0125] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0126] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0127] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0128] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A fatigue state recognition method based on deep learning, characterized in that: The following steps are involved: S1, obtaining video stream data, and obtaining a face image from the video stream data; S2, performing face detection and face key point detection on the face image; S3, obtaining facial feature information of the face image through a face alignment model; S4, performing gaze estimation, head posture estimation and mouth estimation on the facial feature information to obtain gaze estimation information, head posture estimation information and mouth estimation information respectively; S5, combining the gaze estimation information, the head posture estimation information and the mouth estimation information to obtain a fatigue state recognition comprehensive evaluation value; S6, comparing the comprehensive evaluation value with a set value to determine whether the person corresponding to the face image is in a fatigue state; In step S4, the gaze estimation comprises the following steps: S41, obtaining face information from the facial information features; S42, preliminarily determining the eye position of a person according to the face information; S43, using SAGE algorithm to locate the human eye for the second time and determine the position of the human eye; S44, performing human eye gaze estimation analysis; S45, within the screen area, defining an attention concentration area, and if the sight line of the human eye exceeds the attention concentration area within a preset time, it is determined that the person corresponding to the face information is in a fatigue state; In step S44, an improved SAGE algorithm is used to perform eye gaze estimation analysis, wherein the improved SAGE algorithm includes the following steps: S441, performing face detection to obtain key point information of the face; S442, preliminarily obtaining the eye position according to the facial key point information; S443, using the position coordinates of the four corners of the human eye to accurately locate the position of the human eye; S444, using the SAGE algorithm to estimate the sight line, obtain the gaze point of the human eye, and identify the focus of the human eye in the set area; In step S4, the head posture estimation comprises the following steps: S41', obtaining face information from the facial information features; S42', detecting and locating the head position according to the face information; S43', using FSA-Net algorithm to obtain head posture, and obtain the head posture Euler angles yaw, pitch, roll; S44', analyzing the head posture; S45', setting parameter thresholds, if the angle changes in the Pitch and Roll directions within a preset time exceed a preset time ratio, it is determined that the person corresponding to the face information is in a fatigue state.

2. The method according to claim 1, characterized in that Step S2 includes: S21, preprocessing the face image; S22, marking multiple key points and face frames of the face in the face image to obtain marking data; S23, forming a training set with the labeled data and performing training to obtain a facial key point detection model; S24. Performing face detection and face key point detection on the face image using the face key point detection model.

3. The method according to claim 1, characterized in that Step S43' includes: S431', extracting facial features according to the face information; S432', fusing features through a convolutional neural network, generating a feature map, and generating a feature vector; S433', generating a parameter set through the feature vector; S434', convert the parameter set into head posture angles to obtain yaw, pitch, and roll angles.

4. The method according to claim 1, characterized in that The mouth estimation includes: S51, preliminarily judging the mouth opening degree in the face information according to a set first threshold, if the first threshold is greater than a first set threshold, judging that the mouth is in an open state, counting the duration of the open mouth state within a preset time, if the duration exceeds a preset time ratio, judging that the person corresponding to the face information is in a fatigue state; S52. Perform a secondary judgment on a second threshold value that is smaller than the first threshold value. If the second threshold value is larger than a second set threshold value, and the mouth is opened continuously for more than a preset number of frames in the video stream, it is determined that the person corresponding to the facial information is in a fatigue state.

5. A fatigue state recognition system based on deep learning using the method of claim 1, characterized in that: include: A data acquisition module, wherein the data acquisition module acquires video stream data and obtains a face image from the video stream data; A face detection module, which performs face detection and face key point detection on the face image; A face alignment module, which obtains facial feature information of the face image through a face alignment model; An information estimation module, wherein the information estimation module performs gaze estimation, head posture estimation and mouth estimation on the facial feature information to obtain gaze estimation information, head posture estimation information and mouth estimation information respectively; A comprehensive evaluation module, wherein the comprehensive evaluation module combines the gaze estimation information, the head posture estimation information, and the mouth estimation information to obtain a comprehensive evaluation value for fatigue state recognition; A judgment module compares the comprehensive evaluation value with a set value to determine whether the person corresponding to the face image is in a fatigue state.

6. A computer storage medium, characterized in that: The method comprises one or more computer instructions, which implement the method according to any one of claims 1 to 4 when executed.

7. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer instructions; The processor is used to call and execute the one or more computer instructions, so as to implement the method according to any one of claims 1 to 4.

Images