Driver state real-time monitoring method and system

Abstract

The invention provides a driver state real-time monitoring method and system, and the method comprises the steps: obtaining a front image of a driver, and extracting face key points of the driver from the front image in real time; visual distraction detection of the driver is completed according to the head posture and the eye gazing direction; completing drowsiness detection of the driver according to the eye movement state and the mouth movement state; a multi-task neural network model is designed, and abnormal behavior detection and emotion recognition of the driver are completed according to the face image; and a result display interface is developed, and results of visual distraction detection, drowsiness detection, abnormal behavior detection and emotion recognition are displayed in the same display terminal at the same time. According to the method, drowsiness, visual distraction, emotion and abnormal behaviors of the driver can be efficiently monitored at the same time, occupation of computing resources is remarkably reduced, the strict requirements for light weight and real-time performance are met, an automatic driving system is promoted to develop in the safer and more intelligent direction, and meanwhile the use experience of a user is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a method and system for real-time monitoring of driver status. Background Technology

[0002] With the advancement of technology and the rapid development of productivity, people have made significant progress in the field of intelligent driving technology, and it has already been applied to some extent in existing cars, which has made people's lives more convenient.

[0003] In order to ensure the safety of the driver and other people in the vehicle, existing technologies can monitor the driver's driving status and issue corresponding warning information when abnormal driving status is detected, so as to reduce the risk of accidents.

[0004] Furthermore, in the process of real-time monitoring of drivers, existing technologies mostly require the pre-training of multiple monitoring models, and each monitoring model can only complete a single monitoring task. Although this monitoring method can achieve driver status monitoring to a certain extent, it requires a lot of computing resources and a large amount of storage space, which reduces monitoring efficiency. Summary of the Invention

[0005] Therefore, the purpose of this invention is to provide a method and system for real-time monitoring of driver status, so as to solve the problem that the existing technology requires a lot of computing resources and a lot of storage space during the monitoring process.

[0006] The first aspect of the present invention proposes: A method for real-time monitoring of driver status, wherein the method includes: A frontal image corresponding to the driver is acquired in real time using a preset acquisition device, and key facial points of the driver are extracted in real time from the frontal image. The driver's head posture and eye gaze direction are obtained in real time based on the facial key points, and the driver's visual distraction detection is completed based on the head posture and eye gaze direction. The driver's eye movement and mouth movement are acquired in real time based on the facial key points, and the driver's drowsiness is detected based on the eye movement and mouth movement. Design a multi-task neural network model to detect abnormal behavior and recognize emotions of the driver based on the facial image; The development results display interface displays the results of visual distraction detection, drowsiness detection, abnormal behavior detection, and emotion recognition simultaneously within the same display terminal.

[0007] The beneficial effects of this invention are: by acquiring a frontal image of the driver in real time, the key facial points of the driver can be determined in real time based on the current facial image. At the same time, the driver's visual distraction detection, emotion detection, drowsiness detection, and abnormal behavior detection can be completed based on the current facial key points and the current action image. Thus, multiple detections of the driver can be completed using the same data, which significantly reduces the amount of data processing and improves the monitoring efficiency of the driver.

[0008] Furthermore, the step of acquiring the driver's head posture and eye gaze direction in real time based on the facial key points includes: When the facial key points are determined in real time, facial features corresponding to the facial key points are detected in the facial image in real time. In the facial image, the three-dimensional coordinates of the head corresponding to the driver are created in real time based on the facial features; The driver's head posture and eye gaze direction are obtained in real time based on the three-dimensional head coordinates, and the three-dimensional head coordinates are unique.

[0009] Furthermore, the step of detecting driver drowsiness based on eye movement and mouth movement includes: When the eye movement state and the mouth movement state are obtained respectively, the driver's eye closing duration and blinking frequency are detected in real time based on the eye movement state. The driver's yawning frequency is detected in real time based on the mouth movement state, and the driver's drowsiness is detected based on the duration of eye closure, the blinking frequency, and the yawning frequency.

[0010] Furthermore, the step of detecting driver drowsiness based on the duration of eye closure, the blinking frequency, and the yawning frequency includes: In the facial image, the eyes corresponding to the driver are detected in real time, and the aspect ratio corresponding to the eyes is calculated in real time. It is also determined in real time whether the aspect ratio of the eyes is greater than a preset first threshold. If it is determined in real time that the aspect ratio of the eye is greater than the preset first threshold, it is determined that the eye is currently in a closed state, and the corresponding closed eye duration is calculated in real time, so as to complete the driver's drowsiness detection according to the closed eye duration, the blinking frequency and the yawning frequency.

[0011] Furthermore, the step of detecting driver drowsiness based on the duration of eye closure, the blinking frequency, and the yawning frequency includes: When the eye is detected to be closed in real time, the target time interval between two consecutive closed eye states is detected in real time. The blinking frequency corresponding to the driver is calculated based on the target time interval, so as to complete the driver's drowsiness detection based on the eye closure duration, the blinking frequency, and the yawning frequency.

[0012] Furthermore, the step of detecting driver drowsiness based on the duration of eye closure, the blinking frequency, and the yawning frequency includes: In the facial image, the mouth corresponding to the driver is detected in real time, and the aspect ratio corresponding to the mouth is calculated in real time. It is also determined in real time whether the aspect ratio of the mouth is greater than a preset second threshold. If it is determined in real time that the aspect ratio of the mouth is greater than the preset second threshold, then it is determined that the driver is in a yawning state, and the duration of a single yawn is recorded in real time. The yawning frequency is calculated based on the duration of a single yawn, so as to complete the driver's drowsiness detection according to the duration of closing the eyes, the performance frequency and the yawning frequency.

[0013] Furthermore, the step of detecting driver drowsiness based on the duration of eye closure, the blinking frequency, and the yawning frequency includes: When the duration of eye closure, blinking frequency, and yawning frequency are obtained respectively, a drowsiness index corresponding to the driver is calculated in real time based on the duration of eye closure, blinking frequency, and yawning frequency using a preset algorithm, and it is determined in real time whether the drowsiness index is greater than a preset index threshold. If the drowsiness index is determined to be greater than the preset index threshold in real time, the driver is determined to be in a drowsy state, and a corresponding warning message is issued.

[0014] The second aspect of the present invention proposes: A real-time driver status monitoring system, wherein the system includes: The acquisition module is used to acquire frontal images corresponding to the driver in real time through a preset acquisition device; The detection module is used to detect facial key points corresponding to the driver in real time based on the facial image; to acquire the driver's head posture and eye gaze direction in real time based on the facial key points, and to complete the driver's visual distraction detection based on the head posture and eye gaze direction; to acquire the driver's eye movement state and mouth movement state in real time based on the facial key points, and to complete the driver's drowsiness detection based on the eye movement state and mouth movement state; and to design a multi-task neural network to complete the driver's abnormal behavior detection and emotion recognition based on the facial image. The display module is used to develop a result display interface, which simultaneously displays the results of visual distraction detection, drowsiness detection, abnormal behavior detection, and emotion recognition within the same display terminal.

[0015] Furthermore, the step of acquiring the driver's head posture and eye gaze direction in real time based on the facial key points includes: When the facial key points are determined in real time, facial features corresponding to the facial key points are detected in the facial image in real time. In the facial image, the three-dimensional coordinates of the head corresponding to the driver are created in real time based on the facial features; The driver's head posture and eye gaze direction are obtained in real time based on the three-dimensional head coordinates, and the three-dimensional head coordinates are unique.

[0016] Furthermore, the step of detecting driver drowsiness based on eye movement and mouth movement includes: When the eye movement state and the mouth movement state are obtained respectively, the driver's eye closing duration and blinking frequency are detected in real time based on the eye movement state. The driver's yawning frequency is detected in real time based on the mouth movement state, and the driver's drowsiness is detected based on the duration of eye closure, the blinking frequency, and the yawning frequency.

[0017] Furthermore, the step of detecting driver drowsiness based on the duration of eye closure, the blinking frequency, and the yawning frequency includes: In the facial image, the eyes corresponding to the driver are detected in real time, and the aspect ratio corresponding to the eyes is calculated in real time. It is also determined in real time whether the aspect ratio of the eyes is greater than a preset first threshold. If it is determined in real time that the aspect ratio of the eye is greater than the preset first threshold, it is determined that the eye is currently in a closed state, and the corresponding closed eye duration is calculated in real time, so as to complete the driver's drowsiness detection according to the closed eye duration, the blinking frequency and the yawning frequency.

[0018] Furthermore, the step of detecting driver drowsiness based on the duration of eye closure, the blinking frequency, and the yawning frequency includes: When the eye is detected to be closed in real time, the target time interval between two consecutive closed eye states is detected in real time. The blinking frequency corresponding to the driver is calculated based on the target time interval, so as to complete the driver's drowsiness detection based on the eye closure duration, the blinking frequency, and the yawning frequency.

[0019] Furthermore, the step of detecting driver drowsiness based on the duration of eye closure, the blinking frequency, and the yawning frequency includes: In the facial image, the mouth corresponding to the driver is detected in real time, and the aspect ratio corresponding to the mouth is calculated in real time. It is also determined in real time whether the aspect ratio of the mouth is greater than a preset second threshold. If it is determined in real time that the aspect ratio of the mouth is greater than the preset second threshold, then it is determined that the driver is in a yawning state, and the duration of a single yawn is recorded in real time. The yawning frequency is calculated based on the duration of a single yawn, so as to complete the driver's drowsiness detection according to the duration of closing the eyes, the performance frequency and the yawning frequency.

[0020] Furthermore, the step of detecting driver drowsiness based on the duration of eye closure, the blinking frequency, and the yawning frequency includes: When the duration of eye closure, blinking frequency, and yawning frequency are obtained respectively, a drowsiness index corresponding to the driver is calculated in real time based on the duration of eye closure, blinking frequency, and yawning frequency using a preset algorithm, and it is determined in real time whether the drowsiness index is greater than a preset index threshold. If the drowsiness index is determined to be greater than the preset index threshold in real time, the driver is determined to be in a drowsy state, and a corresponding warning message is issued.

[0021] The third aspect of the present invention proposes: A computer includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the real-time driver status monitoring method as described above.

[0022] The fourth aspect of the present invention proposes: A readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the real-time driver status monitoring method as described above.

[0023] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] Figure 1 This is a flowchart of a real-time driver status monitoring method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of facial key points detected in a real-time driver status monitoring method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the three-dimensional head coordinates in a driver status real-time monitoring method provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the eye gaze direction calculation process in a real-time driver status monitoring method provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the eye aspect ratio calculation process in a real-time driver status monitoring method provided in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the mouth aspect ratio calculation process in a real-time driver status monitoring method provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the neural network structure in a real-time driver status monitoring method provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the display interface in a real-time driver status monitoring method provided in an embodiment of the present invention.

[0025] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0026] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0027] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0029] Please see Figure 1 The image shows a real-time driver status monitoring method provided by an embodiment of the present invention. The real-time driver status monitoring method provided by this embodiment can objectively and quickly complete the comprehensive detection of driver status, while avoiding complex calculations and redundant data, thereby improving monitoring efficiency.

[0030] Specifically, this embodiment provides: S1: A vehicle-mounted camera installed facing the driver captures a real-time image of the driver's front. It should be noted that the facial and motion images corresponding to the current driver can be extracted from this front image in real time. S2: The driver's facial landmarks in the input image are detected by the facial landmark detection algorithm. The driver's head pose and eye gaze direction are obtained based on the facial landmarks. Then, the driver's visual distraction is detected based on the head pose and eye gaze direction.

[0031] S2.1: In the actual process of visual distraction detection, the input facial image is first processed using an existing pre-trained facial landmark detection algorithm. In this example, the existing MediaPipe (a real-time media analysis tool) is used to detect the driver's facial landmarks. The obtained facial landmarks are as follows: Figure 2 As shown; S2.2: Next, as Figure 2 As shown, point E, located between the left and right ears, is a key facial feature. l and E r The line connecting E and E is approximated as the Y-axis, and the right side is defined as the positive direction of the Y-axis; l and E rThe midpoint O of the connecting line is taken as the origin of the coordinate system. The straight line connecting point O to the tip of the nose (N) is approximated as the X-axis, and the orientation of the face is defined as the positive direction of the X-axis. Furthermore, the straight line connecting point O to point C, located at the center of the forehead, is approximated as the Z-axis, and the upward direction is defined as the positive direction of the Z-axis. In this way, the driver's head posture can be approximated based on key facial points, such as... Figure 3 As shown.

[0032] S2.3: Then, take the midpoint O of each eyeball. r Pointing to the midpoint O of the line connecting the left and right corners of the eyes e The direction of gaze is approximated as the direction of eye fixation, thus allowing the gaze direction to be determined based on key facial features, such as... Figure 4 As shown.

[0033] S2.4: Let h be the angle of rotation of the head around the Y-axis. y The rotation angle around the Z-axis is denoted as h. z And the angle of rotation of the eye's gaze direction around the Y-axis is denoted as e. y The rotation angle around the Z-axis is denoted as e. z Then the angle s of the driver's line of sight around the Y-axis y =h y +e y The angle s of the driver's line of sight around the Z-axis z =h z +e z When the driver's line of sight deflects at an angle s y s z If the threshold is exceeded, the driver is deemed to be visually distracted.

[0034] S3: Obtain the movement status of the eyes and mouth based on facial key points, and obtain information such as the driver's eye closure duration, blinking frequency, and yawning frequency based on the movement of the eyes and mouth, and then realize the driver's drowsiness detection based on this information.

[0035] S3.1: First, extract the points on the outer contours of the two eyes from the detected facial key points, and then extract the point E at the left corner of each eye. l And point E at the right corner of the eye r The distance between them is denoted as L. eh At the same time, the midpoint E of the upper contour of each eye u Midpoint E of the lower profile d The distance between them is denoted as L. ev ,like Figure 5 As shown. Then the aspect ratio of the eye... When both eyes R e If the average value is greater than the set threshold, it is determined that the driver has closed their eyes, and the duration T of each eye closure is recorded. c .

[0036] S3.2: Then, record the time interval between two consecutive times the eyes are closed. The blinking frequency is .

[0037] S3.3: Next, extract the points on the outer contour of the mouth from the detected facial key points, and select the leftmost point of the corner of the mouth, M. l And the rightmost point M of the corner of the mouth r The distance between them is denoted as L. mh At the same time, the midpoint M of the upper contour of the mouth u Midpoint M of the lower profile d The distance between them is denoted as L. mv ,like Figure 6 As shown. Then the length-to-width ratio of the mouth... When R m If the yawn duration is less than the set threshold, it is determined that the driver is yawning, and the duration T of a single yawn is recorded. y .

[0038] S3.4: The driver's drowsiness index is (in α 1, α 2, α 3 represents the eye-closing coefficient, blinking coefficient, and yawning coefficient, respectively. Drows If the threshold is exceeded, the driver's current state is determined to be drowsy.

[0039] S4: In addition, the method fine-tunes a neural network model by simultaneously using driver facial expression data and abnormal behavior data, so that the model can simultaneously realize driver emotion recognition and driver abnormal behavior detection.

[0040] S4.1: First, design a classic neural network model for classification tasks. In this example, the model structure is as follows: Figure 7 As shown, the network parameters are loaded from ImageNet (an image classification database) and are fully pre-trained.

[0041] S4.2: Mix the training sets for driver emotion recognition and abnormal behavior recognition together while fine-tuning the model.

[0042] S4.3: After sufficient fine-tuning, the model can simultaneously detect the driver's facial expressions and abnormal behavior.

[0043] S5: The system's results display module has developed a visual interface, such as... Figure 8 As shown, the results of the calculation module's real-time detection of the driver's status are displayed on the monitor.

[0044] In summary, the real-time driver status monitoring method and system provided by the above embodiments of the present invention can quickly and effectively complete the all-round detection of the driver, while effectively avoiding complex calculations and redundant data, thereby improving monitoring efficiency.

[0045] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0046] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0047] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0048] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0049] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0050] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for real-time monitoring of driver status, characterized in that, The method includes: A frontal image corresponding to the driver is acquired in real time using a preset acquisition device, and key facial points of the driver are extracted in real time from the frontal image. The driver's head posture and eye gaze direction are obtained in real time based on the facial key points, and the driver's visual distraction detection is completed based on the head posture and eye gaze direction. The driver's eye movement and mouth movement are acquired in real time based on the facial key points, and the driver's drowsiness is detected based on the eye movement and mouth movement. Design a multi-task neural network model to detect abnormal behavior and recognize emotions of the driver based on the facial image; The development results display interface displays the results of visual distraction detection, drowsiness detection, abnormal behavior detection, and emotion recognition simultaneously within the same display terminal.

2. The method for real-time monitoring of driver status according to claim 1, characterized in that: The step of obtaining the driver's head posture and eye gaze direction in real time based on the facial key points includes: When the facial key points are determined in real time, facial features corresponding to the facial key points are detected in the facial image in real time. In the facial image, the three-dimensional coordinates of the head corresponding to the driver are created in real time based on the facial features; The driver's head posture and eye gaze direction are obtained in real time based on the head's three-dimensional coordinates, and the head's three-dimensional coordinates are unique.

3. The method for real-time monitoring of driver status according to claim 1, characterized in that: The step of detecting driver drowsiness based on eye movement and mouth movement includes: When the eye movement state and the mouth movement state are obtained respectively, the driver's eye closing duration and blinking frequency are detected in real time based on the eye movement state. The driver's yawning frequency is detected in real time based on the mouth movement state, and the driver's drowsiness is detected based on the duration of eye closure, the blinking frequency, and the yawning frequency.

4. The method for real-time monitoring of driver status according to claim 3, characterized in that: The step of detecting driver drowsiness based on the duration of eye closure, the blinking frequency, and the yawning frequency includes: In the facial image, the eyes corresponding to the driver are detected in real time, and the aspect ratio corresponding to the eyes is calculated in real time. It is also determined in real time whether the aspect ratio of the eyes is greater than a preset first threshold. If it is determined in real time that the aspect ratio of the eye is greater than the preset first threshold, it is determined that the eye is currently in a closed state, and the corresponding closed eye duration is calculated in real time, so as to complete the driver's drowsiness detection according to the closed eye duration, the blinking frequency and the yawning frequency.

5. The method for real-time monitoring of driver status according to claim 4, characterized in that: The step of detecting driver drowsiness based on the duration of eye closure, the blinking frequency, and the yawning frequency includes: When the eye is detected to be closed in real time, the target time interval between two consecutive closed eye states is detected in real time. The blinking frequency corresponding to the driver is calculated based on the target time interval, so as to complete the driver's drowsiness detection based on the eye closure duration, the blinking frequency, and the yawning frequency.

6. The method for real-time monitoring of driver status according to claim 5, characterized in that: The step of detecting driver drowsiness based on the duration of eye closure, the blinking frequency, and the yawning frequency includes: In the facial image, the mouth corresponding to the driver is detected in real time, and the aspect ratio corresponding to the mouth is calculated in real time. It is also determined in real time whether the aspect ratio of the mouth is greater than a preset second threshold. If it is determined in real time that the aspect ratio of the mouth is greater than the preset second threshold, then it is determined that the driver is in a yawning state, and the duration of a single yawn is recorded in real time. The yawning frequency is calculated based on the duration of a single yawn, so as to complete the driver's drowsiness detection according to the duration of closing the eyes, the performance frequency and the yawning frequency.

7. The method for real-time monitoring of driver status according to claim 6, characterized in that: The step of detecting driver drowsiness based on the duration of eye closure, the blinking frequency, and the yawning frequency includes: When the duration of eye closure, blinking frequency, and yawning frequency are obtained respectively, a drowsiness index corresponding to the driver is calculated in real time based on the duration of eye closure, blinking frequency, and yawning frequency using a preset algorithm, and it is determined in real time whether the drowsiness index is greater than a preset index threshold. If the drowsiness index is determined to be greater than the preset index threshold in real time, the driver is determined to be in a drowsy state, and a corresponding warning message is issued.

8. A real-time driver status monitoring system, characterized in that, The system includes: The acquisition module is used to acquire frontal images corresponding to the driver in real time through a preset acquisition device; The detection module is used to detect facial key points corresponding to the driver in real time based on the facial image; to acquire the driver's head posture and eye gaze direction in real time based on the facial key points, and to complete the driver's visual distraction detection based on the head posture and eye gaze direction; to acquire the driver's eye movement state and mouth movement state in real time based on the facial key points, and to complete the driver's drowsiness detection based on the eye movement state and mouth movement state; and to design a multi-task neural network to complete the driver's abnormal behavior detection and emotion recognition based on the facial image. The display module is used to develop a result display interface, which simultaneously displays the results of visual distraction detection, drowsiness detection, abnormal behavior detection, and emotion recognition within the same display terminal.

9. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the real-time driver status monitoring method as described in any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the real-time driver status monitoring method as described in any one of claims 1 to 7.

Images