Fatigue detection method, device and vehicle

Abstract

The application discloses a fatigue detection method and device and a vehicle. The method comprises the following steps: acquiring a plurality of target images including a target user within a first time length; extracting target features of the target images based on the face of the target user; generating a target feature sequence based on the target features of the plurality of target images; inputting the target feature sequence into a state detection model to output a state detection result of the target user within the first time length, wherein the state detection model is trained based on a plurality of first time lengths of a plurality of images of a known user state; and determining a fatigue detection result of the target user according to state detection results corresponding to a plurality of first time lengths within a second time length, wherein the second time length is equal to the total time length of the plurality of first time lengths. The fatigue detection result is determined according to the state detection results of the plurality of first time lengths within the second time length, the anti-interference capability of fatigue detection under complex environmental changes is improved, and the stability and accuracy of fatigue detection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of assisted driving, and in particular to a fatigue detection method and device and a vehicle. BACKGROUND

[0002] With the rapid development of the transportation industry in China, the number of vehicles as a means of transportation is increasing, and major traffic accidents caused by vehicle driving also occur more frequently. Fatigue driving is one of the main causes of traffic accidents.

[0003] At present, the fatigue detection method mainly detects the single frame image of the collected driver first, and then judges or classifies fatigue according to the opening and closing of the eyes, yawning and other conditions in a given period. However, in the actual driving process, the light conditions and driver facial poses in the vehicle are complex and changeable. The fatigue detection method in the related art cannot adapt to the complex light changes and complex and changeable driver facial poses in the actual driving process when applied to the fatigue detection process of the driver in the vehicle, and is prone to misjudgment or omission. Therefore, the anti-interference ability and accuracy of the detection need to be improved. SUMMARY

[0004] The embodiments of the present application provide a fatigue detection method, device and vehicle, which improve the anti-interference ability of fatigue detection in complex environments and the accuracy of fatigue detection. The technical solution is as follows:

[0005] In a first aspect, the embodiments of the present application provide a fatigue detection method, and the method comprises:

[0006] Obtaining a plurality of target images in a first time period; the target images include a target user;

[0007] Extracting target features of the target images based on the face of the target user;

[0008] Generating a target feature sequence based on the target features of the plurality of target images;

[0009] Inputting the target feature sequence into a state detection model to output a state detection result of the target user in the first time period; the state detection model is obtained by training based on a plurality of first time periods of a plurality of images of a known user state;

[0010] Determining a fatigue detection result of the target user according to the state detection results corresponding to a plurality of first time periods in a second time period; the second time period is equal to the total time period of the plurality of first time periods.

[0011] In a possible implementation manner, the target features include at least one of the following: target yawning state features, target head state features, and target eye state features.

[0012] The target feature of the target image is extracted based on the face of the target user, and includes:

[0013] A target face image and / or a target eye image of the target user are determined based on a preset detection algorithm and the target image;

[0014] A target yawning state feature of the target face image is extracted using a first model; and / or

[0015] A target head state feature of the target face image is extracted using a second model; and / or

[0016] A target eye state feature of the target eye image is extracted using a third model.

[0017] The first model is obtained by contrast learning training based on a plurality of face images with known yawning states; the second model is obtained by regression training based on a plurality of face images with known head states; and the third model is obtained by contrast learning training based on a plurality of eye images with known eye states.

[0018] In a possible implementation, the target feature sequence includes a target yawning state feature sequence, a target head state feature sequence, and / or a target eye state feature sequence.

[0019] In a possible implementation, the target eye image includes a target left eye image and a target right eye image; the target eye state feature includes a target left eye state feature and a target right eye state feature; and the target eye state feature sequence includes a target left eye state feature sequence and a target right eye state feature sequence.

[0020] After the target feature sequence is generated based on the target features of a plurality of target images within a first time length, before the target feature sequence is input into a state detection model and a state detection result of the target user within the first time length is output, the method further includes:

[0021] The target feature sequence is processed by dimension reduction to obtain a first target feature sequence;

[0022] The first target feature sequence is restored to the same dimension as the target feature sequence to obtain a second target feature sequence;

[0023] The target feature sequence is input into the state detection model, and a state detection result of the target user within the first time length is output, including:

[0024] The second target feature sequence is input into a state detection model, and a state detection result of the target user in the first time length is output.

[0025] In a possible implementation, the state detection result includes a head state detection result, a yawning state detection result, and / or an eye state detection result; the head state detection result includes a head posture Euler angle of the target user in the first time length; the yawning state detection result includes a probability that the target user is in a yawning state in the first time length; and the eye state detection result includes a probability that the target user is in a closed-eye state in the first time length.

[0026] In a possible implementation, the fatigue detection result includes a fatigue degree distribution result; and the fatigue degree distribution result includes at least one fatigue degree of the target user in the second time length and a probability corresponding to the fatigue degree.

[0027] The fatigue detection result of the target user is determined according to the state detection results corresponding to the plurality of first time lengths in the second time length, and includes:

[0028] Fatigue feature information in the state detection results corresponding to the plurality of first time lengths in the second time length is extracted.

[0029] The fatigue feature information is input into a fourth model, and a fatigue degree distribution result of the target user in the second time length is output; the fourth model is obtained by training based on a plurality of second time length video images with known fatigue degrees; and the fatigue degrees of the second time length video images include fatigue degrees annotated by a plurality of annotators on the video images.

[0030] In a possible implementation, the fatigue feature information includes at least one of the following: a total closed-eye time length of the target user in the second time length, an average time length of each closed-eye, a blink frequency, a total yawning time length, a mean and variance of a head posture Euler angle, a percentage of a head pitch angle less than a first threshold value, and a percentage of a head yaw angle less than a second threshold value or greater than a third threshold value.

[0031] In a second aspect, an embodiment of the present application provides a fatigue detection device, including:

[0032] A first time length is obtained, and a plurality of target images in the first time length are obtained; the target images include a target user.

[0033] A feature extraction module is configured to extract target features of the target images based on a face of the target user.

[0034] The generating module is configured to generate a target feature sequence based on the target features of the multiple target images;

[0035] The state detection module is configured to input the target feature sequence into a state detection model, and output a state detection result of the target user within the first time length; the state detection model is trained based on multiple images within multiple first time lengths of known user states.

[0036] The determining module is configured to determine a fatigue detection result of the target user according to the state detection results corresponding to the continuous multiple first time lengths within a second time length; the second time length is equal to the total time length of the multiple first time lengths.

[0037] In a possible implementation, the target features include at least one of the following: a target yawning state feature, a target head state feature, and a target eye state feature.

[0038] The feature extraction module includes:

[0039] The determining unit is configured to determine a target face image and / or a target eye image of the target user based on a preset detection algorithm and the target image.

[0040] The first extraction unit is configured to extract a target yawning state feature of the target face image by using a first model; and / or

[0041] The second extraction unit is configured to extract a target head state feature of the target face image by using a second model; and / or

[0042] The third extraction unit is configured to extract a target eye state feature of the target eye image by using a third model.

[0043] The first model is obtained by contrast learning training based on multiple face images of known yawning states; the second model is obtained by regression training based on multiple face images of known head states; and the third model is obtained by contrast learning training based on multiple eye images of known eye states.

[0044] In a possible implementation, the target feature sequence includes a target yawning state feature sequence, a target head state feature sequence, and / or a target eye state feature sequence.

[0045] In a possible implementation, the target eye image includes a target left eye image and a target right eye image; the target eye state feature includes a target left eye state feature and a target right eye state feature; and the target eye state feature sequence includes a target left eye state feature sequence and a target right eye state feature sequence.

[0046] In a possible implementation, the fatigue detection apparatus further includes:

[0047] a dimension reduction processing module, configured to perform dimension reduction processing on the target feature sequence to obtain a first target feature sequence;

[0048] a restoration module, configured to restore the first target feature sequence to the same dimension as the target feature sequence to obtain a second target feature sequence;

[0049] The state detection module is specifically configured to:

[0050] input the second target feature sequence into a state detection model, and output a state detection result of the target user in the first time length.

[0051] In a possible implementation, the state detection result includes a head state detection result and / or a yawn state detection result and / or an eye state detection result; the head state detection result includes a head posture Euler angle of the target user in the first time length; the yawn state detection result includes a probability of the target user being in a yawn state in the first time length; and the eye state detection result includes a probability of the target user being in a closed-eye state in the first time length.

[0052] In a possible implementation, the fatigue detection result includes a fatigue degree distribution result; and the fatigue degree distribution result includes at least one fatigue degree of the target user in the second time length and a probability corresponding to the fatigue degree.

[0053] The determination module includes:

[0054] a fourth extraction unit, configured to extract fatigue feature information in the state detection result corresponding to the continuous multiple first time lengths in the second time length;

[0055] a fatigue degree distribution detection unit, configured to input the fatigue feature information into a fourth model to output a fatigue degree distribution result of the target user in the second time length; the fourth model is obtained by training based on a plurality of second time length video images with known fatigue degrees; and the fatigue degrees of the second time length video images include fatigue degrees annotated by a plurality of annotators on the video images.

[0056] In a possible implementation, the fatigue feature information includes at least one of the following: a total closed-eye time length of the target user in the second time length, an average time length of each closed-eye, a blink frequency, a total yawn time length, a mean and variance of a head posture Euler angle, a percentage of a head pitch angle less than a first threshold value, and a percentage of a head yaw angle less than a second threshold value or greater than a third threshold value.

[0057] In a third aspect, an embodiment of the present application provides a vehicle, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is adapted to be loaded by the processor and execute the method steps provided in the first aspect or any possible implementation manner of the first aspect of the present application.

[0058] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a plurality of instructions, and the instructions are adapted to be loaded by a processor and execute the method steps provided in the first aspect or any possible implementation manner of the first aspect of the present application.

[0059] The technical scheme provided by some embodiments of the present application has at least the following beneficial effects:

[0060] In one or more embodiments of the present application, first, a plurality of target images including a target user in a first time length are acquired, and a target feature of the target image is extracted based on the face of the target user, then a target feature sequence is generated based on the target features of the plurality of target images, and the target feature sequence is input into a state detection model to output a state detection result of the target user in the first time length, the state detection model is trained based on a plurality of first time lengths of a plurality of images of a known user state, finally, a fatigue detection result of the target user is determined according to state detection results corresponding to a plurality of first time lengths in a second time length, and the second time length is equal to the total time length of the plurality of first time lengths. Compared with state detection or fatigue detection based on a single target image, the accuracy is not high due to the influence of complex environment. In the embodiments of the present application, the state detection result of the target user in the first time length is determined based on the target feature sequence generated by the target features of the plurality of target images in the first time length, and the fatigue detection result of the target user in the second time length is determined according to the states corresponding to the plurality of first time lengths in the second time length. The fatigue detection result can be determined by using the plurality of target images in the first time length and the state detection results corresponding to the plurality of first time lengths respectively, so as to improve the anti-interference ability of fatigue detection under complex environment changes, thereby improving the stability of fatigue detection, and improving the accuracy of fatigue detection.

[0061] The above description is only a summary of the technical scheme of the present application. In order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0063] Figure 1 A schematic diagram of a fatigue detection process provided in the related art;

[0064] Figure 2A A structural schematic diagram of a fatigue detection system provided in an example embodiment of the present application;

[0065] Figure 2B A schematic diagram of an application scenario of a fatigue detection method provided in an example embodiment of the present application;

[0066] Figure 3 A flowchart of a fatigue detection method provided in an example embodiment of the present application;

[0067] Figure 4A A structural schematic diagram of a first model provided in an example embodiment of the present application;

[0068] Figure 4B A structural schematic diagram of a second model provided in an example embodiment of the present application;

[0069] Figure 4C A structural schematic diagram of a third model provided in an example embodiment of the present application;

[0070] Figure 5 A schematic diagram of an implementation process of determining a state detection result of a target user in a first time length (target user state detection) in a fatigue detection method provided in an example embodiment of the present application;

[0071] Figure 6 A schematic diagram of an implementation flow of determining a fatigue detection result of a target user according to a state detection result in a fatigue detection method provided in an example embodiment of the present application;

[0072] Figure 7 A schematic diagram of an implementation process of a fatigue detection method provided in an example embodiment of the present application;

[0073] Figure 8 A schematic diagram of an implementation process of extracting fatigue feature information from a state detection result provided in an example embodiment of the present application;

[0074] Figure 9 A flowchart of another fatigue detection method provided in an example embodiment of the present application;

[0075] Figure 10 FIG. 1 shows a structural schematic diagram of a fatigue detection device according to an example embodiment of the present application;

[0076] Figure 11 FIG. 2 shows a structural schematic diagram of a vehicle according to an example embodiment of the present application. DETAILED DESCRIPTION

[0077] In order to make the features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0078] The terms "first", "second", "third", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include other steps or units not listed or can optionally include other steps or units inherent to the process, method, product or device.

[0079] At present, as shown in Figure 1 Related fatigue detection methods mainly determine fatigue detection results directly according to eye features or mouth features in a single target image of a target user collected, for example, first determine whether the target user closes eyes according to eye features and whether the target user yawns according to mouth features, and then determine whether the target user is tired or the fatigue level of the target user according to whether the target user closes eyes and whether the target user yawns in the single target image and the like.

[0080] However, in the actual application process of fatigue detection, the surrounding environment light conditions and the user facial posture are complex and changeable, and yawning or opening and closing eyes and the like of the user when tired are dynamic processes, therefore, the fatigue detection method in the related art as shown in Figure 1 cannot adapt to the situation that the surrounding environment light changes complexly and the user facial posture is complex and changeable in the actual application process, and it is also difficult to obtain relatively accurate eye state and yawning state and the like of the user through a single image, which will affect the stability and accuracy of fatigue detection to some extent.

[0081] Based on this, one or more embodiments are proposed in the present application. The eye state, head state, and yawning state of the target user in the first time length are determined according to the target feature sequence generated according to the target features of the multiple target images in the first time length, and the fatigue detection result of the target user in the second time length is determined according to the states corresponding to the multiple first time lengths in the second time length. This can not only improve the anti-interference ability of fatigue detection under complex environmental changes, thereby improving the stability of fatigue detection, but also improve the accuracy of fatigue detection.

[0082] Please refer to Figure 2A , Figure 2A An exemplary structure diagram of a fatigue detection system provided by an embodiment of the present application is shown.

[0083] As Figure 2A shown, the fatigue detection system can include an image acquisition device 210 and a server 220. Wherein:

[0084] The image acquisition device 210 can be a mobile phone, tablet computer, notebook computer, etc. equipped with user version software and camera, or a camera or other device such as a vehicle equipped with a camera, and the present application does not limit this.

[0085] Optionally, when it is desired to understand the fatigue condition of the target user, the target image corresponding to the target user can be acquired by the image acquisition device 210. At the same time, the image acquisition device 210 can extract the target features of the target image based on the face of the target user in the target image, and generate the target feature sequence and the state detection model based on the target features of the multiple target images in the first time length to determine the state detection result of the target user in the first time length, and then determine the fatigue detection result of the target user according to the state detection results corresponding to the multiple first time lengths in the second time length. The state detection model is trained based on multiple first time lengths of multiple images of known user states, and the second time length is equal to the total time length of the multiple first time lengths.

[0086] Optionally, after the target image of the target user is acquired by the image acquisition device 210, the image acquisition device 210 can establish a data relationship with the network, and establish a data connection relationship with the server 220 through the network, such as sending the target image or video image corresponding to the target user, receiving the fatigue detection result of the target user in the second time length equal to the total time length of the multiple first time lengths determined by the server 220 based on the multiple target images of the target user in the multiple first time lengths, etc.

[0087] The server 220 can be a server capable of providing fatigue detection, can receive data such as target images or video images sent by the image acquisition device 210 or other servers through a network, can extract target features of the target images based on a face of a target user in the target images, can generate a target feature sequence based on target features of multiple frames of target images in a first time length, and can determine a state detection result of the target user in the first time length based on a state detection model, and can then determine a fatigue detection result of the target user according to state detection results corresponding to multiple first time lengths in a second time length. The state detection model is trained based on multiple frames of images in multiple first time lengths of known user states, and the second time length is equal to the total time length of the multiple first time lengths. The server 220 can also send the determined fatigue detection result to the image acquisition device 210 or other terminals through a network, so that the target user or other users can timely understand the fatigue of the target user.

[0088] It can be understood that the target images can be sent frame by frame by the image acquisition device 210 or other servers, can be extracted at equal intervals or randomly from video images of the second time length or video images of the multiple first time lengths sent by the image acquisition device 210 or other servers, or can be each frame of image constituting the video images of the second time length or the video images of the multiple first time lengths, and the embodiments of the present application do not limit this.

[0089] It can be understood that when the target images are extracted from the video images of the second time length or the video images of the multiple first time lengths, in order to ensure the stability and accuracy of fatigue detection, the number of single-frame target images extracted in each first time length needs to be equal.

[0090] Alternatively, in order to ensure the timeliness and effectiveness of fatigue detection, when the image acquisition device 210 continuously acquires target images of the target user or the server 220 continuously receives target images from the image acquisition device 210, the image acquisition device 210 or the server 220 can extract target features of the target images based on a face of a target user in the target images after acquiring each frame of target image, and then execute the target feature sequence based on the target features of the multiple frames of target images in the first time length, and the state detection model determines the state detection result of the target user in the first time length after the first time length from the time when the first frame of target image is acquired. In succession, until the multiple first time lengths, i.e., the second time length, from the time when the first frame of target image is acquired, the fatigue detection result of the target user is determined according to the state detection results corresponding to the multiple first time lengths in the second time length.

[0091] Optionally, after the image acquisition device 210 acquires a video image of the second duration or the server 220 receives a video image of the second duration from the image acquisition device 210 or other servers, the image acquisition device 210 or the server 220 may first divide the video image into multiple video images of the first duration, then extract target features of multiple frames of target images within each first duration based on the target user's face in the video image, and determine the target user's state detection result within each first duration based on the target feature sequence (feature map) generated based on the target features of multiple frames of target images within each first duration and the aforementioned state detection model, and finally determine the target user's fatigue detection result based on the state detection results corresponding to multiple consecutive first durations within the second duration.

[0092] Understandably, the multiple target images within the first duration can be all the target images that make up the video images of the first duration, or multiple target images that are extracted at equal intervals or randomly from the video images of the first duration. This application embodiment does not limit this.

[0093] Understandably, server 220 can be, but is not limited to, a hardware server, a virtual server, a cloud server, etc.

[0094] It is understood that fatigue detection can be performed by the image acquisition device 210 or the server 220 alone, or by a combination of the image acquisition device 210 and the server 220. This application embodiment does not limit this, and the following embodiments are all described using the image acquisition device 210 as an example.

[0095] The network can be a medium that provides a communication link between either the server 220 or the terminal 210, or it can be the Internet, which includes network devices and transmission media, and is not limited thereto. The transmission media can be a wired link (e.g., but not limited to, coaxial cable, fiber optic cable, and digital subscriber line (DSL)) or a wireless link (e.g., but not limited to, wireless fidelity (WIFI), Bluetooth, and mobile device networks).

[0096] For example, Figure 2A The image acquisition device 210 in the middle can be Figure 2BIn the vehicle 210A in the vehicle 210A in the auxiliary driving scene, in order to be able to understand the state of the driver in the driving process in time, reduce the traffic accidents caused by the driver's fatigue driving, the target image 230 of the driver (target user) in the driving process can be continuously collected through the camera 211 installed on the vehicle 210A, and the fatigue detection of the driver (target user) in the driving process is realized based on the collected target image 230 and the fatigue detection method provided in the application. At the same time, once the fatigue detection result that can indicate that the driver (target user) belongs to fatigue driving appears, a prompt message can also be sent immediately, such as but not limited to voice prompt or ring prompt, etc., so as to timely remind the driver (target user) of the safety risk, so that the driver (target user) can adjust his own state or consider parking and resting in time, and the safety of the driver (target user) in driving can be improved. In the embodiment of the application, after the fatigue detection result that can indicate that the driver (target user) belongs to fatigue driving appears, the automatic driving system of the vehicle 210A can also be controlled to start, so as to further ensure the safety of the driver (target user) in driving.

[0097] The fatigue detection method described above can be used in an auxiliary teaching scene to realize timely understanding of the learning state of a student by detecting the fatigue of the student, or can be used in an auxiliary work management scene to realize timely avoiding of safety hazards caused by operating machines due to fatigue by detecting the fatigue of workers on a production line, or can be used in other scenes, which are not limited in the embodiment of the application.

[0098] It can be understood that Figure 2A It can be understood that

[0099] Next, an example of fatigue detection performed by the image collection device 210 will be described in combination with Figure 2A and Figure 2B A fatigue detection method provided in the embodiment of the application will be introduced taking the fatigue detection performed by the image collection device 210 as an example. For details, please refer to Figure 3 which is a flowchart of a fatigue detection method provided in an example embodiment of the application. As shown in Figure 3 The fatigue detection method includes the following steps:

[0100] S301, a plurality of target images in a first time length are acquired, and the target images include a target user.

[0101] Specifically, when it is intended to understand the fatigue condition of the target user, a target image of the target user can be collected by the camera, and the target image at least includes the face of the target user. The face of the target user can represent the part above the neck of the target user, or the part below the hairline, the chin and the part above the chin of the target user, and the like, which is not limited in the embodiments of the present application. The plurality of target images in the first time length can be all target images of the video image constituting the first time length, or can be a plurality of target images extracted at equal intervals or randomly from the video image of the first time length, which is not limited in the embodiments of the present application. The video image of the first time length can be a video image of any time period with a time length of the first time length, which is not limited in the embodiments of the present application. The first time length can be 0.5S, 1S, 2S, and the like, which is not limited in the embodiments of the present application.

[0102] S302, extracting a target feature of the target image based on the face of the target user.

[0103] Specifically, the target feature includes at least one of the target yawning state feature, the target head state feature and the target eye state feature.

[0104] Optionally, after the target image is obtained, at least one of the three features of the target yawning state feature, the target head state feature and the target eye state feature can be directly extracted from the target image by using a deep learning model. For example, when it is intended to extract the target yawning state feature and the target eye state feature of the target image, the target image can be directly input into the trained deep learning model, and the corresponding target yawning state feature and target eye state feature can be extracted by the convolution module of the deep learning model, and the deep learning model is obtained by contrast learning training based on images with known yawning state and eye state.

[0105] Optionally, after the target image is acquired, in order to ensure the effectiveness and accuracy of the extracted target feature, a face image of the target user can be intercepted based on the target image, which is used to extract the target yawning state feature and / or the target head state feature, and / or an eye image of the target user can be intercepted based on the target image, which is used to extract the target eye state feature, thereby avoiding the interference of other regions irrelevant to the target feature in the target image when the target feature is extracted, and greatly ensuring the effectiveness and accuracy of the target feature. At the same time, since a person will not only appear to have a mouth open or cover the mouth with hands when yawning, but also appear to have facial muscle contraction or tears in the eyes, therefore, the embodiment of the present application is different from the way of determining the target yawning state feature only through the mouth region, and the target yawning state feature is extracted from the face image of the target user, which can avoid the case that the target user is mistaken for yawning when speaking with the mouth open, ensure the comprehensiveness of the target yawning state feature, and further ensure the effectiveness and accuracy of the target yawning state feature.

[0106] Further, after the target image is acquired, the target face image of the target user can be determined based on a preset detection algorithm and the target image, that is, the face key point detection can be performed on the target image to obtain the face position of the target user in the target image, and the target face image of the target user can be intercepted from the target image according to the face position of the target user in the target image, and then the first model is used to extract the target yawning state feature of the target face image, and / or the second model is used to extract the target head state feature of the target face image; and / or after the target image is acquired, the target eye image of the target user can be determined based on a preset detection algorithm and the target image, that is, the face key point detection can be performed on the target image to obtain the eye position of the target user in the target image, and the target eye image of the target user can be intercepted from the target image according to the eye position of the target user in the target image, and then the third model is used to extract the target eye state feature of the target eye image. The above-mentioned preset detection algorithm can be a face detection algorithm or a face key point detection algorithm, and the embodiment of the present application is not limited thereto.

[0107] Further, as Figure 4AAs shown, the first model includes a first convolutional module and a first fully connected layer. When extracting the yawning state feature of the target face image using the first model, the target face image can be input into the trained first model, and then the corresponding target yawning state feature can be directly extracted by the first convolutional module of the first model. The first convolutional module includes at least one convolutional layer. The first model is obtained by supervised contrastive learning training based on a plurality of face images with known yawning states, and the loss function is a contrastive learning loss function. The yawning state is used to represent the yawning state of the user in the face image, including being in a yawning state and being in a non-yawning state. In this embodiment of the present application, the first model can be trained in a supervised contrastive learning manner, the distance between the yawning state features of the face images in the same yawning state input into the first model is narrowed, and the distance between the yawning state features of the face images in different yawning states input into the first model is widened, so that the features corresponding to different yawning states can be more obviously distinguished, and the anti-interference ability of the fatigue detection process to a complex environment is increased.

[0108] Further, as shown in Figure 4B The second model includes a second convolutional module and three second fully connected layers. When extracting the target head state feature of the target face image using the second model, the target face image can be input into the trained second model, and then the corresponding target head feature can be directly extracted by the second convolutional module of the second model. The target head state feature can include a target pitch angle feature, a target yaw angle feature, and a target roll angle feature. The third convolutional module can include at least three convolutional layers, and each target head state can be extracted from the target face image by the convolutional layer corresponding thereto. The second model is obtained by regression training based on a plurality of face images with known head states. The head state is used to represent the head posture of the user in the face image, including a pitch angle, a yaw angle, and a roll angle.

[0109] As shown in Figure 4BAs shown, since the trained second model can finally output an estimated angle value corresponding to the head of the target user in the target face image, the value range of which is [-90, 90] degrees, if it is directly regarded as a 180-classification problem, assuming that each class corresponds to a degree, no matter what the finally estimated error angle is, it will lead to the same loss between it and the real angle, for example, 0 degrees, 1 degree and 2 degrees. Obviously, 1 degree is closer to 2 degrees than 0 degrees, and if the corresponding loss value between 0 degrees and 2 degrees is the same as the loss value between 1 degree and 2 degrees, the continuity and size of the angle itself are ignored, thereby affecting the precision of the second model and the accuracy of the head state detection result, further affecting the effectiveness and accuracy of the target head state feature of the target face image extracted by the second model. In order to solve the above problem, in the embodiment of the present application, the second model is trained in a regression manner, that is, the annotated pitch angle, yaw angle and roll angle of the face image are first discretized, assuming that the number of classes after discretization is X, then the above three angles of multiple face images are divided into X-1 groups of binary classification, and the second model is trained according to the X-1 group binary classification results to realize ordered regression, thereby ensuring the effectiveness and accuracy of the target head state feature extracted by the second model, and also ensuring the angle order of the head pose Euler angle (pitch angle, yaw angle and roll angle) estimated based on the target head state feature. The above X is an integer greater than 2.

[0110] It can be understood that the above head state can also include only one or two of the pitch angle, yaw angle and roll angle, which is not limited in the embodiment of the present application. The number of the above target head state features corresponds to the number and type of angles in the above head state, for example, when the head state includes the pitch angle and the yaw angle, the target head state feature extracted from the target face image by the second model trained based on the regression of multiple known head state face images will include the target pitch angle feature and the target yaw angle feature.

[0111] Further, as Figure 4CAs shown, the third model includes a third convolutional module and a third fully connected layer. When extracting the target eye state feature of the target eye image by using the third model, the target eye image can be input into the trained third model, and then the corresponding target eye state feature can be directly extracted by the third convolutional module of the third model. The third convolutional module includes at least one convolutional layer. The third model is obtained by contrastive learning training based on a plurality of eye images of known eye states, and the loss function thereof is a contrastive learning loss function. The eye state is used to represent the open or closed state of the user's eyes in the eye image, and at least includes a closed eye state and an open eye state. In the embodiment of the present application, the third model can be trained in a supervised contrastive learning manner, the distance between the eye state features of the eye images in the same eye state input into the third model is narrowed, and the distance between the eye state features of the eye images in different eye states input into the third model is widened, so that the features corresponding to different eye states can be more obviously distinguished, and the anti-interference ability of the fatigue detection process to complex environments is increased.

[0112] Optionally, the target eye image includes a target left eye image and a target right eye image, and the target eye state feature includes a target left eye state feature and a target right eye state feature, that is, the target left eye state feature of the target left eye image and the target right eye state feature of the target right eye image can be extracted by using the third model respectively, so as to further reduce the interference and influence of other regions irrelevant to the eye in the target image on the extracted target eye state feature.

[0113] After extracting the target feature of the target image based on the face of the target user, as shown in the method for detecting fatigue, Figure 3 the method further includes:

[0114] S303, generating a target feature sequence based on the target features of the plurality of target images.

[0115] Specifically, the target feature sequence can be obtained by splicing the target features corresponding to the plurality of target images in the first time length in chronological order. The target feature sequence can be a multi-dimensional feature or a feature map, which is not limited in the embodiment of the present application. The plurality of target images in the first time length can be all target images of the video image constituting the first time length, or a plurality of target images extracted at equal intervals or randomly from the video image of the first time length, which is not limited in the embodiment of the present application. The video image of the first time length can be a video image of any time period with a time length of the first time length, which is not limited in the embodiment of the present application. The first time length can be 0.5S, 1S, 2S, etc., which is not limited in the embodiment of the present application.

[0116] Specifically, after extracting the target features corresponding to each of the plurality of target images in the first time length, the target features of the same category corresponding to each of the plurality of target images in the first time length can be spliced in the time sequence of the corresponding target images in the first time length to obtain a target feature sequence. The target feature sequence includes a target yawning state feature sequence and / or a target head state feature sequence and / or a target eye state feature sequence. The types of feature sequences included in the target feature sequence correspond to the types of target features extracted from the plurality of target images in the first time length. That is, after extracting the target yawning state features corresponding to each of the plurality of target images in the first time length, the target yawning state features corresponding to each of the plurality of target images in the first time length can be spliced to obtain a target yawning state feature sequence; and / or, after extracting the target head state features corresponding to each of the plurality of target images in the first time length, the target head state features corresponding to each of the plurality of target images in the first time length can be spliced to obtain a target head state feature sequence; and / or, after extracting the target eye state features corresponding to each of the plurality of target images in the first time length, the target eye state features corresponding to each of the plurality of target images in the first time length can be spliced to obtain a target eye state feature sequence.

[0117] Further, when the target head state features of the plurality of target images in the first time length extracted include at least two of the target pitch angle feature, the target yaw angle feature, and the target roll angle feature, the at least two target head state features corresponding to each of the plurality of target images in the first time length can be spliced respectively according to the category, so as to obtain at least two corresponding target head state feature sequences.

[0118] Optionally, when the target left eye state features and the target right eye state features corresponding to each of the plurality of target images in the first time length are extracted, the target left eye state features corresponding to each of the plurality of target images in the first time length can be spliced to obtain a target left eye state feature sequence, and the target right eye state features corresponding to each of the plurality of target images in the first time length can be spliced to obtain a target right eye state feature sequence.

[0119] After generating the target feature sequence based on the target features of the plurality of target images in the first time length, as shown in Figure 3 the fatigue detection method further includes:

[0120] S304, inputting the target feature sequence into a state detection model, and outputting a state detection result of the target user in the first time length.

[0121] Specifically, the state detection model is trained based on a plurality of images in a plurality of first time lengths of a known user state. The state detection result includes a head state detection result and / or a yawning state detection result and / or an eye state detection result. The head state detection result includes at least one of a head posture Euler angle (head pitch angle, head yaw angle, and head roll angle) of the target user in the first time length, the yawning state detection result includes a probability of the target user being in a yawning state in the first time length, and the eye state detection result includes a probability of the target user being in a closed-eye state in the first time length.

[0122] It can be understood that, in the embodiments of the present application, the category to which the target feature sequence belongs, i.e., the state related to the target feature sequence, is the same as the user state involved in the training of the state detection model.

[0123] Specifically, when the target feature sequence includes a target yawning state feature sequence, the target yawning state feature sequence can be input into a yawning state detection model (state detection model) trained based on a plurality of images in a plurality of first time lengths of a known yawning state (user state), so as to output a yawning state detection result of the target user in the first time length; and / or, when the target feature sequence includes a target head state feature sequence, the target head state feature sequence can be input into a head state detection model (state detection model) trained based on a plurality of images in a plurality of first time lengths of a known head state (user state), so as to output a head state detection result of the target user in the first time length; and / or, when the target feature sequence includes a target eye state feature sequence, the target eye state feature sequence can be input into an eye state detection model (state detection model) trained based on a plurality of images in a plurality of first time lengths of a known eye state (user state), so as to output an eye state detection result of the target user in the first time length.

[0124] For example, in the process of fatigue detection, the implementation process of detecting the user states such as the eye state, the yawning state, and the head state of the target user in the first time length is as follows: Figure 5As shown, if the first time length is 0.5S and the time length corresponding to each frame is 1 / 12 second, a total of 6 frames of target images can be included in the first time length. After obtaining the video images in the first time length, i.e. 6 consecutive target images, the target features of every other frame of target images can be extracted based on the face of the target user in the target images, i.e. a total of 3 frames of target features of target images in the first time length are extracted, and then the 3 frames of target features of target images are spliced in the order of the 3 frames of target images in the first time length to obtain a target feature sequence or a target feature map, and then the above target feature sequence or target feature map is input into the trained state detection model to output the state detection result of the target user in the first time length. The images in the n first time lengths (first time length i1, first time length i2,..., first time length in) involved in the training of the above state detection model can be images obtained before the 6 frames of target images in the first time length involved in the state detection.

[0125] It can be understood that when it is desired to determine the yawning state of the target user in the first time length, Figure 5 the target features in the above should be target yawning state features, the target feature sequence should be a target yawning state feature sequence, the state detection model should be a yawning state detection model, the user state involved in the training of the above yawning state detection model should be a yawning state, and the state detection result output by the above yawning state detection model can include the probability of the target user being in a yawning state in the first time length, and can also include the probability of the target user being in a non-yawning state in the first time length.

[0126] It can be understood that when it is desired to determine the head state of the target user in the first time length, Figure 5 the target features in the above should be target head state features (including at least one of the three features related to the head pose Euler angle, i.e. target pitch angle feature, target yaw angle feature, and target roll angle feature), the target feature sequence should be a target head state feature sequence, the state detection model should be a head state detection model, the user state involved in the training of the above head state detection model should be a head state (including the head pose Euler angle corresponding to the target head state feature), and the state detection result output by the above head state detection model includes at least one of the head pose Euler angles of the target user in the first time length, i.e. the head pitch angle, the head yaw angle, and the head roll angle, which corresponds to the category of the head pose Euler angle included in the above user state.

[0127] It can be understood that when it is desired to determine the eye state of the target user in the first time length, Figure 5The target feature in the target image should be a target eye state feature, the target feature sequence should be a target eye state feature sequence, the state detection model should be an eye state detection model, the user state involved in training the above eye state detection model should be an eye state, i.e., whether the user is in an open-eye state or a closed-eye state in the first time length, and the state detection result output by the above eye state detection model can include the probability that the target user is in a closed-eye state in the first time length, and can also include the probability that the target user is in an open-eye state in the first time length.

[0128] Optionally, when the target image includes a target left eye image and a target right eye image, Figure 5 The state detection model in the target image can obtain left eye state detection results and right eye state detection results of the target user in the first time length, i.e., the probabilities that the left eye and the right eye are in a closed-eye state, respectively, and then the left eye state detection results and the right eye state detection results can be weighted and summed, such as but not limited to averaging the sum of the probability that the left eye is in a closed-eye state and the probability that the right eye is in a closed-eye state, etc., to obtain the eye state detection result of the target user in the first time length, i.e., the probability of being in an open-eye state.

[0129] It can be understood that in the embodiments of the present application, each state detection result corresponds to the first time length, rather than a single frame of target image, thereby avoiding the problem that the anti-interference ability to complex environment changes is not strong when determining the user state according to a single frame of image, improving the stability and accuracy of user state detection, thereby enhancing the effectiveness and anti-interference ability of fatigue detection, and further improving the accuracy of fatigue detection. When the fatigue detection is applied to an auxiliary driving scene, the driver state (such as but not limited to whether to close eyes, whether to yawn, whether to lower head, etc.) can be detected more accurately, and whether the driver is in a fatigue driving state can be determined according to the accurate driver state, thereby providing stronger protection for the driving safety of the driver.

[0130] After the state detection results corresponding to the continuous multiple first time lengths in the second time length are determined, as shown in Figure 3 The fatigue detection method further includes:

[0131] S305, determining a fatigue detection result of the target user according to the state detection results corresponding to the continuous multiple first time lengths in the second time length.

[0132] Specifically, the second time length is equal to the total time length of the multiple first time lengths, i.e., the second time length can be composed of multiple first time lengths. The time period corresponding to the multiple first time lengths can be spliced in chronological order to obtain the time period corresponding to the second time length.

[0133] In the embodiments of the present application, compared with the state detection or fatigue detection based on a single target image, which is easily affected by complex environments and thus has low accuracy, the embodiments of the present application determine the eye state, head state, and yawning state of the target user in the first time length according to the target feature sequence generated based on the target features of the multiple target images in the first time length, and determine the fatigue detection result of the target user in the second time length according to the states corresponding to the multiple first time lengths in the second time length, thereby improving the anti-interference ability of fatigue detection under complex environment changes and improving the stability and accuracy of fatigue detection.

[0134] Optionally, the fatigue detection result includes a fatigue degree distribution result; and the fatigue degree distribution result includes at least one fatigue degree of the target user in the second time length and a probability corresponding to the fatigue degree. Figure 6 As shown in FIG. 5, the implementation process of determining the fatigue detection result of the target user in S305 can include the following steps:

[0135] S601, extract fatigue feature information in the state detection result corresponding to the multiple first time lengths in the second time length.

[0136] Specifically, after obtaining the state detection result corresponding to the multiple first time lengths in the second time length, fatigue feature information of the target user in the second time length can be extracted from the state detection result corresponding to the multiple first time lengths. The fatigue feature information includes at least one of the following: total eye closure time of the target user in the second time length, average time of each eye closure, blink frequency, total yawning time, mean and variance of head posture Euler angle, percentage of head pitch angle less than a first threshold value, percentage of head yaw angle less than a second threshold value or greater than a third threshold value.

[0137] Further, since a person tends to lower his head or shake his head too much or turn his head too much to left or right when he is tired, when the state detection result includes the head pitch angle and / or the head yaw angle and / or the head roll angle of the target user within the first time length, the fatigue feature information within the second time length can include but is not limited to the mean and variance of each head attitude Euler angle (head pitch angle, head yaw angle, head roll angle) corresponding to a plurality of consecutive first time lengths, the percentage of the head pitch angles less than a first threshold (equivalent to the target user being in a low head state within the second time length) in the head pitch angles corresponding to the plurality of consecutive first time lengths, the percentage of the head yaw angles less than a second threshold (equivalent to the target user shaking his head too much to the left within the second time length) or greater than a third threshold (equivalent to the target user shaking his head too much to the right within the second time length) in the head yaw angles corresponding to the plurality of consecutive first time lengths, the percentage of the head roll angles less than a fourth threshold (equivalent to the target user turning his head too much to the left within the second time length) or greater than a fifth threshold (equivalent to the target user turning his head too much to the right within the second time length) in the head roll angles corresponding to the plurality of consecutive first time lengths, and the like. The first threshold can be -10 degrees, 0 degrees, and the like, the second threshold can be -30 degrees, -10 degrees, and the like, the third threshold can be 30 degrees, 20 degrees, and the like, the fourth threshold can be -40 degrees, -10 degrees, and the like, and the fifth threshold can be 35 degrees, 25 degrees, and the like, which are not limited in the embodiments of the present application.

[0138] Further, since a person tends to yawn when he is tired, and the more tired a person is, the longer the yawn time or the more times he yawns, when the state detection result includes the probability of the target user being in a yawning state within the first time length, i.e., the yawning state detection result, the fatigue feature information within the second time length can include but is not limited to the number of yawns or the total yawn time of the target user within the second time length, and the like. In the embodiments of the present application, when the probability of being in a yawning state corresponding to the first time length is greater than a yawning probability threshold, it is considered that the target user is in a yawning state within the first time length, and when the probability of being in a yawning state corresponding to the first time length is less than or equal to the yawning probability threshold, it is considered that the target user is not in a yawning state within the first time length. The yawning probability threshold can be 0.5, 0.6, and the like, which are not limited in the embodiments of the present application. The number of yawns within the second time length can be the sum of the number of times that the target user is in a yawning state between two consecutive first time lengths and the number of first time lengths in which the target user is in a yawning state within the second time length. The total yawn time within the second time length can be the sum of all first time lengths in which the target user is in a yawning state within the second time length.

[0139] Further, since people tend to close their eyes or blink slowly when they are tired, when the state detection result includes the probability of the target user being in the closed-eye state in the first time length, i.e., the eye detection result, the fatigue feature information in the second time length can include but is not limited to the total closed-eye time length of the target user in the second time length, the average time length of each closed-eye, the blink frequency, etc. In the embodiments of the present application, when the probability of being in the closed-eye state corresponding to the first time length is greater than the closed-eye probability threshold, it can be considered that the target user is in the closed-eye state in the first time length, and when the probability of being in the closed-eye state corresponding to the first time length is less than or equal to the closed-eye probability threshold, it can be considered that the target user is in the open-eye state in the first time length. The closed-eye probability threshold can be 0.68, 0.7, etc., which is not limited in the embodiments of the present application. The total closed-eye time length in the second time length can be the sum of all first time lengths in which the target user is in the closed-eye state in the second time length. The closed-eye times in the second time length can be the sum of the number of times that the target user is in the closed-eye state between two adjacent first time lengths in which the target user is in the open-eye state and the number of first time lengths in which the target user is in the closed-eye state in the second time length. The average time length of each closed-eye in the second time length is equal to the total closed-eye time length in the second time length divided by the closed-eye times in the second time length. The blink frequency in the second time length can be equal to the closed-eye times in the second time length divided by the second time length.

[0140] Exemplarily, as Figure 7As shown, the second time length can include 3 consecutive first time lengths, wherein the state detection result corresponding to the target user in the 1st first time length includes a yawn probability of 0.3, an eye closure probability of 0.4, and a pitch angle of -3 degrees, the state detection result corresponding to the target user in the 2nd first time length includes a yawn probability of 0.6, an eye closure probability of 0.7, and a pitch angle of -15 degrees, and the state detection result corresponding to the target user in the 3rd first time length includes a yawn probability of 0.8, an eye closure probability of 0.9, and a pitch angle of -30 degrees. If the yawn probability threshold is 0.5, it can be determined that the target user is in a non-yawn state in the 1st first time length, and is in a yawn state in the 2nd first time length and the 3rd first time length. Therefore, the total yawn time length of the target user in the second time length can be determined from the state detection results of the target user corresponding to the 3 first time lengths in the second time length, which is the sum of the 2 first time lengths, for example, if the first time length is 1S and the second time length is 3S, the total yawn time length of the target user in the 3S is 2S; if the eye closure probability threshold is 0.6, it can be determined that the target user is in an open-eye state in the 1st first time length, and is in an eye closure state in the 2nd first time length and the 3rd first time length. Therefore, the total eye closure time length of the target user in the second time length can be determined from the state detection results of the target user corresponding to the 3 first time lengths in the second time length, which is the sum of the 2 first time lengths, for example, if the first time length is 1S and the second time length is 3S, the total eye closure time length of the target user in the 3S is 2S; if the first threshold is -10 degrees, it can be determined that the head pitch angle of the target user in the 1st first time length is not less than the first threshold, i.e., the target user does not lower his head, and the head pitch angles in the 2nd first time length and the 3rd first time length are less than the first threshold, i.e., the target user is in a head-lowering state in the 2nd first time length and the 3rd first time length. Therefore, the percentage of the target user's head pitch angle less than the first threshold in the second time length can be determined from the state detection results of the target user corresponding to the 3 first time lengths in the second time length, which is (2 / 3) x 100%, so that the total yawn time length, the total eye closure time length, and the percentage of the pitch angle less than the first threshold determined according to the state detection results of the target user corresponding to the 3 consecutive first time lengths in the second time length can be used as the fatigue feature information of the target user in the second time length.

[0141] After the fatigue feature information in the second time length is extracted, as shown in Figure 6 the implementation process of determining the fatigue detection result of the target user in S305 further includes:

[0142] S602, input the fatigue feature information into the fourth model, output the fatigue degree distribution result of the target user in the second duration, the fourth model is obtained by training based on a plurality of second duration video images with known fatigue degrees, and the fatigue degrees of the second duration video images include fatigue degrees annotated by a plurality of annotators on the video images.

[0143] Specifically, after extracting the fatigue feature information in the state detection result corresponding to the continuous multiple first durations in the second duration, the fatigue feature information can be directly input into the trained fourth model, so as to output the fatigue degree distribution result of the target user in the second duration. The fatigue degree can include degrees such as no fatigue, slight fatigue, and severe fatigue, and can also include multiple fatigue grades such as 0-grade fatigue, 1-grade fatigue, and 2-grade fatigue, which are not limited by the embodiments of the present application. The fatigue degree of the second duration video image used in the training of the fourth model can include a fatigue degree probability distribution result obtained by normalizing a histogram based on the fatigue degrees annotated by multiple annotators on the video image.

[0144] Specifically, since everyone has different evaluation criteria for fatigue degree in their hearts, in the embodiments of the present application, the fourth model is learned to learn the fatigue degree distribution of multiple second duration video images, so as to learn the evaluation of the fatigue degree of the user in the same video image by multiple annotators, thereby improving the accuracy of fatigue detection and making the fatigue detection result more consistent with the overall cognition of the public.

[0145] Exemplarily, when the fatigue degree is divided into N grades in total, the loss function of the fourth model is wherein p represents the probability of the predicted fatigue degree of the video image, CDF represents the probability distribution of the annotated fatigue degree of the video image, and CDF p (k) represents the probability cumulative value of the predicted k-grade fatigue degree of the video image, CDF represents the cumulative value of the probability distribution of the annotated k-grade fatigue degree of the video image, and r is a hyperparameter.

[0146] Exemplarily, as shown in Figure 8 when the second duration includes three continuous first durations, after determining the state detection result of the target user in the three first durations in the manner shown in S304 in the above embodiment, the fatigue degree distribution result of the target user in the second duration can be output in the manner shown in S306 in the above embodiment. Figure 3 Figure 6 ​The fatigue feature information of the target user in the second time length is extracted from the state detection results corresponding to the last three first time lengths in the manner shown in S601, and then the fatigue feature information is input into the fourth trained model, so as to obtain the fatigue degree distribution result of the target user in the second time length, for example, the probability of the target user being not tired in the second time length is A, the probability of the target user being slightly tired is B, and the probability of the target user being seriously tired is C. When Figure 8 When the fatigue detection process shown in S601 is applied to an assisted driving scene, if the output probability C of serious fatigue is greater than a target fatigue probability threshold (for example, but not limited to 0.6, 0.7, etc.), an alarm information can be immediately sent to prompt that the target user is driving while tired and there is a safety risk. As shown in S602, in order to improve the accuracy of fatigue detection and make the fatigue detection result more consistent with the overall cognition of the crowd, before fatigue detection, K annotators can be invited to annotate the fatigue degree of y video images in the second time length, so that there are K annotations of the fatigue degree of each video image in the second time length, that is, the label corresponding to each video image in the second time length is the fatigue degree probability distribution result, and then the video images in the second time length with known fatigue degree distribution results are used as training data to train the fourth model, so that the fourth model can learn the overall cognition of the fatigue degree of the annotators, and improve the effectiveness and accuracy of fatigue detection. The above-mentioned multiple video images in the second time length can be multiple video images with the same time length but different time periods, or multiple video images with the same time length and different users at the same time, etc. The embodiments of the present application do not limit this. Figure 8 Optionally, after S601, that is, after the fatigue feature information in the second time length is extracted from the state detection results corresponding to the last three first time lengths, it can also be directly judged whether the fatigue feature information in the second time length satisfies the preset fatigue condition. If the fatigue feature information in the second time length satisfies the preset fatigue condition, it is determined that the target user is in a fatigue state in the second time length, and if the fatigue feature information in the second time length does not satisfy the preset fatigue condition, it is determined that the target user is in a non-fatigue state in the second time length. The above-mentioned preset fatigue condition can be but not limited to that the total yawning time or the total eye closing time of the target user in the second time length is greater than 1 / 3 times of the second time length, or the percentage of the target user in the second time length with a pitch angle less than the first threshold is greater than 60%, etc. The embodiments of the present application do not limit this.

[0147]

[0148] ​Optionally, after the fatigue feature information of the target user in the second time length is extracted in S601, the fatigue state of the target user in the second time length can also not be detected through the fourth model, but the fatigue feature information in the second time length is directly matched with the preset feature information corresponding to each fatigue degree. When the fatigue feature information in the second time length matches the feature information corresponding to a certain fatigue degree, the fatigue degree corresponding to the matched feature information is determined as the fatigue degree of the target user in the second time length.

[0149] Optionally, after S303, the state of the target user can also not be detected, but the target feature sequence generated based on the target features of the plurality of target images in the first time length is directly input into the trained fatigue state detection model, so as to output the fatigue state detection result of the target user in the first time length. The fatigue state detection result can include the probability that the target user is in a fatigue state in the first time length, or can include a plurality of fatigue degrees that the target user can be in in the first time length and the probabilities corresponding to each fatigue degree, i.e., the fatigue degree distribution. The training process of the fatigue state detection model is similar to that of the fourth model in Figure 8

[0150] Next, please refer to Figure 9 , which is a flowchart of another fatigue detection method provided by an exemplary embodiment of the present application. As shown in Figure 9 , the fatigue detection method includes the following steps:

[0151] S901, a plurality of target images in a first time length are obtained, and the target images include a target user.

[0152] Specifically, S901 is consistent with S301, which will not be repeated here.

[0153] S902, target features of the target images are extracted based on the face of the target user.

[0154] Specifically, S902 is consistent with S302, which will not be repeated here.

[0155] S903, a target feature sequence is generated based on the target features of the plurality of target images.

[0156] Specifically, S903 is consistent with S303, which will not be repeated here.

[0157] S904, the target feature sequence is processed by dimension reduction to obtain a first target feature sequence.

[0158] ​Specifically, after splicing the target features of the multiple target images in the first time length into the target feature sequence, the target feature sequence can also be processed by dimension reduction, so as to remove the noise in the target feature sequence and obtain the first target feature sequence. The dimension reduction processing can be, but is not limited to, a principal component analysis (PCA) dimension reduction method.

[0159] S905, restoring the first target feature sequence to the same dimension as the target feature sequence to obtain a second target feature sequence.

[0160] Specifically, after the target feature sequence is processed by dimension reduction to obtain the first target feature sequence with noise removed, in order to ensure that the state detection model can output the state detection result of the target user in the first time length based on the feature sequence with noise removed, the first target feature sequence with noise removed also needs to be restored to the same dimension as the target feature sequence, so as to obtain the second target feature sequence.

[0161] S906, inputting the second target feature sequence into the state detection model to output the state detection result of the target user in the first time length.

[0162] Specifically, S906 is consistent with the implementation process of S304, which will not be described here.

[0163] S907, determining the fatigue detection result of the target user according to the state detection results corresponding to the multiple first time lengths in the second time length.

[0164] Specifically, S907 is consistent with S305, which will not be described here.

[0165] In the embodiments of the present application, by first processing the target feature sequence spliced from the target features of the multiple target images in the first time length by dimension reduction to obtain the first target feature sequence, and then restoring to obtain the second target feature sequence, the noise in the feature sequence is removed, the influence of external environmental factors is avoided, finally the second target feature sequence with noise removed is input into the state detection model to output the state detection result of the target user in the first time length, and the fatigue detection result of the target user is determined according to the state detection results corresponding to the multiple first time lengths in the second time length. By detecting the state of the target user through the second target feature sequence corresponding to each first time length with noise removed, and determining the fatigue condition of the target user in the multiple first time lengths, the anti-interference ability of fatigue detection is further improved, and the effectiveness of state detection and the accuracy of fatigue detection are further improved.

[0166] Please refer to Figure 10 , Figure 10 A fatigue detection device is provided for an exemplary embodiment of the present application. The fatigue detection device 1000 comprises:

[0167] The acquisition module 1010 is configured to acquire a plurality of target images in a first time length; the target images include a target user;

[0168] The feature extraction module 1020 is configured to extract target features of the target images based on a face of the target user;

[0169] The generation module 1030 is configured to generate a target feature sequence based on the target features of the plurality of target images;

[0170] The state detection module 1040 is configured to input the target feature sequence into a state detection model to output a state detection result of the target user in the first time length; the state detection model is obtained by training based on a plurality of images in a plurality of first time lengths of known user states;

[0171] The determination module 1050 is configured to determine a fatigue detection result of the target user according to the state detection results corresponding to the plurality of first time lengths in a second time length; the second time length is equal to a total time length of the plurality of first time lengths.

[0172] In a possible implementation, the target features include at least one of a target yawning state feature, a target head state feature, and a target eye state feature.

[0173] The feature extraction module 1020 includes:

[0174] The determination unit is configured to determine a target face image and / or a target eye image of the target user based on a preset detection algorithm and the target images;

[0175] The first extraction unit is configured to extract a target yawning state feature of the target face image by using a first model; and / or

[0176] The second extraction unit is configured to extract a target head state feature of the target face image by using a second model; and / or

[0177] The third extraction unit is configured to extract a target eye state feature of the target eye image by using a third model; wherein the first model is obtained by contrast learning training based on a plurality of face images of known yawning states; the second model is obtained by regression training based on a plurality of face images of known head states; and the third model is obtained by contrast learning training based on a plurality of eye images of known eye states.

[0178] In a possible implementation, the target feature sequence includes a target yawning state feature sequence, a target head state feature sequence, and / or a target eye state feature sequence.

[0179] In a possible implementation, the target eye image includes a target left eye image and a target right eye image; the target eye state feature includes a target left eye state feature and a target right eye state feature; and the target eye state feature sequence includes a target left eye state feature sequence and a target right eye state feature sequence.

[0180] In a possible implementation, the fatigue detection apparatus 1000 further includes:

[0181] a dimension reduction processing module, configured to perform dimension reduction processing on the target feature sequence to obtain a first target feature sequence;

[0182] a restoration module, configured to restore the first target feature sequence to the same dimension as the target feature sequence to obtain a second target feature sequence;

[0183] The state detection module 1040 is specifically configured to: input the second target feature sequence into a state detection model, and output a state detection result of the target user in the first time length.

[0184] In a possible implementation, the state detection result includes a head state detection result and / or a yawning state detection result and / or an eye state detection result; the head state detection result includes a head posture Euler angle of the target user in the first time length; the yawning state detection result includes a probability that the target user is in a yawning state in the first time length; and the eye state detection result includes a probability that the target user is in a closed-eye state in the first time length.

[0185] In a possible implementation, the fatigue detection result includes a fatigue degree distribution result; the fatigue degree distribution result includes at least one fatigue degree of the target user in the second time length and a probability corresponding to the fatigue degree; and the determination module 1050 includes:

[0186] a fourth extraction unit, configured to extract fatigue feature information in the state detection result corresponding to a plurality of continuous first time lengths in a second time length;

[0187] a fatigue degree distribution detection unit, configured to input the fatigue feature information into a fourth model to output a fatigue degree distribution result of the target user in the second time length; the fourth model is obtained by training based on a plurality of second time length video images with known fatigue degrees; and the fatigue degrees of the second time length video images include fatigue degrees annotated by a plurality of annotators on the video images.

[0188] In a possible implementation, the fatigue feature information includes at least one of the following: a total eye-closing time of the target user in the second time length, an average time of each eye-closing, a blink frequency, a total yawning time, a mean and a variance of head posture Euler angles, a percentage of head pitch angles less than a first threshold value, and a percentage of head yaw angles less than a second threshold value or greater than a third threshold value.

[0189] The division of the modules in the fatigue detection apparatus is only for illustration, and in other embodiments, the fatigue detection apparatus can be divided into different modules as needed to complete all or part of the functions of the fatigue detection apparatus. The implementation of each module in the fatigue detection apparatus provided in the embodiments of the present specification can be in the form of a computer program. The computer program can run on a terminal or a server. The program modules constituted by the computer program can be stored on the memory of the terminal or the server. When the computer program is executed by the processor, all or part of the steps of the fatigue detection method described in the embodiments of the present specification are implemented.

[0190] Please refer to Figure 11 , Figure 11 A structural schematic diagram of a vehicle is provided for an exemplary embodiment of the present application. As shown in Figure 11 , the vehicle 1100 can include at least one processor 1110, at least one communication bus 1120, a user interface 1130, at least one network interface 1140, and a memory 1150.

[0191] The communication bus 1120 can be used to realize the connection and communication of the above-mentioned components.

[0192] The user interface 1130 can include a display and a camera, and can also include a standard wired interface and a wireless interface. The camera can be used to capture the face of a target user (driver) driving the vehicle 1100 to obtain a target image.

[0193] The network interface 1140 can include a Bluetooth module, a near field communication (NFC) module, a wireless fidelity (Wi-Fi) module, and the like.

[0194] The processor 1110 can include one or more processing cores. The processor 1110 connects various parts within the vehicle 1100 through various interfaces and lines, performs various functions of the vehicle 1100 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1150, and calling data stored in the memory 1150. Alternatively, the processor 1110 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 1110 can be a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes operating systems, user interfaces, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 1110, but can be realized by a separate chip.

[0195] The memory 1150 can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory 1150 includes a non-transitory computer readable medium. The memory 1150 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1150 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a function of obtaining, a function of detecting a state, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area can store data involved in the above-mentioned various method embodiments, etc. The memory 1150 can also be at least one storage device located away from the above-mentioned processor 1110. As shown, the memory 1150 as a computer storage medium can include an operating system, a network communication module, a user interface module, and program instructions. Figure 11 As shown, the memory 1150 as a computer storage medium can include an operating system, a network communication module, a user interface module, and program instructions.

[0196] In particular, the processor 1110 can be configured to invoke program instructions stored in the memory 1150, and specifically perform the following operations: acquire a plurality of target images within a first time period; the target images include a target user; extract target features of the target images based on the face of the target user; generate a target feature sequence based on the target features of the plurality of target images; input the target feature sequence into a state detection model to output a state detection result of the target user within the first time period; and the state detection model is trained based on a plurality of images within a plurality of first time periods of known user states.

[0197] Determine the fatigue detection result of the target user according to the state detection results corresponding to the plurality of first time periods within a second time period; the second time period is equal to the total time period of the plurality of first time periods.

[0198] In some possible embodiments, the target features include at least one of the following: target yawning state features, target head state features, and target eye state features; when the processor 1110 performs the operation of extracting target features of the target images based on the face of the target user, it specifically performs the following operations: determines a target face image and / or a target eye image of the target user based on a preset detection algorithm and the target images; extracts target yawning state features of the target face image using a first model; and / or extracts target head state features of the target face image using a second model; and / or extracts target eye state features of the target eye image using a third model; wherein the first model is trained based on a plurality of face images with known yawning states through contrastive learning; the second model is trained based on a plurality of face images with known head states through regression; and the third model is trained based on a plurality of eye images with known eye states through contrastive learning.

[0199] In some possible embodiments, the target feature sequence includes a target yawning state feature sequence and / or a target head state feature sequence and / or a target eye state feature sequence.

[0200] In some possible embodiments, the target eye image includes a target left eye image and a target right eye image; the target eye state feature includes a target left eye state feature and a target right eye state feature; and the target eye state feature sequence includes a target left eye state feature sequence and a target right eye state feature sequence.

[0201] In some possible embodiments, after the processor 1110 performs the operation of generating a target feature sequence based on the target features of the plurality of target images, before it performs the operation of inputting the target feature sequence into a state detection model to output a state detection result of the target user within the first time period, it further performs the following operations:

[0202] The target feature sequence is dimensionally reduced to obtain a first target feature sequence; and the first target feature sequence is restored to the same dimension as the target feature sequence to obtain a second target feature sequence.

[0203] The processor 1110 is configured to input the target feature sequence into the state detection model and output a state detection result of the target user in the first time length.

[0204] In some possible embodiments, the state detection result includes a head state detection result, a yawn state detection result, and / or an eye state detection result; the head state detection result includes a head posture Euler angle of the target user in the first time length; the yawn state detection result includes a probability that the target user is in a yawn state in the first time length; and the eye state detection result includes a probability that the target user is in a closed-eye state in the first time length.

[0205] In some possible embodiments, the fatigue detection result includes a fatigue degree distribution result; and the fatigue degree distribution result includes at least one fatigue degree of the target user in the second time length and a corresponding probability of the fatigue degree.

[0206] The processor 1110 is configured to extract fatigue feature information from the state detection result corresponding to the continuous multiple first time lengths in the second time length, and input the fatigue feature information into a fourth model to output a fatigue degree distribution result of the target user in the second time length; the fourth model is trained based on video images of multiple second time lengths with known fatigue degrees; and the fatigue degrees of the video images of the second time length include fatigue degrees annotated by multiple annotators.

[0207] In some possible embodiments, the fatigue feature information includes at least one of the following: a total closed-eye time length of the target user in the second time length, an average time length of each closed-eye, a blink frequency, a total yawn time length, a mean and variance of a head posture Euler angle, a percentage of a head pitch angle less than a first threshold value, and a percentage of a head yaw angle less than a second threshold value or greater than a third threshold value.

[0208] The embodiments of the present application further provide a computer storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps of any one of the above methods. The constituent modules of the above fatigue detection device, if realized in the form of software function units and sold or used as independent products, can be stored in the storage medium.

[0209] In the above embodiments, all or part of the embodiments can be realized by software, hardware, firmware, or any combination thereof. When realized by software, all or part of the embodiments can be realized in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted by the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0210] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, which can be stored in a computer-readable storage medium. The program can include the processes of the above-mentioned embodiments when executed. The aforementioned storage medium includes ROM, RAM, magnetic or optical disks, and various program code storage media. In the case of no conflict, the technical features in the embodiments and the embodiments can be combined arbitrarily.

[0211] The above-described embodiments are merely preferred embodiments of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements of the technical solutions of the present application made by those of ordinary skill in the art shall fall within the protection scope of the claims of the present application.

Claims

1. A fatigue detection method, characterized in that, The method includes: Acquire multiple frames of target images within a first time period; the target images include the target user; Based on the target user's face, target features are extracted from the target image; the target features include at least one of the following: target yawning state features, target head state features, and target eye state features; Generate a target feature sequence based on the target features of multiple frames of the target image; The target feature sequence is input into the state detection model, and the state detection result of the target user within the first time period is output; the state detection model is trained based on multiple frames of images within multiple first time periods with known user states; The fatigue detection result of the target user is determined based on the state detection results corresponding to multiple consecutive first durations within a second duration; the second duration is equal to the total duration of the multiple first durations; the state detection results include: head state detection results and / or yawning state detection results and / or eye state detection results; the head state detection results include the Euler angle of the target user's head posture within the first duration; the yawning state detection results include the probability that the target user is in a yawning state within the first duration; the eye state detection results include the probability that the target user is in a closed-eye state within the first duration.

2. The method as described in claim 1, characterized in that, The step of extracting target features from the target image based on the target user's face includes: The target face image and / or target eye image of the target user are determined based on a preset detection algorithm and the target image. The first model is used to extract the target yawning state features from the target facial image; and / or The second model is used to extract the target head state features of the target face image; and / or The target eye state features of the target eye image are extracted using a third model; The first model is trained by comparative learning based on multiple known facial images of yawning states; the second model is trained by regression based on multiple facial images of known head states; and the third model is trained by comparative learning based on multiple eye images of known eye states.

3. The method as described in claim 2, characterized in that, The target feature sequence includes the target yawning state feature sequence and / or the target head state feature sequence and / or the target eye state feature sequence.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: The target feature sequence is subjected to dimensionality reduction processing to obtain the first target feature sequence; The first target feature sequence is restored to the same dimension as the target feature sequence to obtain the second target feature sequence; The step of inputting the target feature sequence into the state detection model and outputting the state detection result of the target user within the first time period includes: The second target feature sequence is input into the state detection model, and the state detection result of the target user within the first time period is output.

5. The method as described in claim 1, characterized in that, The fatigue detection results include fatigue level distribution results; the fatigue level distribution results include at least one fatigue level of the target user within the second duration and the probability corresponding to the fatigue level. The step of determining the fatigue detection result of the target user based on the state detection results corresponding to multiple consecutive first time periods within a second time period includes: Extract fatigue feature information from the state detection results corresponding to multiple consecutive first time periods within the second time period; The fatigue feature information is input into the fourth model, and the fatigue level distribution of the target user within the second duration is output. The fourth model is trained based on multiple video images of the second duration with known fatigue levels. The fatigue level of the video images of the second duration includes the fatigue levels of multiple annotators who annotated the video images.

6. The method as described in claim 5, characterized in that, The fatigue characteristic information includes at least one of the following: the total duration of the target user's closed eyes within the second duration, the average duration of each eye closure, the blinking frequency, the total duration of yawning, the mean and variance of the Euler angles of the head posture, the percentage of head pitch angles less than the first threshold, and the percentage of head yaw angles less than the second threshold or greater than the third threshold.

7. A fatigue detection device, characterized in that, include: The acquisition module is used to acquire multiple frames of target images within a first time period; The target image includes the target user; The feature extraction module is used to extract target features from the target image based on the target user's face; the target features include at least one of the following: target yawning state features, target head state features, and target eye state features; The generation module is used to generate a target feature sequence based on the target features of the multi-frame target images; The state detection module is used to input the target feature sequence into the state detection model and output the state detection result of the target user within the first time period; The state detection model is trained based on multiple frames of images within a first time period with known user states; The determination module is used to determine the fatigue detection result of the target user based on the state detection results corresponding to multiple consecutive first time periods within the second time period; The second duration is equal to the total duration of the plurality of first durations; The state detection results include: head state detection results and / or yawning state detection results and / or eye state detection results; The head state detection result includes the Euler angle of the target user's head posture during the first duration; the yawning state detection result includes the probability that the target user is in a yawning state during the first duration; the eye state detection result includes the probability that the target user is in a closed-eye state during the first duration.

8. The apparatus as claimed in claim 7, characterized in that, The fatigue detection results include fatigue level distribution results; the fatigue level distribution results include at least one fatigue level of the target user within the second duration and the probability corresponding to the fatigue level. The determining module includes: The extraction unit is used to extract fatigue feature information from the state detection results corresponding to multiple consecutive first time periods within the second time period; The fatigue level distribution detection unit is used to input the fatigue feature information into the fourth model and output the fatigue level distribution result of the target user within the second duration; the fourth model is trained based on multiple video images of the second duration with known fatigue levels; the fatigue level of the video images of the second duration includes the fatigue levels annotated by multiple annotators for the video images.

9. A vehicle, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed the method steps as claimed in any one of claims 1-6.

Images