Method, device, medium and electronic device for fatigue detection

Abstract

Disclosed are a method, device, medium and electronic device for fatigue detection, wherein the method comprises: determining a group of eye key points of a target object in each video frame of a plurality of video frames, wherein the group of eye key points of the target object in each video frame forms an eye key point group sequence; determining an eye closure degree sequence of the target object according to each group of eye key points in the eye key point group sequence; and determining a fatigue state of the target object according to the eye closure degree sequence of the target object. The technical solution provided by the present disclosure is conducive to accurate detection of fatigue state without installing additional equipment in the vehicle, thereby helping to reduce the cost of fatigue detection and improving the accuracy of fatigue detection. In the field of driving, it is conducive to improving the driving safety of the vehicle.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle driving technology, and in particular to a method for fatigue detection, a device for fatigue detection, a storage medium, and an electronic device. Background Art

[0002] In the field of driving, since driver fatigue driving is a potential safety hazard during vehicle driving and has a great impact on vehicle safe driving, driver fatigue detection is an important part of DMS (Driver Monitor System).

[0003] How to obtain fatigue detection results in a timely and accurate manner is a technical issue that deserves attention. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure is proposed. Embodiments of the present disclosure provide a method, an apparatus, a storage medium and an electronic device for fatigue detection.

[0005] According to one aspect of an embodiment of the present disclosure, a method for fatigue detection is provided, the method comprising: determining a group of eye key points of a target object in each video frame of a plurality of video frames, the group of eye key points of the target object in each video frame forming an eye key point group sequence; determining an eye closure degree sequence of the target object based on each group of eye key points in the eye key point group sequence; determining a fatigue state of the target object based on the eye closure degree sequence of the target object.

[0006] According to another aspect of the embodiments of the present disclosure, there is provided a device for fatigue detection, the device comprising: a key point detection module, used to determine a group of eye key points of a target object in each video frame of a plurality of video frames, wherein the group of eye key points of the target object in each video frame forms an eye key point group sequence; a closure sequence determination module, used to determine an eye closure sequence of the target object according to each group of eye key points in the eye key point group sequence obtained by the key point detection module; and a fatigue state determination module, used to determine the fatigue state of the target object according to the eye closure sequence of the target object obtained by the closure sequence determination module.

[0007] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to implement the above method.

[0008] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, which includes: a processor; a memory for storing instructions executable by the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the above method.

[0009] Based on a method and device for fatigue detection provided by the above-mentioned embodiments of the present disclosure, by detecting the eye key points of the target object in multiple video frames, the eye closure degree of the target object in each video frame can be accurately determined without installing additional equipment inside the vehicle, thereby providing an accurate data basis for judging the fatigue state of the target object; by using the eye closure degree sequence formed by the eye closure degree of the target object in multiple video frames to judge the fatigue state of the target object, it is helpful to avoid misjudgment caused by phenomena such as glare, and greatly improve the accuracy of fatigue detection. It can be seen that the technical solution provided by the present disclosure is conducive to reducing the cost of fatigue detection, and is conducive to improving the accuracy of fatigue detection. For the driving field, it is conducive to improving the driving safety of the vehicle.

[0010] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] By describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present disclosure will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 is a schematic diagram of a scenario to which the present disclosure is applicable;

[0013] Figure 2 A flowchart of an embodiment of a method for fatigue detection disclosed herein;

[0014] Figure 3 A schematic diagram of an embodiment of the eye key point disclosed in the present invention;

[0015] Figure 4 A flowchart of an embodiment of a sequence for determining eye closure of a target object disclosed herein;

[0016] Figure 5 A flowchart of an embodiment of obtaining the head posture and sight direction of a target object in a video frame disclosed in the present invention;

[0017] Figure 6 A schematic diagram of an embodiment of a neural network for multi-task learning disclosed in the present invention;

[0018] Figure 7 It is a structural schematic diagram of an embodiment of a device for fatigue detection disclosed in the present invention;

[0019] Figure 8 It is a structural diagram of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0020] The exemplary embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0021] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0022] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0023] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0024] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0025] In addition, the term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0026] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0027] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0028] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0029] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0030] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0031] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems or servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0032] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, target programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks can be performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0033] SUMMARY OF THE DISCLOSURE

[0034] In the process of realizing the present disclosure, the inventors found that PERCLOS (percentage of eyelid closure over the pupil over time) is usually used to determine the fatigue state of the target object. In the application process of PERCLOS, it is usually necessary to detect the eye position and eye closure degree of the target object, and in the process of determining the eye closure degree of the target object, it is usually necessary to use additional equipment such as multiple camera devices, infrared equipment and sensors, which not only take up a certain space (for example, occupying limited vehicle space), but also increase a certain cost.

[0035] If the degree of eye closure of the target object can be known by utilizing existing facilities in the space (such as a monocular camera in a vehicle, etc.), the fatigue state of the target object can be conveniently determined without installing additional equipment and making full use of the existing facilities in the space.

[0036] Exemplary Overview

[0037] The technical solution for fatigue detection provided by the present disclosure can be applied to a variety of scenarios. For example, the technical solution for fatigue detection provided by the present disclosure can be applied to DMS applications. Figure 1 shown.

[0038] Figure 1 In the embodiment, the vehicle 100 is equipped with a DMS, and the photos or videos (i.e., monocular images) taken by the monocular camera device 101 in the vehicle 100 can be provided to the DMS in real time. When the driver is in the driving position of the vehicle 100, the driver's face should be within the boundary line of the field of view of the monocular camera device 101, that is, the photos or videos captured by the monocular camera device 101 usually include the driver's face (such as the front face, etc.).

[0039] The DMS in the vehicle 100 can determine the driver's fatigue state in a timely manner based on the video transmitted in real time by the monocular camera device 101. When it is determined that the driver's fatigue state within a period of time belongs to a pre-set mild fatigue state, the DMS in the vehicle can remind the driver to consider taking a short break through text, voice, light, video, etc. When it is determined that the driver's fatigue state within a period of time belongs to a pre-set moderate fatigue state, the DMS can remind the driver to take a short break through text, voice, light, video, etc. When it is determined that the driver's fatigue state within a period of time belongs to a pre-set severe fatigue state, the DMS can give an emergency warning to the driver through text, voice, light, video, etc. to remind the driver that his current driving behavior is a dangerous driving behavior and the driver must take a short rest before driving on the road to ensure the driving safety of the vehicle 100.

[0040] Exemplary Methods

[0041] Figure 2 FIG. 1 is a flow chart of an embodiment of a method for fatigue detection disclosed in the present invention. Figure 2 The method shown includes: S200, S201 and S202. Each step is described below.

[0042] S200: Determine a set of eye key points of a target object in each of a plurality of video frames.

[0043] The multiple video frames in the present disclosure may be multiple video frames arranged continuously in a video sequence captured by a camera device. The multiple video frames in the present disclosure may also be multiple video frames arranged non-continuously in a video sequence captured by a camera device. For example, the multiple video frames in the present disclosure may be the 1st video frame, the 3rd video frame, the 5th video frame, ... and the nth (n is an odd number greater than 5) video frame in a video sequence.

[0044] The target object in the present disclosure generally refers to an object that needs to be fatigue monitored. For example, the target object may be a driver, a monitoring person in a monitoring room, or a worker on an assembly line.

[0045] A group of eye key points in the present disclosure generally includes multiple eye key points, and a group of eye key points generally corresponds to one video frame, that is, the present disclosure can extract a group of eye key points of a target object from one video frame, and then the present disclosure can extract multiple groups of eye key points of a target object from multiple video frames. The eye key point in the present disclosure may refer to a point that can characterize a specific position on the eye. The eye key point is generally an eye key point based on a two-dimensional plane. Each eye key point generally has semantics, and the semantics of an eye key point can indicate the specific position of the eye key point on the eye. For example, each eye key point has a label, and the label of the eye key point can indicate the specific position of the eye key point on the eye.

[0046] An example of a set of eye key points of a target object extracted from a video frame is as follows: Figure 3 shown. Figure 3 In the embodiment, a group of eye key points includes: 18 eye key points, and 9 of the 18 eye key points are key points for describing the eyelid contour of the eye, another 8 of the 18 eye key points are key points for describing the iris contour of the eye, and the last of the 18 eye key points is the pupil center key point. The present disclosure does not limit the number of eye key points contained in a group of eye key points and the semantics of each eye key point. In addition, the present disclosure can use a neural network to extract the eye key points of the target object in each video frame, and the present disclosure does not limit the specific implementation method of obtaining the eye key points of the target object in the video frame.

[0047] A set of eye key points in the present disclosure may be based on the eye key points of the left eye of the target object, may also be based on the eye key points of the right eye of the target object, or may also be based on the eye key points of both eyes of the target object.

[0048] S201. Determine an eye closure degree sequence of a target object according to each group of eye key points in an eye key point group sequence.

[0049] A group of eye key points corresponding to each video frame in the present disclosure is arranged according to the arrangement order of the video frames to form an eye key point group sequence.

[0050] The eye closure sequence in the present disclosure generally includes multiple eye closures, and each eye closure corresponds to a time, and the time corresponding to the eye closure can be the time of the video frame corresponding to the eye closure (such as the acquisition time of the video frame or the playback time of the video frame, etc.). An eye closure in the present disclosure can represent the degree of eye closure of the target object at a point in time. The eye closure sequence in the present disclosure can refer to a sequence formed by arranging multiple eye closures in the order of their corresponding times.

[0051] Each eye closure degree in the eye closure degree sequence in the present disclosure is usually determined by a group of eye key points. For example, the present disclosure can determine an eye closure degree by calculating the positional relationship between multiple eye key points in a group of eye key points at a certain time. A group of eye key points corresponding to two adjacent eye closure degrees in the eye closure degree sequence can be a group of eye key points of a target object in two adjacent video frames in a video sequence.

[0052] S202: Determine the fatigue state of the target object according to the eye closure degree sequence of the target object.

[0053] The eye closure sequence in the present disclosure can represent changes in the degree of eye closure of the target object within a time range. By analyzing the changes in the eye closure of each eye in the eye closure sequence of the target object, the present disclosure can more accurately determine normal eye closure changes of the target object, such as normal blinking and squinting to avoid strong light, thereby facilitating accurate determination of the fatigue state of the target object.

[0054] The present disclosure can determine a fatigue state of a target object based on an eye closure sequence with a predetermined length, that is, the present disclosure can determine m fatigue states of a target object based on m (m is an integer greater than 1) eye closure sequences. The m fatigue states can form a fatigue state sequence, and the present disclosure can decide whether to form fatigue warning information based on the fatigue state sequence to prompt the target object. Each fatigue state in the fatigue state sequence of the present disclosure corresponds to an eye closure sequence, and the eye closure sequences corresponding to two adjacent fatigue states in the fatigue state sequence may have an intersection. As an example, assuming that the i-th fatigue state in the fatigue state sequence corresponds to the j-th eye closure sequence, and the i+1-th fatigue state in the fatigue state sequence corresponds to the j+1-th eye closure sequence, the intersection between the j-th eye closure sequence and the j+1-th eye closure sequence generally includes at least one eye closure.

[0055] The present disclosure can accurately determine the eye closure degree of the target object in each video frame without installing additional equipment inside the vehicle by detecting the key points of the eyes of the target object in multiple video frames, thereby providing an accurate data basis for judging the fatigue state of the target object; by using the eye closure degree sequence formed by the eye closure degree of the target object in multiple video frames to judge the fatigue state of the target object, it is helpful to avoid misjudgment caused by phenomena such as glare, and greatly improve the accuracy of fatigue detection. It can be seen that the technical solution provided by the present disclosure is conducive to reducing the cost of fatigue detection and improving the accuracy of fatigue detection. For the driving field, it is conducive to improving the driving safety of the vehicle.

[0056] In an optional example, the present disclosure determines an example of an eye closure sequence of a target object according to each group of eye key points in the eye key point group sequence, such as Figure 4 shown.

[0057] Figure 4 In step S400, according to each group of eye key points in the eye key group sequence, determine the area-based eye parameters and / or distance-based eye parameters corresponding to each group of eye key points.

[0058] Optionally, the area-based eye parameter in the present disclosure may refer to the area calculation result formed by the eye key points obtained by calculating the area using the positions of multiple eye key points in a group of eye key points. The area-based eye parameter may represent the visible area of ​​the eye or the visible area of ​​a specific part of the eye, etc. In one example, for any group of eye key points in the eye key point sequence, the present disclosure may calculate the area of ​​the eye visible region using the positions of each eye key point related to the eyelid in the group of eye key points, thereby obtaining the area of ​​the eye visible region corresponding to the group of eye key points. In another example, for any group of eye key points in the eye key point sequence, the present disclosure may calculate the area of ​​the iris visible region using the positions of each eye key point related to the iris in the group of eye key points, thereby obtaining the area of ​​the iris visible region corresponding to the group of eye key points. Since both the eye visible region area and the iris visible region area can better reflect the eye closure state of the target object, the present disclosure forms an eye closure degree sequence by using the eye visible region area and the iris visible region area as basic data, which is conducive to improving the accuracy of the eye closure degree sequence.

[0059] Optionally, the distance-based eye parameter in the present disclosure may refer to the distance calculation result formed by the eye key points by using the positions of multiple eye key points in a group of eye key points. The distance-based eye parameter can represent the maximum visible height of the eye or the maximum visible height of a specific part of the eye, etc. In one example, for any group of eye key points in the eye key point sequence, the present disclosure can use the positions of each eye key point related to the eyelid in the group of eye key points to calculate the distance between the upper eyelid and the lower eyelid in the vertical direction (such as the y-coordinate axis direction of the plane coordinate system of the video frame), thereby obtaining the maximum distance between the upper eyelid and the lower eyelid corresponding to the group of eye key points in the vertical direction. In another example, for any group of eye key points in the eye key point sequence, the present disclosure can use the positions of each eye key point related to the iris in the group of eye key points to calculate the distance of the visible iris in the vertical direction, thereby obtaining the maximum distance of the visible iris corresponding to the group of eye key points in the vertical direction. Since the maximum distance between the upper eyelid and the lower eyelid in the vertical direction, and the maximum distance between the visible iris in the vertical direction can better reflect the eye closure state of the target object, the present invention uses the maximum distance between the upper eyelid and the lower eyelid in the vertical direction, and the maximum distance between the visible iris in the vertical direction as basic data to form an eye closure sequence, which is conducive to improving the accuracy of the eye closure sequence.

[0060] Optionally, the position of the eye key point in the present disclosure may refer to the plane coordinates of the eye key point in the plane coordinate system of the video frame. For example, the origin of the plane coordinate system of the video frame is located at the upper left corner or the lower left corner or the center of the video frame, the horizontal coordinate axis (i.e., the x-coordinate axis) of the plane coordinate system is the width direction of the video frame, and the vertical coordinate axis (i.e., the y-coordinate axis) of the plane coordinate system is the height direction of the video frame.

[0061] In an optional example, a group of eye key points in the present disclosure corresponds to at least one area-based eye parameter. For example, a group of eye key points in the present disclosure corresponds to an eye visible area. For another example, a group of eye key points in the present disclosure corresponds to an iris visible area. For another example, a group of eye key points in the present disclosure corresponds to an eye visible area and an iris visible area.

[0062] In an optional example, a set of eye key points in the present disclosure corresponds to at least one distance-based eye parameter. For example, a set of eye key points in the present disclosure corresponds to the maximum distance between the upper eyelid and the lower eyelid in the vertical direction. For another example, a set of eye key points in the present disclosure corresponds to the maximum distance of the visible iris in the vertical direction. For another example, a set of eye key points in the present disclosure corresponds to the maximum distance between the upper eyelid and the lower eyelid in the vertical direction and the maximum distance of the visible iris in the vertical direction.

[0063] In an optional example, a set of eye key points disclosed herein corresponds to at least one area-based eye parameter and at least one distance-based eye parameter. For example, a set of eye key points disclosed herein corresponds to at least one of the eye visible area and the iris visible area, and at least one of the maximum distance between the upper eyelid and the lower eyelid in the vertical direction and the maximum distance of the visible iris in the vertical direction.

[0064] S401: Determine an eye closure degree sequence of a target object according to area-based eye parameters and / or distance-based eye parameters corresponding to each group of eye key points.

[0065] Optionally, the present disclosure may determine the eye closure corresponding to each group of eye key points according to the area-based eye parameters corresponding to each group of eye key points (such as the area of ​​the visible area of ​​the eye and / or the area of ​​the visible area of ​​the iris, etc.), thereby forming an eye closure sequence of the target object using the eye closure corresponding to each group of eye key points. The present disclosure may also determine the eye closure corresponding to each group of eye key points according to the distance-based eye parameters corresponding to each group of eye key points (such as the maximum distance between the upper eyelid and the lower eyelid in the vertical direction, and / or the maximum distance of the visible iris in the vertical direction, etc.), thereby forming an eye closure sequence of the target object using the eye closure corresponding to each group of eye key points. The present disclosure may also determine the eye closure corresponding to each group of eye key points according to the area-based eye parameters corresponding to each group of eye key points and the distance-based eye parameters corresponding to each group of eye key points, thereby forming an eye closure sequence of the target object using the eye closure corresponding to each group of eye key points.

[0066] Optionally, for any video frame among the multiple video frames, the present disclosure may calculate the ratio of the visible area of ​​the eye of the target object in the video frame to the first predetermined area of ​​the target object to obtain a first ratio, and the present disclosure may use the first ratio as the eye closure corresponding to the video frame. Using the method of obtaining the first ratio, the present disclosure may obtain the eye closure corresponding to each of the multiple video frames, thereby forming an eye closure sequence of the target object. The first predetermined area area in the present disclosure may be a pre-set known value. For example, the first predetermined area area may be the average of the maximum areas of the visible areas of the eyes of multiple target objects. For another example, the first predetermined area area may be the maximum area of ​​the visible area of ​​the eyes of a target object. In a more specific example, the present disclosure may prompt the target object to face the camera device and avoid blinking, so that the present disclosure may obtain multiple video frames including the front face of the target object through the camera device, and the present disclosure may select a video frame with the largest eye opening of the target object from the multiple video frames, and calculate the visible area of ​​the eyes using the positions of the key points of each eye related to the eyelid of the target object in the video frame, and use the calculation result as the first predetermined area area.

[0067] Optionally, for any video frame among the multiple video frames, the present disclosure may calculate the ratio of the maximum distance between the upper eyelid and the lower eyelid of the target object in the video frame in the vertical direction (i.e., the y-axis direction) to the first predetermined distance of the target object to obtain a second ratio, and the present disclosure may use the second ratio as the eye closure corresponding to the video frame. Using the method of obtaining the second ratio, the present disclosure can obtain the eye closure corresponding to each video frame in the multiple video frames, thereby forming an eye closure sequence of the target object. The first predetermined distance in the present disclosure may be a pre-set known value. For example, the first predetermined distance may be the average of the maximum eye opening distances of multiple target objects. For another example, the first predetermined distance may be the maximum eye opening distance of a target object. In a more specific example, the present disclosure can prompt the target object to face the camera device and avoid blinking, so that the present disclosure can obtain multiple video frames including the front face of the target object through the camera device. The present disclosure can select a video frame in which the eyes of the target object are most open from the multiple video frames, and use the positions of each eye key point related to the eyelids of the target object in the video frame to calculate the maximum distance of the eye opening, and use the calculation result as the first predetermined distance.

[0068] Optionally, for any video frame among the multiple video frames, the present disclosure may calculate the ratio of the visible iris area of ​​the target object in the video frame to the second predetermined area of ​​the target object to obtain a third ratio, and the present disclosure may use the third ratio as the eye closure corresponding to the video frame. Using the method of obtaining the third ratio, the present disclosure may obtain the eye closure corresponding to each video frame in the multiple video frames, thereby forming an eye closure sequence of the target object. The second predetermined area area in the present disclosure may be a pre-set known value. For example, the second predetermined area area may be the average of the maximum areas of the visible iris areas of the multiple target objects. For another example, the second predetermined area area may be the maximum area of ​​the visible iris area of ​​a target object. In a more specific example, the present disclosure may prompt the target object to face the camera device and avoid blinking, so that the present disclosure may obtain multiple video frames including the front face of the target object through the camera device, and the present disclosure may select a video frame with the largest eye opening of the target object from the multiple video frames, and calculate the visible iris area area using the positions of the key points of the eyes of the target object related to the iris in the video frame, and use the calculation result as the second predetermined area area.

[0069] Optionally, for any video frame among the multiple video frames, the present disclosure may calculate the ratio of the maximum distance of the visible iris of the target object in the video frame in the vertical direction to the second predetermined distance of the target object to obtain a fourth ratio, and the present disclosure may use the fourth ratio as the eye closure corresponding to the video frame. Using the method of obtaining the fourth ratio, the present disclosure can obtain the eye closure corresponding to each video frame in the multiple video frames, thereby forming an eye closure sequence of the target object. The second predetermined distance in the present disclosure may be a pre-set known value. For example, the second predetermined distance may be the average value of the maximum distances of the visible irises of multiple target objects in the vertical direction. For another example, the first predetermined distance may be the maximum distance of the visible iris of a target object in the vertical direction. In a more specific example, the present disclosure can prompt the target object to face the camera device and avoid blinking, so that the present disclosure can obtain multiple video frames including the front face of the target object through the camera device. The present disclosure can select a video frame with the target object's eyes open the most from the multiple video frames, and use the positions of various eye key points related to the iris of the target object in the video frame to calculate the maximum distance of the visible iris in the vertical direction, and use the calculation result as the second predetermined distance.

[0070] Optionally, for any video frame among the multiple video frames, the present disclosure may take any two ratios of the first ratio, the second ratio, the third ratio and the fourth ratio calculated for the video frame as the eye closure corresponding to the video frame, so that the present disclosure can obtain the eye closure corresponding to each video frame in the multiple video frames, and then form an eye closure sequence of the target object.

[0071] Optionally, for any video frame among the multiple video frames, the present disclosure may take any three ratios of the first ratio, the second ratio, the third ratio and the fourth ratio calculated for the video frame as the eye closure corresponding to the video frame, so that the present disclosure can obtain the eye closure corresponding to each video frame in the multiple video frames, and then form an eye closure sequence of the target object.

[0072] Optionally, for any video frame among the multiple video frames, the present disclosure may use the first ratio, the second ratio, the third ratio and the fourth ratio calculated for the video frame as the eye closure corresponding to the video frame, so that the present disclosure can obtain the eye closure corresponding to each video frame among the multiple video frames, and then form an eye closure sequence of the target object.

[0073] Since the present disclosure can form the eye closure sequence of the target object in a variety of ways, it is helpful to improve the flexibility of the eye closure sequence.

[0074] In an optional example, the normal blinking phenomenon and squinting phenomenon to avoid strong light of the target object usually have certain regularities in terms of eye closure. The present disclosure can set first predetermined rules for the above phenomena respectively, and the present disclosure can also set second predetermined rules for different degrees of fatigue states respectively. Therefore, in the process of detecting the fatigue state of the target object, the present disclosure can compare the eye closure sequence with the first predetermined rule and the second predetermined rule respectively, so as to eliminate false alarms of fatigue state caused by squinting to avoid strong light, etc., which is conducive to improving the detection accuracy of the fatigue state of the target object.

[0075] In an optional example, the present disclosure may also consider the head posture sequence of the target object obtained from multiple video frames in the process of determining the fatigue state of the target object by using the eye closure sequence of the target object. That is, the present disclosure may perform head posture detection of the target object on each of the multiple video frames, thereby obtaining the head posture corresponding to each video frame, and the head posture of the target object corresponding to each of the multiple video frames forms the head posture sequence of the target object; then, the present disclosure may determine the fatigue state of the target object according to the eye closure sequence and the head posture sequence of the target object. In one example, while determining the fatigue state of the target object by using the eye closure sequence of the target object, the first predetermined rule and the second predetermined rule, the present disclosure determines the fatigue state of the target object by using the head posture sequence of the target object (such as using the head posture sequence of the target object and the third predetermined rule set for the head posture, etc.), and then, the present disclosure may determine the final fatigue state of the target object by using the two determined fatigue states and the pre-set fatigue state discrimination logic. The fatigue state discrimination logic may be set according to actual needs. For example, the pre-set fatigue state discrimination logic may include but is not limited to: the final fatigue state of the target object is the more serious fatigue state of the two fatigue states, etc. The present disclosure does not limit the specific content of the fatigue state discrimination logic. Since the fatigue state of the target object is not only reflected in the eye closure, but also in the head posture (such as the head of the target object drooping due to dozing off, etc.), the present disclosure determines the fatigue state of the target object by combining the eye closure sequence of the target object and the head posture sequence of the target object, which is conducive to improving the accuracy of determining the fatigue state.

[0076] In an optional example, the present disclosure may also consider the sight direction sequence of the target object obtained from multiple video frames in the process of determining the fatigue state of the target object by using the eye closure sequence of the target object. That is, the present disclosure may perform sight direction detection of the target object on each of the multiple video frames, thereby obtaining the sight direction corresponding to each video frame, and the sight direction of the target object corresponding to each of the multiple video frames forms the sight direction sequence of the target object; then, the present disclosure may determine the fatigue state of the target object according to the eye closure sequence and the sight direction sequence of the target object. In one example, while determining the fatigue state of the target object by using the eye closure sequence of the target object, the first predetermined rule and the second predetermined rule, the present disclosure determines the fatigue state of the target object by using the sight direction sequence of the target object (such as using the sight direction sequence of the target object and the fourth predetermined rule set for the sight direction, etc.), and then, the two determined fatigue states and the pre-set fatigue state discrimination logic may be used to determine the final fatigue state of the target object. For example, the pre-set fatigue state discrimination logic may include but is not limited to: the final fatigue state of the target object is the more serious fatigue state of the two fatigue states, etc. Since the fatigue state of the target object is not only reflected in the eye closure, but also in the line of sight direction (for example, the line of sight will droop due to the drooping head, so that the line of sight is no longer looking at the direction of the vehicle, etc.), the present invention determines the fatigue state of the target object by combining the target object's eye closure sequence and the target object's line of sight direction sequence, which is conducive to improving the accuracy of determining the fatigue state.

[0077] In an optional example, the present disclosure may also consider the target object's head posture sequence and sight line direction sequence obtained from multiple video frames in the process of determining the target object's fatigue state by using the target object's eye closure sequence. That is, the present disclosure may perform head posture detection on each of the multiple video frames, thereby obtaining the head posture corresponding to each video frame, and the head posture of the target object corresponding to each of the multiple video frames forms the target object's head posture sequence; the present disclosure may also perform sight line direction detection on each of the multiple video frames, thereby obtaining the sight line direction corresponding to each video frame, and the sight line direction of the target object corresponding to each of the multiple video frames forms the target object's sight line direction sequence; thereafter, the present disclosure may determine the target object's fatigue state based on the target object's eye closure sequence, head posture sequence, and sight line direction sequence. In one example, while the fatigue state of the target object is determined by using the target object's eye closure sequence, the first predetermined rule, and the second predetermined rule, the fatigue state of the target object is determined by using the target object's head posture sequence (such as using the target object's head posture sequence and the third predetermined rule set for the head posture, etc.), and the fatigue state of the target object is determined by using the target object's line of sight direction sequence (such as using the target object's line of sight direction sequence and the fourth predetermined rule set for the line of sight direction, etc.), and then, the three determined fatigue states and the pre-set fatigue state discrimination logic can be used to determine the final fatigue state of the target object. For example, the pre-set fatigue state discrimination logic may include but is not limited to: the final fatigue state of the target object is the more serious fatigue state among the three fatigue states, etc. Since the fatigue state of the target object will not only be reflected in the eye closure situation, but also in the head posture and line of sight direction (such as the head will droop due to dozing, the line of sight direction will droop due to the drooping head, etc.), therefore, the present disclosure determines the fatigue state of the target object by combining the target object's eye closure sequence, the target object's head posture sequence, and the target object's line of sight direction sequence, which is conducive to improving the accuracy of determining the fatigue state.

[0078] In an optional example, the present disclosure performs head posture detection processing and sight direction detection processing on any video frame in multiple video frames to obtain an example of the head posture and sight direction of the target object in the video frame, such as Figure 5 shown.

[0079] Figure 5 In step S500, a head image block and an eye image block of a target object in a video frame are obtained.

[0080] Optionally, the present disclosure may utilize head / face region positioning technology to perform positioning processing on the head region of the target object in the video frame, so that the head image block of the target object can be cut out from the video frame. The present disclosure may utilize eye region positioning technology to perform positioning processing on the eye region of the target object in the video frame, so that the eye image block of the target object can be cut out from the video frame. In addition, the present disclosure may also obtain the eye image block of the target object from the head image block of the target object. For example, the present disclosure may determine the eye region in the head image block based on the predetermined positional relationship between the head and the eye, and cut out the eye image block of the target object from the head image block. The predetermined positional relationship may include: the positional relationship between the highest position of the eye and the entire height of the head image block, the positional relationship between the lowest position of the eye and the entire height of the head image block, and the positional relationship between the height of the eye and the entire height of the head image block, etc.

[0081] Optionally, the eye image block in the present disclosure may be an image block including both eyes. The eye image block in the present disclosure may also be an image block including a single eye. For example, an image block including the left eye or an image block including the right eye.

[0082] Optionally, the spatial resolution of the head image block and the spatial resolution of the eye image block in the present disclosure may be the same. For example, the present disclosure may adjust the spatial resolution of the cut-out head area and the spatial resolution of the eye area to predetermined spatial resolutions, respectively, thereby obtaining a head image block and an eye image block with the same spatial resolution.

[0083] S501, extracting features of a head image block and an eye image block to obtain a first feature map of the head image block and a second feature map of the eye image block.

[0084] Optionally, the present disclosure can provide the head image block and the eye image block as inputs to a neural network for extracting image features, and obtain a first feature map of the head image block and a second feature map of the eye image block based on the output of the neural network. In one example, the neural network for extracting image features can be a multi-task learning neural network, that is, the present disclosure can use a feature extraction operation based on a multi-task learning neural network to obtain a first feature map of the head image block and a second feature map of the eye image block.

[0085] Optionally, the multi-task learning neural network in the present disclosure generally includes a trunk part (i.e., a trunk unit) and multiple branch parts (i.e., multiple branch units), and the trunk part and all the branch parts can be combined together to complete multiple tasks at the same time. For example, the head posture detection task and the line of sight direction detection task can be completed at the same time.

[0086] Optionally, an example of a neural network for multi-task learning in the present disclosure is as follows: Figure 6 shown. Figure 6 The multi-task learning neural network includes: a trunk unit 600 and at least two branch units. Figure 6 The branch unit 601 and the branch unit 602 are schematically shown in FIG.

[0087] Optionally, the present disclosure can provide the head image block and the eye image block as inputs of the backbone unit 600 of the neural network for multi-task learning, so that based on the feature extraction operation of the backbone unit 600 of the neural network, the head image block and the eye image block can be subjected to feature extraction operations, respectively. Then, the present disclosure can obtain a first feature map of the head image block and a second feature map of the eye image block according to the output of the backbone unit 600.

[0088] Optionally, the backbone unit 600 in the present disclosure may adopt a structure such as CNN (Convolutional Neural Networks). In one example, the backbone unit 600 may include: multiple convolutional layers and multiple activation layers. In another example, the backbone unit 600 may include: multiple convolutional layers, multiple activation layers, at least one pooling layer and at least one normalization layer. The present disclosure does not limit the specific structure of the backbone unit 600.

[0089] Optionally, the present disclosure may splice the head image block and the eye image block together and provide them to the neural network as the input of the backbone unit 600. The present disclosure may perform a segmentation operation on the output of the backbone unit 600 to obtain a first feature map of the head image block and a second feature map of the eye image block.

[0090] Since in the training process of the neural network for multi-task learning, the losses corresponding to different branches will be back-propagated to the backbone unit of the multi-task learning, it is beneficial to make the features extracted by the backbone unit after successful training more accurate. Therefore, the feature extraction operation of the backbone unit of the neural network used in the present disclosure is conducive to obtaining more accurate first feature maps and second feature maps.

[0091] S502: Acquire the head posture of the target object in the video frame according to the first feature map.

[0092] Optionally, the present disclosure may perform head posture detection processing on the first feature map, and obtain the head posture (Head Pose) of the target object in the video frame according to the result of the head posture detection processing. The head posture of the target object in the present disclosure may refer to information used to describe the rotation of the head of the target object on at least one of the X, Y and Z coordinate axes. The head posture may also be referred to as the posture angle of the head. For example, the head posture of the present disclosure may include three Euler angles, namely the pitch angle, yaw angle and roll angle of the head. The pitch angle is the rotation angle pitch around the X coordinate axis of the target object's head coordinate system, and the pitch angle can indicate the head-up situation of the target object. The yaw angle is the rotation angle yaw around the Y coordinate axis of the target object's head coordinate system, and the yaw angle can indicate the head shaking situation of the target object. The roll angle is the rotation angle roll around the Z coordinate axis of the target object's head coordinate system, and the roll angle can indicate the head turning situation of the target object.

[0093] Optionally, the present disclosure may provide the first feature map as input to a neural network for detecting head posture, and obtain the head posture of the target object based on the output of the neural network, such as pitch (pitch angle), yaw (yaw angle) and roll (roll angle). In one example, the neural network for detecting head posture may be a multi-task learning neural network, that is, the present disclosure may utilize the head posture detection operation based on the multi-task learning neural network to obtain the head posture of the target object. In a more specific example, the present disclosure may utilize one of the branch units of the multi-task learning neural network to complete the head posture detection operation. For example, the present disclosure may use the first feature map output by the trunk unit 400 as the first branch unit of the multi-task learning neural network (such as Figure 4 The first branch unit in the present disclosure may adopt a structure such as CNN. In one example, the first branch unit may include: at least one convolutional layer and at least one activation layer, etc. The present disclosure does not limit the specific structure of the first branch unit.

[0094] The present invention completes the head posture detection operation by utilizing one of the branch units of the multi-task learning neural network. Since the first feature map extracted by the trunk unit of the multi-task learning neural network is more accurate, during the training process of the multi-task learning neural network, a more accurate first feature map is beneficial to optimizing the network parameters of the first branch unit, thereby helping to improve the accuracy of obtaining the head posture.

[0095] S503: Acquire the sight direction of the target object in the video frame according to the second feature map and the head posture.

[0096] Optionally, the present disclosure may perform gaze direction detection processing on the second feature map and the head posture, and obtain the gaze direction of the target object in the video frame based on the result of the gaze direction detection processing. The present disclosure may provide the second feature map and the head posture as input to a neural network for detecting the gaze direction, and obtain the gaze direction of the target object based on the output of the neural network. In one example, the neural network for detecting the gaze direction may be a multi-task learning neural network, that is, the present disclosure may utilize the gaze direction detection operation based on the multi-task learning neural network to obtain the gaze direction of the target object. More specifically, the present disclosure may utilize one of the branch units of the multi-task learning neural network to complete the gaze direction detection operation. For example, the present disclosure may combine the second feature map output by the trunk unit with the above-mentioned first branch unit (such as Figure 6 The head posture output by the branch unit 601 in FIG. 6 is used as the second branch unit of the neural network for multi-task learning (such as Figure 6 The second branch unit in the present disclosure may adopt a network structure such as CNN. In one example, the second branch unit may include: at least one convolutional layer and at least one activation layer, etc. The present disclosure does not limit the specific structure of the second branch unit.

[0097] Optionally, in the case where the eye image block is an image block containing both eyes, the second feature map provided by the present disclosure for the second branch unit is a second feature map based on both eyes, and the gaze direction detection operation performed by the second branch unit may be a gaze direction detection operation based on the left eye and a gaze direction detection operation based on the right eye. The present disclosure may obtain the left eye gaze direction and the right eye gaze direction, respectively, according to the output of the second branch unit. The present disclosure may perform weighted calculation on the left eye gaze direction and the right eye gaze direction to obtain a weighted average value, and the present disclosure may use the weighted average value as the gaze direction of the target object.

[0098] It should be specially noted that, when performing weighted calculation in the present disclosure, the weights corresponding to the left eye sight direction and the right eye sight direction may be the same or different.

[0099] Optionally, in the case where the eye image block is an image block containing a single eye, the second feature map provided by the present disclosure for the second branch unit is a second feature map based on a single eye, and the sight direction detection operation performed by the second branch unit may be a sight direction detection operation based on a single eye, for example, a sight direction detection operation based on the left eye or a sight direction detection operation based on the right eye. The present disclosure may obtain the left eye sight direction or the right eye sight direction based on the output of the second branch unit, and the present disclosure may directly use the left eye sight direction or the right eye sight direction as the sight direction of the target object.

[0100] The present disclosure utilizes a multi-task learning neural network to implement feature extraction processing, head posture detection processing, and line of sight direction detection processing, and in particular utilizes one of the branch units of the multi-task learning neural network to complete the line of sight direction detection operation. Since the second feature map extracted by the trunk unit of the multi-task learning neural network is more accurate, and there is a certain degree of correlation between the head posture and the line of sight direction, therefore, in the training process of the multi-task learning neural network, the more accurate second feature map and head posture are conducive to optimizing the network parameters of the second branch unit, thereby utilizing the second branch unit of the neural network to implement the line of sight direction detection operation, which is conducive to improving the accuracy of the line of sight direction. In addition, the present disclosure can utilize the left eye line of sight direction and / or the right eye line of sight direction of the target object to obtain the line of sight direction of the target object, which is conducive to improving the flexibility of the technical solution.

[0101] The present disclosure obtains the sight direction of the target object by utilizing the second feature map of the eye image block and the head posture of the target object. Since there is a certain correlation between the head posture and the realization direction, the present disclosure is conducive to reducing the error of the determined sight direction, thereby helping to improve the accuracy of the sight direction.

[0102] In an optional example, for any video frame in a plurality of video frames, the present disclosure can use the above-mentioned multi-task learning neural network to obtain a set of eye key points of the target object in the video frame. For example, the above-mentioned second branch unit can not only perform sight line direction detection processing, but also simultaneously perform eye key point detection processing, that is, after the second feature map is provided to the second branch unit, the second branch unit performs eye key point detection processing on the second feature map to obtain multiple eye key points in the eye image block (such as obtaining the label of each eye key point in the eye image block and the two-dimensional coordinates of each eye key point in the eye image block, etc.), and the present disclosure can obtain a set of eye key points corresponding to the video frame (such as a set of eye key points based on the left eye or a set of eye key points based on the right eye) according to the output of the second branch unit. Since the two branch units of the multi-task learning neural network can be used to obtain a set of eye key points corresponding to the video frame, and can also be used to obtain the head posture and sight line direction of the target object in the video frame, the present disclosure can conveniently obtain a set of eye key points corresponding to the video frame while making full use of computing resources, thereby helping to save computing resources.

[0103] Exemplary Devices

[0104] Figure 7 This is a schematic diagram of the structure of an embodiment of a service discovery device of the present disclosure. The device of this embodiment can be used to implement the corresponding method embodiment of the present disclosure.

[0105] Figure 7 The device shown includes: a key point detection module 700, a closure sequence determination module 701, and a fatigue state determination module 702. The device may also optionally include: a detection module 703.

[0106] The key point detection module 700 is used to determine a group of eye key points of a target object in each video frame of a plurality of video frames, wherein the group of eye key points of the target object in each video frame forms an eye key point group sequence.

[0107] The closure sequence determination module 701 is used to determine the eye closure sequence of the target object according to each group of eye key points in the eye key point group sequence obtained by the key point detection module 700 .

[0108] The fatigue state determination module 702 is used to determine the fatigue state of the target object according to the eye closure sequence of the target object obtained by the closure sequence determination module 701 .

[0109] Optionally, the closure sequence determination module 701 in the present disclosure may include: a first submodule 7011 and a second submodule 7012. The first submodule 7011 is used to determine the area-based eye parameters and / or distance-based eye parameters corresponding to each group of eye key points in the eye key group sequence obtained by the key point detection module 700. The second submodule 7012 is used to determine the eye closure sequence of the target object according to the area-based eye parameters and / or distance-based eye parameters corresponding to each group of eye key points obtained by the first submodule 7011.

[0110] Optionally, the first submodule 7011 can determine at least one of the eye visible area area, iris visible area area, maximum distance between the upper eyelid and the lower eyelid in the vertical direction, and maximum distance of the visible iris in the vertical direction corresponding to each group of eye key points in the eye key point group sequence obtained by the key point detection module 700.

[0111] Optionally, the second submodule 7012 can determine the eye closure sequence of the target object based on at least one of the ratio of the area of ​​the target object's eye visible area in multiple video frames obtained by the first submodule 7011 to the area of ​​a first predetermined area of ​​the target object, the ratio of the area of ​​the iris visible area obtained by the first submodule 7011 to the area of ​​a second predetermined area of ​​the target object, the ratio of the maximum distance between the upper eyelid and the lower eyelid in the vertical direction obtained by the first submodule 7011 to the first predetermined distance of the target object, and the ratio of the maximum distance of the visible iris in the vertical direction obtained by the first submodule 7011 to the second predetermined distance of the target object.

[0112] Optionally, the detection module 703 is used to detect at least one of the head posture and the line of sight direction of the target object in any video frame among the multiple video frames, the head posture of the target object in each of the multiple video frames forming a head posture sequence of the target object, and the video line direction of the target object in each of the multiple video frames forming a line of sight direction sequence of the target object.

[0113] Optionally, the fatigue state determination module 702 may be used to determine the fatigue state of the target object according to the eye closure sequence of the target object obtained by the closure sequence determination module 701 and the head posture sequence of the target object obtained by the detection module 703 .

[0114] Optionally, the fatigue state determination module 702 may be used to determine the fatigue state of the target object according to the eye closure sequence of the target object obtained by the closure sequence determination module 701 and the sight direction sequence of the target object obtained by the detection module 703 .

[0115] Optionally, the fatigue state determination module 702 can be used to determine the fatigue state of the target object based on the eye closure sequence of the target object obtained by the closure sequence determination module 701, and the head posture sequence of the target object and the sight direction sequence of the target object obtained by the detection module 703.

[0116] Optionally, the detection module 703 may include: a third submodule 7031, a fourth submodule 7032, a fifth submodule 7033, and a sixth submodule 7034. The third submodule 7031 is used to obtain the head image block and the eye image block of the target object in any video frame of the multiple video frames. The fourth submodule 7032 is used to extract the features of the head image block and the eye image block obtained by the third submodule 7031, and obtain the first feature map of the head image block and the second feature map of the eye image block. The fifth submodule 7033 is used to obtain the head posture of the target object in the video frame according to the first feature map obtained by the fourth submodule 7032. The sixth submodule 7034 is used to obtain the sight direction of the target object in the video frame according to the second feature map obtained by the fourth submodule 7032 and the head posture of the target object in the video frame obtained by the fifth submodule 7033.

[0117] Optionally, the fourth submodule 7032 in the present disclosure may include a trunk unit in a neural network based on multi-task learning. The fifth submodule 7033 in the present disclosure may include a first branch unit in a neural network based on multi-task learning. The sixth submodule 7034 in the present disclosure may include a second branch unit in a neural network based on multi-task learning.

[0118] Optionally, the fourth submodule 7032 can use the head image block and the eye image block obtained by the third submodule 7031 as the input of the backbone unit of the neural network, respectively, and based on the feature extraction operation of the backbone unit, perform feature extraction processing on the head image block and the eye image block, respectively, so that the fourth submodule 7032 obtains a first feature map of the head image block and a second feature map of the eye image block.

[0119] Optionally, the fifth submodule 7033 can use the first feature map obtained by the fourth submodule 7032 as the input of the first branch unit of the neural network, and perform head posture detection processing on the first feature map based on the head posture detection operation of the first branch unit, so that the fifth submodule 7033 obtains the head posture of the target object in the video frame.

[0120] Optionally, the sixth submodule 7034 can use the second feature map obtained by the fourth submodule 7032 and the head posture obtained by the fifth submodule 7033 as the input of the second branch unit of the neural network, and based on the gaze direction detection operation of the second branch unit, perform gaze direction detection processing on the second feature map and the head posture of the target object in the video frame, so that the sixth submodule 7034 obtains the gaze direction of the target object in the video frame.

[0121] Optionally, for any video frame among the multiple video frames, based on the eye key point detection operation of the second branch unit, the second feature map is subjected to eye key point detection processing, so that the key point detection module 700 obtains a set of eye key points of the target object in the video frame. The key point detection module 700 may include a portion of the second branch unit for performing eye key point detection processing.

[0122] The modules, sub-modules, units and specific operations performed by the respective units included in the device of the present disclosure can be found in the relevant descriptions in the above method embodiments, and will not be described in detail here.

[0123] Exemplary Electronic Devices

[0124] Reference below Figure 8 An electronic device according to an embodiment of the present disclosure is described. Figure 8 1 shows a block diagram of an electronic device according to an embodiment of the present disclosure. Figure 8 As shown, the electronic device 81 includes one or more processors 811 and a memory 812 .

[0125] The processor 811 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 81 to perform desired functions.

[0126] The memory 812 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, may include: random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory, for example, may include: read-only memory (ROM), hard disk, and flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 811 may run the program instructions to implement the methods for fatigue detection of the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0127] In one example, the electronic device 81 may also include: an input device 813 and an output device 814, etc., which are interconnected through a bus system and / or other forms of connection mechanisms (not shown). In addition, the input device 813 may also include, for example, a keyboard, a mouse, etc. The output device 814 may output various information to the outside. The output device 814 may include, for example, a display, a speaker, a printer, and a communication network and a remote output device connected thereto, etc.

[0128] Of course, to simplify, Figure 8 Only some of the components related to the present disclosure in the electronic device 81 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 81 may also include any other appropriate components.

[0129] Exemplary computer program products and computer-readable storage media

[0130] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the method for fatigue detection according to various embodiments of the present disclosure or the probability prediction method based on a language model described in the above "Exemplary Method" section of this specification.

[0131] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0132] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the method for fatigue detection according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0133] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive enumeration) of readable storage media can include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0134] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0135] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0136] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present disclosure are intended only as illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems may be connected, arranged, and configured in any manner. Words such as "include," "comprise," "have," and the like are open words, meaning "including but not limited to," and may be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and may be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and may be used interchangeably therewith.

[0137] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0138] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0139] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects, etc., will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0140] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for fatigue detection, comprising: Determine a group of eye key points of the target object in each video frame of the plurality of video frames, wherein the group of eye key points of the target object in each video frame forms an eye key point group sequence; Determine an eye closure degree sequence of the target object according to each group of eye key points in the eye key point group sequence, wherein the eye closure degree sequence includes a plurality of eye closure degrees, each eye closure degree represents the degree of eye closure of the target object at a time point, and the eye closure degree sequence is formed by arranging the plurality of eye closure degrees in the order of time of the video frames corresponding to each of the plurality of eye closure degrees; determining a fatigue state of the target object according to the eye closure degree sequence of the target object; Wherein, determining the fatigue state of the target object according to the eye closure degree sequence of the target object includes: According to the m eye closure sequences of the target object, a first predetermined rule and a second predetermined rule, the m fatigue states of the target object are determined, wherein the first predetermined rule includes the eye closure rules corresponding to normal blinking and squinting, and the second predetermined rule includes the eye closure rules corresponding to fatigue states of different degrees, m is an integer greater than 1, and the m fatigue states constitute a fatigue state sequence for determining whether to generate fatigue warning information, each fatigue state in the fatigue state sequence corresponds to an eye closure sequence, and the intersection of the eye closure sequences corresponding to two adjacent fatigue states in the fatigue state sequence includes at least one eye closure.

2. The method according to claim 1, wherein: The step of determining the eye closure degree sequence of the target object according to each group of eye key points in the eye key point group sequence comprises: Determining, according to each group of eye key points in the eye key group sequence, an area-based eye parameter and / or a distance-based eye parameter corresponding to each group of eye key points; The eye closure degree sequence of the target object is determined according to the area-based eye parameters and / or distance-based eye parameters corresponding to each group of eye key points.

3. The method according to claim 2, wherein: The step of determining, according to each group of eye key points in the eye key point group sequence, the area-based eye parameter and / or the distance-based eye parameter corresponding to each group of eye key points, comprises: According to each group of eye key points in the eye key point group sequence, determine at least one of the eye visible area area, iris visible area area, maximum distance between the upper eyelid and the lower eyelid in the vertical direction, and maximum distance of the visible iris in the vertical direction corresponding to each group of eye key points.

4. The method according to claim 3, wherein: The step of determining the eye closure degree sequence of the target object according to the area-based eye parameters and / or distance-based eye parameters corresponding to each group of eye key points includes: The eye closure sequence of the target object is determined based on at least one of the ratio of the area of ​​the target object's eye visible area in the multiple video frames to the area of ​​the first predetermined area of ​​the target object, the ratio of the area of ​​the iris visible area to the area of ​​the second predetermined area of ​​the target object, the ratio of the maximum distance between the upper eyelid and the lower eyelid in the vertical direction to the first predetermined distance of the target object, and the ratio of the maximum distance of the visible iris in the vertical direction to the second predetermined distance of the target object.

5. The method according to any one of claims 1 to 4, wherein: The method further comprises: For any video frame among the multiple video frames, detecting at least one of a head posture and a sight line direction of a target object in the video frame, wherein the head posture of the target object in each of the multiple video frames forms a head posture sequence of the target object, and the video line direction of the target object in each of the multiple video frames forms a sight line direction sequence of the target object; The step of determining the fatigue state of the target object according to the eye closure degree sequence of the target object comprises: determining the fatigue state of the target object according to the eye closure degree sequence of the target object and the head posture sequence of the target object; or determining the fatigue state of the target object according to the eye closure degree sequence of the target object and the sight direction sequence of the target object; or The fatigue state of the target object is determined according to the eye closure degree sequence of the target object, the head posture sequence of the target object, and the sight direction sequence of the target object.

6. The method according to claim 5, wherein: The detecting, for any video frame among the multiple video frames, at least one of a head posture and a sight line direction of a target object in the video frame comprises: For any video frame among the multiple video frames, obtaining a head image block and an eye image block of a target object in the video frame; Extracting features of the head image block and the eye image block to obtain a first feature map of the head image block and a second feature map of the eye image block; Acquire the head posture of the target object in the video frame according to the first feature map; The sight direction of the target object in the video frame is obtained according to the second feature map and the head posture of the target object in the video frame.

7. The method according to claim 6, wherein: The steps of extracting features of the head image block and the eye image block to obtain a first feature map of the head image block and a second feature map of the eye image block, the step of obtaining the head posture of the target object in the video frame according to the first feature map, and the step of obtaining the sight direction of the target object in the video frame according to the second feature map and the head posture of the target object in the video frame are implemented by using a feature extraction operation, a head posture detection operation, and a sight direction detection operation of a neural network based on multi-task learning.

8. The method according to claim 7, wherein: The step of extracting features of the head image block and the eye image block to obtain a first feature map of the head image block and a second feature map of the eye image block includes: The head image block and the eye image block are respectively used as inputs of the backbone unit of the neural network, and based on the feature extraction operation of the backbone unit, feature extraction processing is performed on the head image block and the eye image block to obtain a first feature map of the head image block and a second feature map of the eye image block; The step of acquiring the head posture of the target object in the video frame according to the first feature map includes: Using the first feature map as an input of a first branch unit of the neural network, and performing head posture detection processing on the first feature map based on a head posture detection operation of the first branch unit to obtain a head posture of the target object in the video frame; The acquiring the sight direction of the target object in the video frame according to the second feature map and the head posture of the target object in the video frame includes: The second feature map and the head posture are used as inputs of the second branch unit of the neural network. Based on the gaze direction detection operation of the second branch unit, the second feature map and the head posture of the target object in the video frame are subjected to gaze direction detection processing to obtain the gaze direction of the target object in the video frame.

9. The method according to claim 8, wherein: The step of determining a set of eye key points of the target object in each of the plurality of video frames comprises: For any video frame among the multiple video frames, based on the eye key point detection operation of the second branch unit, the second feature map is processed by eye key point detection to obtain a group of eye key points of the target object in the video frame.

10. A device for fatigue detection, comprising: A key point detection module is used to determine a group of eye key points of a target object in each video frame of a plurality of video frames, wherein the group of eye key points of the target object in each video frame forms an eye key point group sequence; a closure sequence determination module, configured to determine an eye closure sequence of the target object according to each group of eye key points in the eye key point group sequence obtained by the key point detection module, wherein the eye closure sequence includes a plurality of eye closures, each of which represents the degree of eye closure of the target object at a time point, and the eye closure sequence is formed by arranging the plurality of eye closures in the order of time of the video frames corresponding to each of the plurality of eye closures; a fatigue state determination module, configured to determine the fatigue state of the target object according to the eye closure sequence of the target object obtained by the closure sequence determination module; Wherein, the fatigue state determination module is specifically used for: According to the m eye closure sequences of the target object, a first predetermined rule and a second predetermined rule, the m fatigue states of the target object are determined, wherein the first predetermined rule includes the eye closure rules corresponding to normal blinking and squinting, and the second predetermined rule includes the eye closure rules corresponding to fatigue states of different degrees, m is an integer greater than 1, and the m fatigue states constitute a fatigue state sequence for determining whether to generate fatigue warning information, each fatigue state in the fatigue state sequence corresponds to an eye closure sequence, and the intersection of the eye closure sequences corresponding to two adjacent fatigue states in the fatigue state sequence includes at least one eye closure.

11. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 9.

12. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1 to 9.

Images