A yawp detection method, device and equipment

Abstract

The application provides a yawning detection method and device, electronic equipment and medium. The method acquires a plurality of continuous face images, and extracts face key point coordinates in each face image. Based on the face key point coordinates, a plurality of target features of each face image are calculated. The plurality of target features include a lower jaw displacement feature, an eye closure degree feature and a mouth opening degree feature. Based on the plurality of target features of each face image, a single-frame yawning result is determined. According to the change trend of the lower jaw displacement feature in the plurality of continuous face images, a yawning stage of each face image is determined. The yawning stage includes an opening period, a peak period and a closing period. Based on the single-frame yawning result and the yawning stage of the plurality of continuous face images, a yawning detection result is determined, thereby improving the accuracy of yawning detection.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and more specifically, to a yawn detection method, apparatus, and device. Background Technology

[0002] Existing yawn detection technologies mainly suffer from the following problems: some methods rely solely on a single feature of the mouth (such as the mouth's aspect ratio MAR) for judgment, which is prone to misjudgment; moreover, treating yawning as a static state or using a simple frame counting method leads to missed detection of atypical yawning actions such as small amplitude or occlusion; finally, when interference common in real-world scenarios occurs, such as frame loss, sudden changes in lighting, or brief occlusion, the yawn detection logic will fail due to these interferences. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a yawn detection method, apparatus and equipment that can improve the accuracy and stability of yawn detection.

[0004] This application provides a yawn detection method, the method comprising: Acquire multiple consecutive frames of face images and extract the coordinates of facial key points in each frame; Based on the coordinates of the facial key points, multiple target features are calculated for each frame of the face image; the multiple target features include jaw displacement features, eye closure degree features, and mouth opening degree features; Based on multiple target features of each frame of the face image, the result of yawning in a single frame is determined; Based on the changing trend of the mandibular displacement features in the continuous multi-frame face images, the yawning stage of each face image is determined; the yawning stage includes the mouth opening stage, the peak stage, and the closing stage; Based on the single-frame yawning result and yawning stage of the continuous multi-frame face images, the yawning detection result is determined.

[0005] In some embodiments, the yawn detection method, determining the yawn result of a single frame based on multiple target features of each frame of facial image, includes: Based on multiple target features, various yawning evaluation results are determined for the face image frame; the yawning evaluation results include qualified and unqualified. A value of 1 is assigned to the yawning evaluation result that meets the standard, and a value of 0 is assigned to the yawning evaluation result that does not meet the standard. The weighted sum of the values ​​corresponding to the various yawning evaluation results is used to obtain the yawning score for a single frame. Based on the comparison between the single-frame yawn score and the first preset threshold, the single-frame yawn result is determined; the single-frame yawn result includes yawning or not yawning.

[0006] In some embodiments, the yawning detection method, wherein determining multiple yawning evaluation results for the face image frame based on multiple target features includes: Determine whether the mandibular displacement feature is greater than a second preset threshold, whether the eye closure feature is less than a third preset threshold, and whether the mouth opening feature is less than a fourth preset threshold; When the mandibular displacement feature is greater than the second preset threshold, the initial yawning judgment result corresponding to the mandibular displacement feature is qualified; When the degree of eye closure is less than the third preset threshold, the initial yawning judgment result corresponding to the degree of eye closure is qualified; When the mouth opening degree feature is less than the fourth preset threshold, the initial yawning judgment result corresponding to the mouth opening degree feature is qualified.

[0007] In some embodiments, the yawning detection method, the mandibular displacement feature is calculated based on the following steps: Select the coordinates of the chin tip and the nose tip from the facial key points in the current frame to determine the chin-nose tip distance in the current frame; Using the chin-nose tip distance of the first frame in the continuous multi-frame face images as the first reference value, the rate of change of the chin-nose tip distance of the current frame relative to the first reference value is calculated, and the rate of change is used as the chin displacement feature.

[0008] In some embodiments, the yawning detection method, determining the yawning stage of each frame of the face image based on the changing trend of the jaw displacement features in the consecutive multi-frame face images, includes: Select a comparison frame that is preceding the current frame and meets the preset comparison conditions from the continuous multi-frame face images; If the ratio of the mandibular displacement feature of the current frame to that of the comparison frame is greater than the fifth preset threshold, the current frame is determined to be in the mouth opening phase. If the ratio of the mandibular displacement feature of the current frame to that of the comparison frame is less than the sixth preset threshold, the current frame is determined to be in the closure period. If the rate of change of the mandibular displacement feature of the current frame relative to the preset benchmark value is greater than the seventh preset threshold, and the mandibular displacement feature of the current frame meets the preset distance stability condition, then the current frame is determined to be in the peak period. If none of the above conditions are met, the current frame is determined to be in the normal period.

[0009] In some embodiments, the yawning detection method, determining the yawning detection result based on the single-frame yawning result and yawning stage of the continuous multi-frame face images, includes: Based on the yawning result of each frame of the continuous multi-frame face images, a frame ratio score is determined; the frame ratio score represents the proportion of face image frames that yawn. Based on the yawning phase of each frame of the consecutive multi-frame face images, calculate the yawning pattern integrity score. The frame ratio score and pattern integrity score are combined to determine the sequence comprehensive score of the consecutive multi-frame face images; The yawn detection result is determined based on the frame ratio score and the sequence comprehensive score.

[0010] In some embodiments, after determining the yawn detection result based on the frame ratio score and the sequence comprehensive score, the method further includes: Determine the duration of yawning in the consecutive multi-frame face images; When the duration is greater than or equal to the seventh preset threshold, the yawn detection result is determined to pass the verification.

[0011] In some embodiments, the yawning detection method calculates a yawning pattern integrity score based on the yawning phase of each frame in the multi-frame face images, including: Based on the yawning phase of each frame in the continuous multi-frame face images, the matching result of the mouth opening process of the continuous multi-frame face images and the preset complete yawning pattern is determined; wherein, the preset complete yawning pattern includes, in sequence: mouth opening phase, peak phase and closing phase. Based on the matching results, a pattern integrity score for yawning is determined.

[0012] In some embodiments, a yawn detection device is also provided, the device comprising: The acquisition module is used to acquire multiple consecutive frames of face images and extract the coordinates of facial key points in each frame of face image; The calculation module is used to calculate multiple target features of each frame of face image based on the coordinates of the facial key points; the multiple target features include jaw displacement features, eye closure degree features, and mouth opening degree features; The first determining module is used to determine the yawning result of a single frame based on multiple target features of each frame of face image; The second determining module is used to determine the yawning stage of each frame of face image based on the changing trend of the mandibular displacement features in the continuous multi-frame face images; the yawning stage includes the mouth opening stage, the peak stage, and the closing stage; The third determining module is used to determine the yawning detection result based on the single-frame yawning result and yawning stage of the continuous multi-frame face images.

[0013] In some embodiments, an electronic device is also provided, the electronic device including: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor communicating with the memory via the bus when the electronic device is running, and the steps of the yawn detection method being executed by the processor when the machine-readable instructions are executed.

[0014] This application provides a yawn detection method, device, and electronic device. The method acquires multiple consecutive frames of facial images and extracts the coordinates of facial key points in each frame. Based on the facial key point coordinates, it calculates multiple target features for each frame of the facial image. These multiple target features include jaw displacement features, eye closure degree features, and mouth opening degree features. Based on these multiple target features, it determines the yawning result for a single frame, fully utilizing the synergistic changes in multiple physiological responses during yawning to effectively distinguish yawning from mouth-opening actions such as speaking, singing, and laughing, thus preventing false detections. The method also considers the jaw displacement features in the multiple consecutive frames of the facial image. The changing trends of features are used to determine the yawning stage of each frame of a face image. The yawning stage includes the mouth opening period, the peak period, and the closing period. The focus is on the dynamic process of the yawning action rather than the static threshold. Yawning with a small mouth opening amplitude but conforming to a typical temporal pattern can still be accurately identified. Based on the single-frame yawning results and yawning stages of the continuous multi-frame face images, the yawning detection result is determined. By analyzing the changing trends of the mandibular displacement features in the continuous multi-frame face images, the overall feature distribution within the time window is analyzed to further improve the detection accuracy. Together, they constitute a multi-dimensional detection framework from local to global and from static to dynamic, effectively improving the robustness of the detection system. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of the yawn detection method described in an embodiment of this application is shown; Figure 2 A flowchart of the method for determining the result of a single frame of yawning as described in an embodiment of this application is shown; Figure 3 A flowchart of the method for determining the yawning stage of each frame of a face image according to an embodiment of this application is shown; Figure 4 A flowchart of the method for determining yawn detection results according to an embodiment of this application is shown; Figure 5 A schematic diagram of the yawn detection device according to an embodiment of this application is shown; Figure 6 A schematic diagram of the structure of the electronic device described in an embodiment of this application is shown. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0018] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0020] Existing yawn detection technologies mainly suffer from the following problems: some methods rely solely on a single feature of the mouth (such as the mouth's aspect ratio MAR) for judgment, which is prone to misjudgment; moreover, treating yawning as a static state or using a simple frame counting method leads to missed detection of atypical yawning actions such as small amplitude or occlusion; finally, when interference common in real-world scenarios occurs, such as frame loss, sudden changes in lighting, or brief occlusion, the yawn detection logic will fail due to these interferences.

[0021] Based on this, this application provides a yawn detection method, device, and electronic device. The method acquires multiple consecutive frames of facial images and extracts the coordinates of facial key points in each frame. Based on the facial key point coordinates, it calculates multiple target features for each frame of facial images. These multiple target features include jaw displacement features, eye closure degree features, and mouth opening degree features. Based on these multiple target features, it determines the yawning result for a single frame, fully utilizing the synergistic changes in multiple physiological responses during yawning to effectively distinguish yawning from mouth-opening actions such as speaking, singing, and laughing, thus preventing false detections. The method also considers the jaw displacement in the multiple consecutive frames of facial images. The changing trend of jaw displacement features is used to determine the yawning stage of each frame of a face image. The yawning stage includes the mouth opening period, peak period, and closing period. The focus is on the dynamic process of the yawning action rather than the static threshold. Yawning with a small mouth opening amplitude but conforming to a typical temporal pattern can still be accurately identified. Based on the single-frame yawning results and yawning stages of the continuous multi-frame face images, the yawning detection result is determined. By analyzing the changing trend of the jaw displacement features in the continuous multi-frame face images, the overall feature distribution within the time window is analyzed to further improve the detection accuracy. Together, they constitute a multi-dimensional detection framework from local to global and from static to dynamic, effectively improving the robustness of the detection system.

[0022] Please refer to Figure 1 , Figure 1 A flowchart of the yawn detection method described in an embodiment of this application is shown, as follows: Figure 1 As shown, the method includes the following steps S101-S105: S101. Acquire multiple consecutive frames of face images and extract the coordinates of facial key points in each frame of face image; S102. Based on the facial key point coordinates, calculate multiple target features for each frame of the face image; the multiple target features include jaw displacement features, eye closure degree features, and mouth opening degree features; S103. Based on the multiple target features of each frame of face image, determine the yawning result of a single frame; S104. Based on the changing trend of the mandibular displacement features in the continuous multi-frame face images, determine the yawning stage of each frame face image; the yawning stage includes the mouth opening stage, the peak stage, and the closing stage; S105. Based on the single-frame yawning result and yawning stage of the continuous multi-frame face images, determine the yawning detection result.

[0023] In this embodiment of the application, the yawn detection method can run on a terminal device or a server; wherein, the terminal device can be a local terminal device, and when the yawn detection method runs on a server, the yawn detection method can be implemented and executed based on a cloud interactive system, wherein the yawn detection method includes at least a server and a client device (i.e., a terminal device).

[0024] The following explanation uses a server as an example. The yawning detection method is applied to a yawning detection system, which is deployed on a server. The overall architecture of the system includes the following main modules: Facial landmark detection module: Based on the PFPLD (Practical Facial Landmark Detector) model, it extracts the coordinates of 106 facial landmarks from the input image; Facial feature extraction module: Calculates three types of yawn-related features from key points; Single-frame yawning detection module: Determines the probability of yawning in the current frame based on a weighted score; Timing phase identification module: Identifies the yawning phase in which the current frame is located; Sequence validity assessment module: comprehensively evaluates yawning sequences within a time window; Status output module: Outputs the final yawning status and related information.

[0025] In step S101, multiple consecutive frames of face images are acquired, and the coordinates of facial key points in each frame of face image are extracted.

[0026] The continuous multi-frame facial images are a sequence of continuous video frames containing a face, captured in real time by an image acquisition device (such as a camera).

[0027] For each frame of a series of captured facial images, a facial landmark detection model is used to extract the coordinates of facial landmarks. In this embodiment, for example, the PPFPLD (Practical Facial Landmark Detector) model is used to extract the two-dimensional coordinates (x, y) of 106 facial landmarks from each frame.

[0028] Facial key points refer to pixels in a face image that have specific semantic locations, including but not limited to the corners of the eyes, the tip of the nose, the corners of the mouth, the tip of the chin, eyebrow key points, and facial contour points. Facial key points are usually represented in the form of two-dimensional coordinates (x, y) or three-dimensional coordinates (x, y, z) and are used to quantitatively describe the structural and pose information of the face.

[0029] In some embodiments, the facial landmark detection model may also be implemented using the MediaPipeFaceMesh model (468 points), the Dlib model (68 points), the OpenFace model, or a custom landmark detection network based on deep learning.

[0030] In step S102, based on the facial key point coordinates, multiple target features of each frame of face image are calculated; the multiple target features include jaw displacement features, eye closure degree features, and mouth opening degree features.

[0031] In some embodiments, the mandibular displacement feature is calculated based on the following steps: Select the coordinates of the chin tip and the nose tip from the facial key points in the current frame to determine the chin-nose tip distance in the current frame; Using the chin-nose tip distance of the first frame in the continuous multi-frame face images as the first reference value, the rate of change of the chin-nose tip distance of the current frame relative to the first reference value is calculated, and the rate of change is used as the chin displacement feature.

[0032] Specifically, from the 106 facial key points output by the PFPLD model, the 16th key point (chin tip) and the 54th key point (nose tip) are selected, and the Euclidean distance between the two points is calculated based on the following formula (1) as the chin-nose tip distance of the current frame.

[0033] ...(1); in, The distance between the chin and the tip of the nose in the current frame, ( () is the location of the 16th key point; () indicates the location of key point number 54.

[0034] When you yawn, your jaw drops, and the distance between your jaw and the tip of your nose increases significantly.

[0035] Based on the following formula (2), the chin-nose tip distance of the first frame in the consecutive multi-frame face images is taken as the first reference value. Calculate the rate of change of the chin-nose tip distance in the current frame relative to the first reference value, and then... As a feature of mandibular displacement: ...(2); in, This is a characteristic of mandibular displacement. The distance between the chin and the tip of the nose in the current frame. This is the first benchmark value.

[0036] In some embodiments, the eye closure degree feature is calculated based on the following steps: From the aforementioned facial key points, select multiple pairs of corresponding points on the upper and lower eyelids of the left eye and multiple pairs of corresponding points on the upper and lower eyelids of the right eye.

[0037] Specifically, select three pairs of corresponding points on the upper and lower eyelids of the left eye (61-67, 62-66, 63-65) and three pairs of corresponding points on the upper and lower eyelids of the right eye (69-75, 70-74, 71-73).

[0038] The average vertical distance between corresponding points on the upper and lower eyelids of the left eye is calculated as the height of the left eye. The distance between the two furthest points in the horizontal direction of the left eye (e.g., point 61 and point 67) is calculated as the width of the left eye. Calculate the aspect ratio of the left eye. = / .

[0039] Calculate the aspect ratio of the right eye using the same method. = / .

[0040] Based on the following formula (3), calculate the average aspect ratio of the left and right eyes, and use a preset normalization coefficient. (In this embodiment, 0.5 is preferred) Normalization is performed to obtain the eye opening and closing degree. : ... (3); Among them, the This refers to the degree of eye closure, i.e., the degree of eye opening and closing; The height ratio of the left eye. The height ratio of the right eye. This is the normalization coefficient.

[0041] The value ranges from 0 to 1. The smaller the value, the higher the degree of eye closure. 0 means the eyes are completely closed, and 1 means they are completely open.

[0042] In some embodiments, the mouth opening degree characteristic is calculated based on the following steps: Select the left inner corner of the mouth, the right inner corner of the mouth, and the midpoint of the lower end of the upper lip from the aforementioned facial key points. Specifically, select key point number 88 (left inner corner of the mouth) P. 88 Key point number 92 (right inner corner of the mouth) P 92 And the 90th key point (midpoint of the lower end of the upper lip) P 90 .

[0043] Using the midpoint of the lower end of the upper lip as the vertex, calculate the first vector between the left inner corner of the mouth and the vertex. and the second vector between the right inner corner of the mouth and the vertex. : ; ; Based on the following formula (4), calculate the angle between the first vector and the second vector, and use the angle as the opening angle of the corners of the mouth: ... (4); in, This is the angle at which the corners of the mouth open, measured in degrees; when the mouth is closed, this angle is close to 180°, and decreases when the mouth is open.

[0044] In other embodiments, the mandibular displacement feature may be replaced by the mouth aspect ratio (MAR) or the distance between the upper and lower lips; the eye closure feature may be replaced by the eye aspect ratio (EAR); and the mouth opening feature may be replaced by the lip curvature or the mouth area.

[0045] In step S103, the yawning result of a single frame is determined based on multiple target features of each frame of face image.

[0046] Please refer to Figure 2 In some embodiments, determining the yawning result of a single frame based on multiple target features of each frame of facial image includes the following steps S201-S204: S201. Based on multiple target features, determine multiple yawning evaluation results for the face image frame; the yawning evaluation results include those that meet the standards and those that do not. S202. Assign a value of 1 to the yawning evaluation result that meets the standard, and assign a value of 0 to the yawning evaluation result that does not meet the standard. S203. Weighted summation of the values ​​corresponding to the various yawning evaluation results to obtain the yawning score for a single frame. S204. Based on the comparison result between the single-frame yawning score and the first preset threshold, determine the single-frame yawning result; the single-frame yawning result includes yawning or not yawning.

[0047] In some embodiments, determining multiple yawning evaluation results for the face image frame based on multiple target features includes: Determine whether the mandibular displacement feature is greater than a second preset threshold, whether the eye closure feature is less than a third preset threshold, and whether the mouth opening feature is less than a fourth preset threshold; When the mandibular displacement feature is greater than the second preset threshold, the initial yawning judgment result corresponding to the mandibular displacement feature is qualified; When the degree of eye closure is less than the third preset threshold, the initial yawning judgment result corresponding to the degree of eye closure is qualified; When the mouth opening degree feature is less than the fourth preset threshold, the initial yawning judgment result corresponding to the mouth opening degree feature is qualified.

[0048] In other words, based on the three types of features extracted, feature binarization is first performed: is_chin_changed (Mandibular movement has reached the target): >chin_nose_change_threshold, is_eyes_closed (eyes closed as required): <eye_closure_threshold is_mouth_open (mouth opening meets the standard): <mouth_angle_threshold。

[0049] Among them, chin_nose_change_threshold is the threshold for mandibular displacement change (second preset threshold); eye_closure_threshold is the threshold for eye closure degree (third preset threshold); and mouth_angle_threshold is the threshold for mouth corner opening angle (fourth preset threshold).

[0050] In other words, it is determined whether the mandibular displacement feature is greater than the second preset threshold chin_nose_change_threshold, whether the eye closure feature is less than the third preset threshold eye_closure_threshold, and whether the mouth opening feature is less than the fourth preset threshold mouth_angle_threshold.

[0051] For example, the second preset threshold chin_nose_change_threshold is preferably 0.15 (i.e. 15%), the third preset threshold eye_closure_threshold is preferably 0.2, and the fourth preset threshold mouth_angle_threshold is preferably 130°.

[0052] When the mandibular displacement feature is greater than the second preset threshold, the initial yawning judgment result corresponding to the mandibular displacement feature is considered satisfactory (i.e., the mandibular displacement is satisfactory), and this is recorded as... =1; otherwise record =0.

[0053] When the eye closure degree feature is less than the third preset threshold, the initial yawning judgment result corresponding to the eye closure degree feature is considered satisfactory (i.e., eye closure is satisfactory), and this is recorded as... =1; otherwise record =0.

[0054] When the mouth opening degree feature is less than the fourth preset threshold, the initial yawning judgment result corresponding to the mouth opening degree feature is considered satisfactory (i.e., the mouth opening degree meets the standard), and is recorded as follows. =1; otherwise record =0.

[0055] Then calculate the score for yawning in a single frame.

[0056] A score of 1 is assigned to a yawn that meets the standard, and a score of 0 is assigned to a yawn that does not meet the standard. The scores of the various yawning evaluation results are then weighted and summed to obtain the yawning score for a single frame. Specifically, as shown in the following formula (5): ... (5); in, , , The weights for the mandibular displacement feature, the eye closure feature, and the mouth opening feature are respectively, and the sum of the three is 1; A yawn for a single frame.

[0057] In some embodiments, the weights are equal, i.e. = = =1 / 3.

[0058] In this embodiment, a weighted summation mechanism is used to fuse the three types of binarized judgment results into a single-frame yawn score, realizing comprehensive decision-making based on multi-source information rather than isolated judgment based on a single feature, thus significantly improving the reliability of single-frame judgment. In other implementations, the weight ratio of each feature can be adjusted according to the actual needs of the application scenario. For example, in a mask-wearing application scenario, the detection of the mouth opening degree feature may not be accurate enough, so the weight ratio can be reduced accordingly. And improve and The specific values ​​for the weight configuration can be preset and adjusted by those skilled in the art based on the actual testing environment and accuracy requirements.

[0059] Based on the comparison between the single-frame yawn score and the first preset threshold, the single-frame yawn result is determined; the single-frame yawn result includes yawning or not yawning.

[0060] Specifically, when a single frame of yawning is scored... If the value is greater than or equal to the single-frame_threshold (i.e., the first preset threshold, such as 0.5), the frame is determined to be a single-frame suspected yawning frame, meaning the single-frame yawning result is a yawn; otherwise, the frame is determined to be a single-frame non-yawning frame, meaning the single-frame yawning result is a non-yawn.

[0061] In step S104, the yawning phase of each frame of face image is determined based on the changing trend of the mandibular displacement features in the consecutive multi-frame face images; the yawning phase includes the mouth opening phase, the peak phase, and the closing phase.

[0062] In some embodiments, please refer to Figure 3 The step of determining the yawning stage of each frame of facial image based on the changing trend of the jaw displacement features in the continuous multi-frame facial images includes the following steps S301-S305: S301. Select a comparison frame that is before the current frame and meets the preset comparison conditions from the continuous multi-frame face images; S302. If the ratio of the mandibular displacement feature of the current frame and the comparison frame is greater than the fifth preset threshold, it is determined that the current frame is in the mouth opening phase. S303. If the ratio of the mandibular displacement feature of the current frame and the comparison frame is less than the sixth preset threshold, it is determined that the current frame is in the closure period. S304. If the rate of change of the mandibular displacement feature of the current frame relative to the preset reference value is greater than the seventh preset threshold, and the mandibular displacement feature of the current frame meets the preset distance stability condition, then the current frame is determined to be in the peak period. S305. If none of the above conditions are met, then the current frame is determined to be in the normal period.

[0063] From the continuous multi-frame face images, a comparison frame that precedes the current frame and meets the preset comparison conditions is selected. Specifically, the chin-nose tip distance of each frame within a preset time window (preferably 0.5 seconds in this embodiment) preceding the current frame is obtained. The first frame within this time window (i.e., the frame that is one second prior to the current frame) is taken as the comparison frame, and its chin-nose tip distance is denoted as... .

[0064] In some embodiments, the comparison frame may be selected as the Nth frame before the current frame (N is a preset positive integer), or a representative frame within a preset time period before the current frame.

[0065] If the ratio of the mandibular displacement feature of the current frame to that of the comparison frame is greater than the fifth preset threshold, the current frame is determined to be in the mouth opening phase.

[0066] Specifically, obtain the jaw-nose tip distance of the current frame. The distance between the jawline and the tip of the nose in the comparison frame .when > When ×M, the current frame is determined to be in the mouth opening period, where M is the fifth preset threshold (i.e., the mouth opening period determination multiple threshold). In this embodiment, M is preferably 1.1 (i.e., when the current distance is more than 10% greater than the distance in the previous 0.5 seconds, it is determined to be in the mouth opening period).

[0067] If the ratio of the mandibular displacement feature of the current frame to that of the comparison frame is less than the sixth preset threshold, the current frame is determined to be in the closure period.

[0068] Specifically, when < When the distance is ×N, the current frame is determined to be in the closing period, where N is the sixth preset threshold (i.e., the closing period determination multiple threshold). In this embodiment, N is preferably 0.9 (i.e., when the current distance is more than 10% smaller than the distance in the previous 0.5 seconds, it is determined to be closing).

[0069] If the rate of change of the mandibular displacement feature of the current frame relative to the preset baseline value is greater than the seventh preset threshold, and the mandibular displacement feature of the current frame meets the preset distance stability condition, then the current frame is determined to be in the peak period.

[0070] Specifically, when The current frame is considered to be in a peak period if the absolute value of the difference between the chin-nose tip distance in the current frame and the chin-nose tip distance in the comparison frame is less than the stability threshold T_stable (e.g., 2 pixels). This condition means that the chin has moved significantly downwards (exceeding the threshold) since the beginning, and the chin position has remained relatively stable (stabilized) within the recent preset time period.

[0071] If none of the above conditions are met, the current frame is determined to be in the normal period.

[0072] It should be noted that the use of the fifth preset threshold M=1.1 and the sixth preset threshold N=0.9 in the above embodiments is only an optional implementation method.

[0073] In some embodiments, the specific value of the above-mentioned multiple threshold can be adjusted according to actual detection needs.

[0074] In some embodiments, the stage determination can also be made using the first derivative (rate of change) or the second derivative (acceleration), and the characteristic trend fitting within the sliding window can also replace the above point-to-point comparison method.

[0075] Please refer to Table 1 below, which shows the yawning stage identification rules described in the embodiments of this application.

[0076] Table 1

[0077] Here, a point-to-point comparison is performed between the current frame and the reference frame, which effectively filters out random jitter noise between frames, making the stage determination more stable. When comparing, the rate of change or multiple threshold is used, which can accurately distinguish the process changes of yawning and identify the three different stages of opening, peak and closing. In particular, the peak period determination condition of "large change and stable distance" fundamentally avoids misjudging the process frame of "opening" or "closing" as the peak period, ensuring the correctness and integrity of the three-stage timing pattern.

[0078] In step S105, the yawning detection result is determined based on the single-frame yawning result and yawning stage of the continuous multi-frame face images.

[0079] In some embodiments, please refer to Figure 4 The determination of the yawn detection result based on the single-frame yawning result and yawning stage of the continuous multi-frame face images includes the following steps S401-S404: S401. Based on the single-frame yawning result of each frame of the continuous multi-frame face images, determine the frame ratio score; the frame ratio score represents the proportion of face image frames that yawn. S402. Calculate the yawning pattern integrity score based on the yawning stage of each frame of the continuous multi-frame face images. S403. Combine the frame ratio score and the pattern integrity score to determine the sequence comprehensive score of the consecutive multi-frame face images; S404. Based on the frame ratio score and the sequence comprehensive score, determine the yawn detection result.

[0080] Here, the yawning results and yawning stages of each frame in the continuous multi-frame face images within the sliding time window are obtained, and a comprehensive analysis is performed on all frame records within the sliding time window (max_time_window).

[0081] Then, based on the single-frame yawning result of each frame of the continuous multi-frame face images, a frame ratio score is determined; the frame ratio score represents the proportion of face image frames that yawn.

[0082] Specifically, the number of frames within the sliding time window where a single frame shows a yawn is counted. Count the total number of frames within the sliding time window. The frame ratio score is calculated based on the following formula (6). : ... (6); Based on the yawning phase of each frame of the continuous multi-frame face images, a yawning pattern integrity score is calculated.

[0083] In some embodiments, a yawning pattern integrity score is calculated based on the yawning phase of each frame in the consecutive multi-frame face images, including: Based on the yawning phase of each frame in the continuous multi-frame face images, the matching result of the mouth opening process of the continuous multi-frame face images and the preset complete yawning pattern is determined; wherein, the preset complete yawning pattern includes, in sequence: mouth opening phase, peak phase and closing phase. Based on the matching results, a pattern integrity score for yawning is determined.

[0084] Here, it is determined whether the opening phase, peak phase, and closing phase occur within the sliding time window. Based on the phase combination, the pattern integrity score (pattern_score) is determined according to the following rules, as shown in Table 2 below: Table 2:

[0085] It should be noted that the values ​​of 1.0, 0.7, 0.4, and 0.0 mentioned above are only one possible implementation method. Depending on the actual situation, other numerical configurations that can reflect the completeness of the stage combination (such as 0.9 / 0.6 / 0.3 / 0.0, etc.) can also be used.

[0086] In other words, .

[0087] Then, the average single-frame yawn score for all frames within the sliding time window is calculated based on the following formula (7). : ... (7); in, Represents the total number of frames The first in frame, Indicates the first Scoring is based on the number of yawns in a single frame.

[0088] By fusing the frame ratio score and the pattern integrity score, a sequence comprehensive score for the consecutive multi-frame face images is determined. Specifically, the pattern integrity score and the average score of a single frame are weighted and fused, and the sequence comprehensive score is obtained based on the following formula (8). : ... (8); in, Score for pattern integrity The fusion weights, for example, the The value is 0.5. The average score per frame is positively correlated with the frame ratio score (both derived from the yawning score per frame), therefore the above weighted fusion is equivalent to fusing the frame ratio score (reflected by the average score per frame) with the pattern integrity score.

[0089] In this embodiment of the application, the yawning detection result is determined based on the frame ratio score and the sequence comprehensive score.

[0090] Specifically, a valid yawning sequence is determined to exist within the current sliding time window when both of the following conditions are met: Sequence composite score ≥Eighth preset threshold (For example, 0.5); Frame Ratio Score ≥ Ninth preset threshold (For example, 0.3).

[0091] In some embodiments, after determining the yawn detection result based on the frame ratio score and the sequence comprehensive score, the yawn detection method further includes: Determine the duration of yawning in the consecutive multi-frame face images; When the duration is greater than or equal to the seventh preset threshold, the yawn detection result is determined to pass the verification.

[0092] Specifically, the start time t_start of the valid yawn sequence is recorded, and the difference between the current time t_current and the start time is calculated as the duration duration = t_current - t_start.

[0093] When the duration is greater than or equal to the seventh preset threshold T_time (preferably 1.5 seconds in this embodiment), the final yawn detection result is determined to be "confirmed yawn," and a yawn confirmation signal is output. When the duration is less than the seventh preset threshold, the yawn is in the "in progress but not yet confirmed" state.

[0094] In some embodiments, the duration of a yawn is determined as follows: when the frame ratio score and the sequence comprehensive score within the sliding time window simultaneously meet their respective threshold conditions for the first time (i.e., the dual-condition judgment is met), this moment is recorded as the start time t_start of the valid yawn sequence; thereafter, the dual-condition judgment results are continuously monitored, and the system continues to keep timing as long as the dual-condition judgment remains met; once either condition in the dual-condition judgment is not met (i.e., the frame ratio score is lower than the threshold or the sequence comprehensive score is lower than the threshold), the yawn sequence is determined to end, and the end time t_end is recorded; the difference between the end time and the start time is taken as the yawn duration duration = t_end – t_start. When this duration reaches a seventh preset threshold (preferably 1.5 seconds), a yawn confirmation signal is output.

[0095] In some embodiments, the method outputs at least one of the following five yawning states, as shown in Table 3.

[0096] Table 3

[0097] In some embodiments, when the STATE_START or STATE_ONGOING state is output, the application layer can trigger a mild warning (such as prompting "Beware of fatigue"); when the STATE_CONFIRMED state is output, the application layer can trigger a severe warning (such as issuing a strong alarm); when the STATE_END state is output, the current yawn detection ends, and the system can accumulate the number of yawns for long-term fatigue assessment.

[0098] In some embodiments, noise immunity processing is also performed based on a noise immunity processing mechanism.

[0099] When the time interval between two adjacent frames exceeds the eighth preset threshold (e.g., 1 second), it indicates that the video stream may have been interrupted or skipped. At this time, it is determined as a sequence interruption, the sequence state is automatically reset, all historical frame records are cleared, and the chin-nose tip distance of the current frame is reset to the first reference value, thereby avoiding incorrect association across long time intervals.

[0100] Only frames within the sliding time window (max_time_window, for example, 3 seconds) are retained, and expired data is automatically cleared to maintain the timeliness of the judgment. This ensures that the system only analyzes recent valid data and avoids outdated data interfering with the current judgment.

[0101] When judging whether yawning has ended, the proportion R_recent of single frames of suspected yawning frames within the nearest preset time window recent_time_window (for example, 0.5 seconds) going back from the current moment is used for judgment, instead of only relying on the judgment result of the current single frame. Specifically, when R_recent<T_recent (preferably 0.1 in this embodiment), it is determined that the yawning behavior has ended. This mechanism enhances the tolerance to single-frame noise, and avoids misjudging the end due to accidental detection failure of a certain frame.

[0102] Based on the same inventive concept, an embodiment of the present application further provides a yawning detection apparatus corresponding to the yawning detection method. Since the principle of the apparatus in the embodiment of the present application for solving the problem is similar to that of the aforementioned yawning detection method in the embodiment of the present application, the implementation of the apparatus may refer to the implementation of the method, and repeated parts will not be described again.

[0103] Please refer to Figure 5 , Figure 5 which shows a schematic structural diagram of the yawning detection apparatus according to the embodiment of the present application; the yawning detection apparatus includes: an obtaining module 501, configured to obtain multiple consecutive frames of face images, and extract coordinates of facial key points in each frame of face image; a calculation module 502, configured to calculate multiple target features of each frame of face image based on the coordinates of the facial key points; the multiple target features include a jaw displacement feature, an eye closure degree feature, and a mouth opening degree feature; a first determining module 503, configured to determine a single-frame yawning result based on the multiple target features of each frame of face image; a second determining module 504, configured to determine the yawning phase of each frame of face image according to the change trend of the jaw displacement feature in the multiple consecutive frames of face images; the yawning phase includes a mouth opening period, a peak period, and a closing period; a third determining module 505, configured to determine a yawning detection result based on the single-frame yawning result and the yawning phase of the multiple consecutive frames of face images.

[0104] In some embodiments, in the yawning detection apparatus, the first determining module, when determining the single-frame yawning result based on the multiple target features of each frame of face image, is specifically configured to: determine multiple yawning evaluation results for the current frame of face image respectively based on the multiple target features; the yawning evaluation results include reaching the standard and not reaching the standard; assign a value of 1 to a yawning evaluation result that reaches the standard, and assign a value of 0 to a yawning evaluation result that does not reach the standard; perform weighted summation on the assigned values corresponding to the multiple yawning evaluation results to obtain a single-frame yawning score; Based on the comparison between the single-frame yawn score and the first preset threshold, the single-frame yawn result is determined; the single-frame yawn result includes yawning or not yawning.

[0105] In some embodiments, the yawning detection device, when determining multiple yawning evaluation results for a face image frame based on multiple target features, is specifically used for: Determine whether the mandibular displacement feature is greater than a second preset threshold, whether the eye closure feature is less than a third preset threshold, and whether the mouth opening feature is less than a fourth preset threshold; When the mandibular displacement feature is greater than the second preset threshold, the initial yawning judgment result corresponding to the mandibular displacement feature is qualified; When the degree of eye closure is less than the third preset threshold, the initial yawning judgment result corresponding to the degree of eye closure is qualified; When the mouth opening degree feature is less than the fourth preset threshold, the initial yawning judgment result corresponding to the mouth opening degree feature is qualified.

[0106] In some embodiments, the yawning detection device uses a mandibular displacement feature calculated based on the following steps: Select the coordinates of the chin tip and the nose tip from the facial key points in the current frame to determine the chin-nose tip distance in the current frame; Using the chin-nose tip distance of the first frame in the continuous multi-frame face images as the first reference value, the rate of change of the chin-nose tip distance of the current frame relative to the first reference value is calculated, and the rate of change is used as the chin displacement feature.

[0107] In some embodiments, in the yawning detection device, the second determining module, when determining the yawning stage of each frame of face image based on the changing trend of the jaw displacement features in the consecutive multiple frames of face images, is specifically used for: Select a comparison frame that is preceding the current frame and meets the preset comparison conditions from the continuous multi-frame face images; If the ratio of the mandibular displacement feature of the current frame to that of the comparison frame is greater than the fifth preset threshold, the current frame is determined to be in the mouth opening phase. If the ratio of the mandibular displacement feature of the current frame to that of the comparison frame is less than the sixth preset threshold, the current frame is determined to be in the closure period. If the rate of change of the mandibular displacement feature of the current frame relative to the preset benchmark value is greater than the seventh preset threshold, and the mandibular displacement feature of the current frame meets the preset distance stability condition, then the current frame is determined to be in the peak period. If none of the above conditions are met, the current frame is determined to be in the normal period.

[0108] In some embodiments, in the yawning detection device, the third determining module, when determining the yawning detection result based on the single-frame yawning result and the yawning stage of the continuous multi-frame face images, is specifically used for: Based on the yawning result of each frame of the continuous multi-frame face images, a frame ratio score is determined; the frame ratio score represents the proportion of face image frames that yawn. Based on the yawning phase of each frame of the consecutive multi-frame face images, calculate the yawning pattern integrity score. The frame ratio score and pattern integrity score are combined to determine the sequence comprehensive score of the consecutive multi-frame face images; The yawn detection result is determined based on the frame ratio score and the sequence comprehensive score.

[0109] In some embodiments, the yawn detection device further includes The verification module is used to determine the duration of yawning in the consecutive multi-frame face images after determining the yawning detection result based on the frame ratio score and the sequence comprehensive score. When the duration is greater than or equal to the seventh preset threshold, the yawn detection result is determined to pass the verification.

[0110] In some embodiments, in the yawning detection device, the third determining module, when calculating the yawning pattern integrity score based on the yawning stage of each frame in the continuous multi-frame face images, is specifically used for: Based on the yawning phase of each frame in the continuous multi-frame face images, the matching result of the mouth opening process of the continuous multi-frame face images and the preset complete yawning pattern is determined; wherein, the preset complete yawning pattern includes, in sequence: mouth opening phase, peak phase and closing phase. Based on the matching results, a pattern integrity score for yawning is determined.

[0111] Based on the same inventive concept, this application also provides an electronic device corresponding to the yawn detection method. Since the principle of the electronic device in this application is similar to the yawn detection method described above, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be described again.

[0112] Please refer to Figure 6 , Figure 6A schematic diagram of the structure of the electronic device according to an embodiment of this application is shown; the electronic device 600 includes: a processor 601, a memory 602 and a bus. The memory 602 stores machine-readable instructions that can be executed by the processor 601. When the electronic device 600 is running, the processor 601 and the memory 602 communicate through the bus. When the machine-readable instructions are executed by the processor 601, the steps of the yawn detection method are performed.

[0113] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0116] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0118] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for detecting yawning, characterized in that: The method includes: Acquire multiple consecutive frames of face images and extract the coordinates of facial key points in each frame; Based on the coordinates of the facial key points, multiple target features are calculated for each frame of the face image; the multiple target features include jaw displacement features, eye closure degree features, and mouth opening degree features; Based on multiple target features of each frame of the face image, the result of yawning in a single frame is determined; Based on the changing trend of the mandibular displacement features in the continuous multi-frame face images, the yawning stage of each face image is determined; the yawning stage includes the mouth opening stage, the peak stage, and the closing stage; Based on the single-frame yawning result and yawning stage of the continuous multi-frame face images, the yawning detection result is determined.

2. The yawning detection method according to claim 1, characterized in that, The determination of a single-frame yawning result based on multiple target features of each frame of facial image includes: Based on multiple target features, various yawning evaluation results are determined for the face image frame; the yawning evaluation results include qualified and unqualified. A value of 1 is assigned to the yawning evaluation result that meets the standard, and a value of 0 is assigned to the yawning evaluation result that does not meet the standard. The weighted sum of the values ​​corresponding to the various yawning evaluation results is used to obtain the yawning score for a single frame. Based on the comparison between the single-frame yawn score and the first preset threshold, the single-frame yawn result is determined; the single-frame yawn result includes yawning or not yawning.

3. The yawning detection method according to claim 2, characterized in that, The method of determining multiple yawning evaluation results for a facial image frame based on multiple target features includes: Determine whether the mandibular displacement feature is greater than a second preset threshold, whether the eye closure feature is less than a third preset threshold, and whether the mouth opening feature is less than a fourth preset threshold; When the mandibular displacement feature is greater than the second preset threshold, the initial yawning judgment result corresponding to the mandibular displacement feature is qualified; When the degree of eye closure is less than the third preset threshold, the initial yawning judgment result corresponding to the degree of eye closure is qualified; When the mouth opening degree feature is less than the fourth preset threshold, the initial yawning judgment result corresponding to the mouth opening degree feature is qualified.

4. The yawning detection method according to claim 1 or 2, characterized in that, The mandibular displacement feature is Calculated based on the following steps: Select the coordinates of the chin tip and the nose tip from the facial key points in the current frame to determine the chin-nose tip distance in the current frame; Using the chin-nose tip distance of the first frame in the continuous multi-frame face images as the first reference value, the rate of change of the chin-nose tip distance of the current frame relative to the first reference value is calculated, and the rate of change is used as the chin displacement feature.

5. The yawning detection method according to claim 1, characterized in that, The step of determining the yawning stage of each frame of facial images based on the changing trend of the mandibular displacement features in the consecutive multiple frames of facial images includes: Select a comparison frame that is preceding the current frame and meets the preset comparison conditions from the continuous multi-frame face images; If the ratio of the mandibular displacement feature of the current frame to that of the comparison frame is greater than the fifth preset threshold, the current frame is determined to be in the mouth opening phase. If the ratio of the mandibular displacement feature of the current frame to that of the comparison frame is less than the sixth preset threshold, the current frame is determined to be in the closure period. If the rate of change of the mandibular displacement feature of the current frame relative to the preset benchmark value is greater than the seventh preset threshold, and the mandibular displacement feature of the current frame meets the preset distance stability condition, then the current frame is determined to be in the peak period. If none of the above conditions are met, the current frame is determined to be in the normal period.

6. The yawning detection method according to claim 1, characterized in that, The determination of yawn detection results based on the single-frame yawning result and yawning stage of the continuous multi-frame face images includes: Based on the yawning result of each frame of the continuous multi-frame face images, a frame ratio score is determined; the frame ratio score represents the proportion of face image frames that yawn. Based on the yawning phase of each frame of the consecutive multi-frame face images, calculate the yawning pattern integrity score. The frame ratio score and pattern integrity score are combined to determine the sequence comprehensive score of the consecutive multi-frame face images; The yawn detection result is determined based on the frame ratio score and the sequence comprehensive score.

7. The yawn detection method according to claim 6, characterized in that, After determining the yawn detection result based on the frame ratio score and the sequence comprehensive score, the method further includes: Determine the duration of yawning in the consecutive multi-frame face images; When the duration is greater than or equal to the seventh preset threshold, the yawn detection result is determined to pass the verification.

8. The yawning detection method according to claim 6, characterized in that, Based on the yawning phase of each frame in the continuous multi-frame face images, a yawning pattern integrity score is calculated, including: Based on the yawning phase of each frame in the continuous multi-frame face images, the matching result of the mouth opening process of the continuous multi-frame face images and the preset complete yawning pattern is determined; wherein, the preset complete yawning pattern includes, in sequence: mouth opening phase, peak phase and closing phase. Based on the matching results, a pattern integrity score for yawning is determined.

9. A yawn detection device, characterized in that, The device includes: The acquisition module is used to acquire multiple consecutive frames of face images and extract the coordinates of facial key points in each frame of face image; The calculation module is used to calculate multiple target features of each frame of face image based on the coordinates of the facial key points; the multiple target features include jaw displacement features, eye closure features, and mouth opening features; The first determining module is used to determine the yawning result of a single frame based on multiple target features of each frame of face image; The second determining module is used to determine the yawning stage of each frame of face image based on the changing trend of the mandibular displacement features in the continuous multi-frame face images; the yawning stage includes the mouth opening stage, the peak stage, and the closing stage; The third determining module is used to determine the yawning detection result based on the single-frame yawning result and yawning stage of the continuous multi-frame face images.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the yawn detection method as described in any one of claims 1 to 8 are performed.

Images