Sleep detection method, device, computer-readable storage medium, and terminal device

Abstract

This application belongs to the field of sleep detection technology, and more particularly relates to a sleep detection method, apparatus, computer-readable storage medium, and terminal device. The method comprises: obtaining a facial image to be detected; performing facial key point detection in the facial image using a preset facial key point detection model to obtain a facial key point set; determining the eye open / close state of a person based on the facial key point set; and determining the sleep state based on the eye open / close state. Through this application, the sleep state can be determined by analyzing the facial image. For the user, the entire detection process is seamless and does not cause physical discomfort, effectively ensuring the user's normal sleep.

Description

Technical Field

[0001] The present application belongs to the field of sleep detection technology, and in particular relates to a sleep detection method, apparatus, computer-readable storage medium, and terminal device. Background Art

[0002] Traditional sleep detection methods typically use smart wearable devices such as smartwatches and smart bracelets to collect various physiological parameters of the user and then calculate and analyze these parameters to determine whether the user is asleep. Although this method can obtain relatively accurate detection results, it requires the user to wear the smart wearable device at all times, which can cause physical discomfort to some users, including infants, and affect their sleep. Summary of the Invention

[0003] In view of this, the embodiments of the present application provide a sleep detection method, apparatus, computer-readable storage medium, and terminal device to solve the problem that existing sleep detection methods may cause physical discomfort to some users, including infants, and affect their normal sleep.

[0004] A first aspect of an embodiment of the present application provides a sleep detection method, which may include:

[0005] Get the face image to be detected;

[0006] Using a preset facial key point detection model to detect facial key points in the facial image to obtain a facial key point set;

[0007] Determining the eye opening and closing status of a person based on the set of facial key points;

[0008] The sleeping state is determined according to the eye opening and closing state of the person.

[0009] In a specific implementation of the first aspect, performing facial key point detection in the facial image using a preset facial key point detection model to obtain a facial key point set may include:

[0010] Performing a first facial key point detection on the facial image using the facial key point detection model to obtain an initial facial key point set;

[0011] Determining a face rotation angle according to the initial face key point set;

[0012] Performing rotation transformation on the face image according to the face rotation angle to obtain a face transformation image;

[0013] The facial key point detection model is used to perform a second facial key point detection in the face transformation image to obtain the facial key point set.

[0014] In a specific implementation of the first aspect, determining the face rotation angle according to the initial face key point set may include:

[0015] Extracting the coordinates of the left eye key point and the right eye key point from the initial face key point set;

[0016] The face rotation angle is calculated according to the coordinates of the left eye key point and the coordinates of the right eye key point.

[0017] In a specific implementation of the first aspect, calculating the face rotation angle according to the coordinates of the left eye key point and the right eye key point may include:

[0018] Determine the quadrant where the face is located according to the coordinates of the left eye key point and the coordinates of the right eye key point;

[0019] Calculate a reference rotation angle according to the coordinates of the left eye key point and the coordinates of the right eye key point;

[0020] determining a quadrant rotation angle corresponding to the quadrant in which the face is located;

[0021] The face rotation angle is calculated according to the reference rotation angle and the quadrant rotation angle.

[0022] In a specific implementation of the first aspect, determining the quadrant in which the face is located based on the coordinates of the left eye key point and the coordinates of the right eye key point may include:

[0023] If the horizontal coordinate of the right eye key point is greater than the horizontal coordinate of the left eye key point, and the vertical coordinate of the right eye key point is greater than or equal to the vertical coordinate of the left eye key point, then the quadrant where the face is located is determined to be the first face quadrant;

[0024] If the horizontal coordinate of the right eye key point is less than or equal to the horizontal coordinate of the left eye key point, and the vertical coordinate of the right eye key point is greater than the vertical coordinate of the left eye key point, then the quadrant where the face is located is determined to be the second face quadrant;

[0025] If the horizontal coordinate of the right eye key point is smaller than the horizontal coordinate of the left eye key point, and the vertical coordinate of the right eye key point is smaller than or equal to the vertical coordinate of the left eye key point, then the quadrant where the face is located is determined to be the third face quadrant;

[0026] If the horizontal coordinate of the right eye key point is greater than or equal to the horizontal coordinate of the left eye key point, and the vertical coordinate of the right eye key point is less than the vertical coordinate of the left eye key point, the quadrant where the face is located is determined to be the fourth face quadrant.

[0027] In a specific implementation of the first aspect, determining the eye open / closed state of a person based on the set of facial key points may include:

[0028] Extract the coordinates of the left eye key point, the right eye key point, the upper eye key point and the lower eye key point from the face key point set;

[0029] Calculate the eye opening angle according to the coordinates of the left key point of the eye, the coordinates of the right key point of the eye, the coordinates of the upper key point of the eye, and the coordinates of the lower key point of the eye;

[0030] The eye opening / closing state of the person is determined according to the eye opening angle.

[0031] In a specific implementation of the first aspect, calculating the eye opening angle according to the coordinates of the left eye key point, the coordinates of the right eye key point, the coordinates of the upper eye key point, and the coordinates of the lower eye key point may include:

[0032] Calculate the left eye corner opening angle according to the coordinates of the left key point of the eye, the coordinates of the upper key point of the eye and the coordinates of the lower key point of the eye;

[0033] Calculate the right eye corner opening angle according to the coordinates of the right eye key point, the coordinates of the upper eye key point and the coordinates of the lower eye key point;

[0034] The eye opening angle is calculated according to the left eye corner opening angle and the right eye corner opening angle.

[0035] A second aspect of an embodiment of the present application provides a sleep detection device, which may include:

[0036] A face image acquisition module is used to acquire the face image to be detected;

[0037] A facial key point detection module, configured to detect facial key points in the facial image using a preset facial key point detection model to obtain a facial key point set;

[0038] A module for determining the eye opening and closing state of a person, configured to determine the eye opening and closing state of a person according to the set of key points of the face;

[0039] The sleep state determination module is used to determine the sleep state according to the eye opening and closing state of the person.

[0040] In a specific implementation of the second aspect, the facial key point detection module may include:

[0041] A first facial key point detection submodule, configured to perform a first facial key point detection in the facial image using the facial key point detection model to obtain an initial facial key point set;

[0042] A face rotation angle determination submodule, configured to determine the face rotation angle based on the initial face key point set;

[0043] A rotation transformation submodule, configured to perform rotation transformation on the face image according to the face rotation angle to obtain a face transformation image;

[0044] The second facial key point detection submodule is configured to perform a second facial key point detection in the face transformation image using the facial key point detection model to obtain the facial key point set.

[0045] In a specific implementation of the second aspect, the face rotation angle determination submodule may include:

[0046] A coordinate extraction unit, configured to extract the coordinates of the left eye key point and the right eye key point from the initial set of facial key points;

[0047] A face rotation angle calculation unit is used to calculate the face rotation angle according to the coordinates of the left eye key point and the coordinates of the right eye key point.

[0048] In a specific implementation of the second aspect, the face rotation angle calculation unit may include:

[0049] a quadrant determination subunit, configured to determine the quadrant in which the face is located based on the coordinates of the left eye key point and the coordinates of the right eye key point;

[0050] a reference rotation angle calculation subunit, configured to calculate a reference rotation angle according to the coordinates of the left eye key point and the coordinates of the right eye key point;

[0051] a quadrant rotation angle determination subunit, configured to determine a quadrant rotation angle corresponding to the quadrant where the face is located;

[0052] The face rotation angle calculation subunit is configured to calculate the face rotation angle according to the reference rotation angle and the quadrant rotation angle.

[0053] In a specific implementation of the second aspect, the quadrant determination subunit can be specifically used to: if the horizontal coordinate of the right eye key point is greater than the horizontal coordinate of the left eye key point, and the vertical coordinate of the right eye key point is greater than or equal to the vertical coordinate of the left eye key point, then determine that the quadrant where the face is located is the first facial quadrant; if the horizontal coordinate of the right eye key point is less than or equal to the horizontal coordinate of the left eye key point, and the vertical coordinate of the right eye key point is greater than the vertical coordinate of the left eye key point, then determine that the quadrant where the face is located is the second facial quadrant; if the horizontal coordinate of the right eye key point is less than the horizontal coordinate of the left eye key point, and the vertical coordinate of the right eye key point is less than or equal to the vertical coordinate of the left eye key point, then determine that the quadrant where the face is located is the third facial quadrant; if the horizontal coordinate of the right eye key point is greater than or equal to the horizontal coordinate of the left eye key point, and the vertical coordinate of the right eye key point is less than the vertical coordinate of the left eye key point, then determine that the quadrant where the face is located is the fourth facial quadrant.

[0054] In a specific implementation of the second aspect, the human eye open / closed state determination module may include:

[0055] A coordinate extraction submodule, configured to extract the coordinates of the left eye key point, the right eye key point, the upper eye key point, and the lower eye key point from the face key point set;

[0056] an eye opening angle calculation submodule, configured to calculate the eye opening angle according to the coordinates of the left eye key point, the coordinates of the right eye key point, the coordinates of the upper eye key point, and the coordinates of the lower eye key point;

[0057] The human eye opening / closing state determination submodule is configured to determine the human eye opening / closing state according to the eye opening angle.

[0058] In a specific implementation of the second aspect, the eye opening angle calculation submodule may include:

[0059] a left eye corner opening angle calculation unit, configured to calculate the left eye corner opening angle according to the coordinates of the left key point of the eye, the coordinates of the upper key point of the eye, and the coordinates of the lower key point of the eye;

[0060] a right eye corner opening angle calculation unit, configured to calculate the right eye corner opening angle according to the coordinates of the right key point of the eye, the coordinates of the upper key point of the eye, and the coordinates of the lower key point of the eye;

[0061] An eye opening angle calculation unit is used to calculate the eye opening angle according to the left eye corner opening angle and the right eye corner opening angle.

[0062] A third aspect of the embodiments of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above sleep detection methods are implemented.

[0063] A fourth aspect of an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above sleep detection methods when executing the computer program.

[0064] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the steps of any one of the above sleep detection methods.

[0065] Compared to the prior art, the embodiments of the present application have the following advantages: the embodiments of the present application obtain a facial image to be detected; use a preset facial key point detection model to detect facial key points in the facial image to obtain a set of facial key points; determine the eye open / closed state of a person based on the set of facial key points; and determine the sleep state based on the eye open / closed state of a person. Through the embodiments of the present application, the sleep state can be determined by analyzing the facial image. For the user, the entire detection process is seamless and does not cause physical discomfort, effectively ensuring the user's normal sleep. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0067] Figure 1 This is a schematic diagram of a typical sleep detection scenario in an embodiment of the present application;

[0068] Figure 2 This is a flow chart of an embodiment of a sleep detection method in an embodiment of the present application;

[0069] Figure 3 This is a schematic diagram of the network structure of a facial key point detection model in an embodiment of the present application;

[0070] Figure 4 Schematic diagram of each facial key point;

[0071] Figure 5 This is a schematic diagram of the facial posture of a baby while sleeping;

[0072] Figure 6 A schematic flow chart for detecting facial key points in a face image using a preset facial key point detection model to obtain a set of facial key points;

[0073] Figure 7 Schematic diagram of some facial key points;

[0074] Figure 8 A schematic diagram of the quadrants of a human face;

[0075] Figure 9 Schematic diagram of the effect of rotating a face image;

[0076] Figure 10 This is a structural diagram of an embodiment of a sleep detection device in an embodiment of the present application;

[0077] Figure 11 This is a schematic block diagram of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0079] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0080] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0081] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0082] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0083] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0084] Figure 1 The following is a schematic diagram of a typical sleep detection scenario in an embodiment of the present application. As shown, a sleep detection terminal device can be placed on a bracket or wall next to the bed. The terminal device uses its camera to capture images of the bed area and determines the user's sleep state through a series of visual image analyses. For the user, the entire sleep detection process is seamless and does not cause physical discomfort, effectively ensuring the user's normal sleep.

[0085] See also Figure 2 In an embodiment of the present application, a sleep detection method may include:

[0086] Step S201: Acquire a face image to be detected.

[0087] In an embodiment of the present application, the angle of the camera can be adjusted in advance so that it can face the user on the bed as directly as possible, so as to capture a relatively complete facial image.

[0088] Step S202: Use a preset facial key point detection model to perform facial key point detection in the face image to obtain a facial key point set.

[0089] In the embodiment of the present application, any facial key point detection model in the prior art can be selected to perform facial key point detection according to actual conditions.

[0090] Considering that the existing face key point detection model has weak feature extraction capabilities and poor robustness in the face of complex conditions, such as light disturbances and poor image quality, which can cause key point deviations. In a specific implementation of the embodiment of the present application, a method is further proposed. Figure 3 The new facial key point detection model shown in the figure adopts the lightweight network structure shown in the following table:

[0091]

[0092]

[0093] Among them, Input is the input size of each layer, Operator is the data processing process of each layer, Conv is a 3*3 convolution layer, bottleneck is the bottleneck layer, GMP is the global maximum pooling layer, and Linear is the linear layer. The face image passes through Conv, stage1, stage2 and Conv in sequence, and feature data is extracted layer by layer, and then processed by GMP. Finally, it passes through two linear layers for two full-connection processing processes, which are respectively recorded as fea_fc and res_fc. Stage 1 (stage1) includes a 3*3 convolution layer and a bottleneck layer, and stage 2 (stage2) includes a 3*3 convolution layer and two bottleneck layers. t is the transpose magnification factor of the bottleneck layer, c is the number of convolution kernel channels, n is the number of times the Operator is repeated, and s is the stride. The output of the face key point detection model is 98 face key points. Since each key point consists of a horizontal coordinate and a vertical coordinate, it is 196-dimensional data. Figure 4 Shown is a schematic diagram of each facial key point.

[0094] Considering that the loss functions such as L1_smooth and L2 commonly used in the prior art are not good at capturing subtle differences and have low positioning accuracy, in a specific implementation of the embodiment of the present application, the following loss function can be used:

[0095]

[0096] Where x is the error between the actual output and the expected output (i.e., groundtruth), w is a preset positive number used to limit the range of the nonlinear region to the interval [-w, w], ∈ is the preset curvature of the constrained nonlinear region, C is a preset constant, and wingloss is the loss function. The response of this loss function to slight errors is much higher than that of the L2 function, which can achieve more refined positioning of key points.

[0097] In the prior art, in order to locate the position of the human eyes more accurately, the first consideration is to not learn other key points, but only learn the eye key points to reduce the learning of invalid information. However, this design will destroy some implicit structures of the human face. The eye key points may appear in areas such as eyebrows and nostrils. This targeted learning cannot achieve good results because other key points are also supervising the spatial information of the eye key points, which has a good auxiliary effect on the positioning of the eye key points. In order to retain the auxiliary information of other key points and highlight the key learning of the eyes, in a specific implementation method of the embodiment of the present application, the eye key points can be learned by highlighting the local weights. After this strategy, it is easier to capture the subtle changes in the eyes.

[0098] like Figure 5 As shown in the figure, in real-world baby sleep scenes, babies are generally hyperactive, and their faces are often tilted in the image. Furthermore, parents may position the device arbitrarily, resulting in high uncertainty in facial pose and the potential for keypoint detection errors.

[0099] In the face of uncertainty, the embodiment of the present application can try to find a positive perspective to standardize the face, that is, estimate the angle of the face, and then rotate and transform it according to the angle to obtain the front face, which can greatly reduce the complexity of key point prediction. In a specific implementation of the embodiment of the present application, step S202 can specifically include the following: Figure 6 The process shown:

[0100] Step S2021: Use the facial key point detection model to perform the first facial key point detection in the face image to obtain an initial facial key point set.

[0101] Step S2022: Determine the face rotation angle based on the initial face key point set.

[0102] Figure 7 The following is a schematic diagram of some of the facial key points in the initial facial key point set, where pt1 is the left eye key point, corresponding to Figure 4 The key point 96 in pt2 is the key point of the right eye, corresponding to Figure 4 The key point 97 in pt3 is the nose key point, corresponding to Figure 4 The key point 54 in pt4 is the key point of the left corner of the mouth, corresponding to Figure 4 The key point 76 in pt5 is the key point of the right corner of the mouth, corresponding to Figure 4 Key point 82.

[0103] In a specific implementation method of an embodiment of the present application, the coordinates of the left eye key point (x1, y1) and the coordinates of the right eye key point (x2, y2) can be extracted from the initial face key point set, and the face rotation angle can be calculated based on the coordinates of the left eye key point and the coordinates of the right eye key point.

[0104] Specifically, the quadrant where the face is located can be determined based on the coordinates of the left eye key point and the right eye key point. Figure 8As shown, if the abscissa of the right eye key point is greater than that of the left eye key point, and the ordinate of the right eye key point is greater than or equal to that of the left eye key point, that is, x2>x1&&y2>=y1, then it is determined that the quadrant where the face is located is the first face quadrant; if the abscissa of the right eye key point is less than or equal to that of the left eye key point, and the ordinate of the right eye key point is greater than that of the left eye key point, that is, x2<=x1&&y2>y1, then it is determined that the quadrant where the face is located is the second face quadrant; if the abscissa of the right eye key point is less than that of the left eye key point, and the ordinate of the right eye key point is less than or equal to that of the left eye key point, that is, x2<x1&&y2<=y1, then it is determined that the quadrant where the face is located is the third face quadrant; if the abscissa of the right eye key point is greater than or equal to that of the left eye key point, and the ordinate of the right eye key point is less than that of the left eye key point, that is, x2>=x1&&y1<y2, then it is determined that the quadrant where the face is located is the fourth face quadrant.

[0105] Then, the reference rotation angle can be calculated based on the coordinates of the left eye key point and the right eye key point. The specific calculation formula is as follows:

[0106]

[0107] Among them, θ is the reference rotation angle.

[0108] Next, the quadrant rotation angle corresponding to the quadrant where the face is located can be determined. If the quadrant where the face is located is the first face quadrant, the quadrant rotation angle is 0 degrees; if the quadrant where the face is located is the second face quadrant, the quadrant rotation angle is 90 degrees; if the quadrant where the face is located is the third face quadrant, the quadrant rotation angle is 180 degrees; if the quadrant where the face is located is the fourth face quadrant, the quadrant rotation angle is 270 degrees.

[0109] After obtaining the reference rotation angle and the quadrant rotation angle, the face rotation angle can be calculated based on the two. That is, the sum of the reference rotation angle and the quadrant rotation angle is determined as the face rotation angle.

[0110] It should be noted that the above process is the method of calculating the face rotation angle based on the left eye key point and the right eye key point. In another specific implementation manner of the embodiments of the present application, the face rotation angle can also be calculated based on the left mouth corner key point and the right mouth corner key point, and only the left eye key point and the right eye key point in the above calculation process need to be replaced with the left mouth corner key point and the right mouth corner key point. The embodiments of the present application will not elaborate on this further. Further, a face rotation angle can be calculated based on the left eye key point and the right eye key point, and a face rotation angle can be calculated based on the left mouth corner key point and the right mouth corner key point, and the average value of the two is determined as the final face rotation angle. The angle obtained by this method is more accurate.

[0111] Step S2023: performing rotation transformation on the face image according to the face rotation angle to obtain a face transformation image.

[0112] Image rotation transformation is relatively common in the prior art and will not be described in detail in the embodiments of the present application. Figure 8 The following is a schematic diagram of the effect of rotation transformation.

[0113] Step S2024: Use the facial key point detection model to perform a second facial key point detection in the transformed face image to obtain a facial key point set.

[0114] like Figure 8 As shown in the lower left figure, this tilted face will face the situation of inaccurate face frame detection, so environmental information will be introduced after the rotation transformation, which will interfere with the key point detection. Therefore, in the embodiment of the present application, a second face key point detection can be performed in the face transformation image to filter the background information, thereby obtaining a more accurate face key point set.

[0115] Step S203: determining the eye opening or closing state of the person based on the set of facial key points.

[0116] In a specific implementation of the embodiment of the present application, the left eye key point (i.e. Figure 4 The coordinates of the key point 60 in the image, the coordinates of the right key point of the eye (i.e. Figure 4 Key points 64 in the eye (i.e. Figure 4 The coordinates of the key point 62 in the image and the key point under the eye (i.e. Figure 4 The coordinates of the key point 66) in.

[0117] Then, the eye opening angle can be calculated based on the coordinates of the left key point of the eye, the coordinates of the right key point of the eye, the coordinates of the upper key point of the eye, and the coordinates of the lower key point of the eye.

[0118] Specifically, the left eye corner opening angle can be calculated based on the coordinates of the left eye key point, the coordinates of the upper eye key point, and the coordinates of the lower eye key point. The calculation formula is as follows:

[0119]

[0120] Among them, θ1 is the opening angle of the left eye corner, d1 is the distance between the coordinates of the left key point of the eye and the coordinates of the key point on the eye, d2 is the distance between the coordinates of the left key point of the eye and the coordinates of the key point under the eye, and d3 is the distance between the coordinates of the key point on the eye and the coordinates of the key point under the eye.

[0121] The right eye corner opening angle can be calculated based on the coordinates of the right eye key point, the coordinates of the upper eye key point, and the coordinates of the lower eye key point. The calculation formula is as follows:

[0122]

[0123] Among them, θ2 is the opening angle of the right eye corner, d4 is the distance between the coordinates of the right key point of the eye and the coordinates of the key point above the eye, and d5 is the distance between the coordinates of the right key point of the eye and the coordinates of the key point below the eye.

[0124] After obtaining the left eye corner opening angle and the right eye corner opening angle, the eye opening angle can be calculated based on the two. The calculation formula is as follows:

[0125]

[0126] Among them, θ open This is the eye opening angle.

[0127] After obtaining the eye opening angle, the eye opening state can be determined based on the eye opening angle. Specifically, if the eye opening angle is less than a preset angle threshold, the eye opening state can be determined to be closed. Conversely, if the eye opening angle is greater than or equal to the angle threshold, the eye opening state can be determined to be open. The specific value of the angle threshold can be set according to actual conditions. For example, it can be set to 20 degrees or other values, and this embodiment of the application does not specifically limit this.

[0128] It should be noted that the above process determines the open and closed state of the left eye. The process of determining the open and closed state of the right eye is similar. It is only necessary to replace the key point 60 in the above calculation process with the key point 68, the key point 62 with the key point 70, the key point 64 with the key point 72, and the key point 66 with the key point 74. This embodiment of the present application will not be repeated here.

[0129] Step S204: determining the sleeping state according to the eye opening and closing state of the person.

[0130] Since only one eye may be visible when the baby is lying on his side, as long as one of the left eye or the right eye is closed, it can be determined that the baby is in a sleeping state; otherwise, it can be determined that the baby is not in a sleeping state.

[0131] In summary, the embodiments of the present application acquire a facial image to be detected; use a preset facial key point detection model to detect facial key points in the facial image to obtain a set of facial key points; determine the eye open / closed state of a person based on the set of facial key points; and determine the sleep state based on the eye open / closed state of the person. Through the embodiments of the present application, the sleep state can be determined by analyzing the facial image. For the user, the entire detection process is seamless and does not cause any physical discomfort, effectively ensuring the user's normal sleep.

[0132] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0133] Corresponding to the sleep detection method described in the above embodiment, Figure 10 A structural diagram of an embodiment of a sleep detection device provided in an embodiment of the present application is shown.

[0134] In this embodiment, a sleep detection device may include:

[0135] The face image acquisition module 1001 is used to acquire the face image to be detected;

[0136] A facial key point detection module 1002 is configured to detect facial key points in the facial image using a preset facial key point detection model to obtain a facial key point set;

[0137] A human eye open / closed state determination module 1003 is configured to determine the human eye open / closed state based on the set of facial key points;

[0138] The sleep state determination module 1004 is configured to determine the sleep state according to the eye opening and closing state of the person.

[0139] In a specific implementation of the embodiment of the present application, the facial key point detection module may include:

[0140] A first facial key point detection submodule, configured to perform a first facial key point detection in the facial image using the facial key point detection model to obtain an initial facial key point set;

[0141] A face rotation angle determination submodule, configured to determine the face rotation angle based on the initial face key point set;

[0142] A rotation transformation submodule, configured to perform rotation transformation on the face image according to the face rotation angle to obtain a face transformation image;

[0143] The second facial key point detection submodule is configured to perform a second facial key point detection in the face transformation image using the facial key point detection model to obtain the facial key point set.

[0144] In a specific implementation of the embodiment of the present application, the face rotation angle determination submodule may include:

[0145] A coordinate extraction unit, configured to extract the coordinates of the left eye key point and the right eye key point from the initial set of facial key points;

[0146] A face rotation angle calculation unit is used to calculate the face rotation angle according to the coordinates of the left eye key point and the coordinates of the right eye key point.

[0147] In a specific implementation of the embodiment of the present application, the face rotation angle calculation unit may include:

[0148] a quadrant determination subunit, configured to determine the quadrant in which the face is located based on the coordinates of the left eye key point and the coordinates of the right eye key point;

[0149] a reference rotation angle calculation subunit, configured to calculate a reference rotation angle according to the coordinates of the left eye key point and the coordinates of the right eye key point;

[0150] a quadrant rotation angle determination subunit, configured to determine a quadrant rotation angle corresponding to the quadrant where the face is located;

[0151] The face rotation angle calculation subunit is configured to calculate the face rotation angle according to the reference rotation angle and the quadrant rotation angle.

[0152] In a specific implementation of an embodiment of the present application, the quadrant determination subunit can be specifically used to: if the horizontal coordinate of the right eye key point is greater than the horizontal coordinate of the left eye key point, and the vertical coordinate of the right eye key point is greater than or equal to the vertical coordinate of the left eye key point, then determine that the quadrant where the face is located is the first facial quadrant; if the horizontal coordinate of the right eye key point is less than or equal to the horizontal coordinate of the left eye key point, and the vertical coordinate of the right eye key point is greater than the vertical coordinate of the left eye key point, then determine that the quadrant where the face is located is the second facial quadrant; if the horizontal coordinate of the right eye key point is less than the horizontal coordinate of the left eye key point, and the vertical coordinate of the right eye key point is less than or equal to the vertical coordinate of the left eye key point, then determine that the quadrant where the face is located is the third facial quadrant; if the horizontal coordinate of the right eye key point is greater than or equal to the horizontal coordinate of the left eye key point, and the vertical coordinate of the right eye key point is less than the vertical coordinate of the left eye key point, then determine that the quadrant where the face is located is the fourth facial quadrant.

[0153] In a specific implementation of the embodiment of the present application, the human eye open / closed state determination module may include:

[0154] A coordinate extraction submodule, configured to extract the coordinates of the left eye key point, the right eye key point, the upper eye key point, and the lower eye key point from the face key point set;

[0155] an eye opening angle calculation submodule, configured to calculate the eye opening angle according to the coordinates of the left eye key point, the coordinates of the right eye key point, the coordinates of the upper eye key point, and the coordinates of the lower eye key point;

[0156] The human eye opening / closing state determination submodule is configured to determine the human eye opening / closing state according to the eye opening angle.

[0157] In a specific implementation of the embodiment of the present application, the eye opening angle calculation submodule may include:

[0158] a left eye corner opening angle calculation unit, configured to calculate the left eye corner opening angle according to the coordinates of the left key point of the eye, the coordinates of the upper key point of the eye, and the coordinates of the lower key point of the eye;

[0159] a right eye corner opening angle calculation unit, configured to calculate the right eye corner opening angle according to the coordinates of the right key point of the eye, the coordinates of the upper key point of the eye, and the coordinates of the lower key point of the eye;

[0160] An eye opening angle calculation unit is used to calculate the eye opening angle according to the left eye corner opening angle and the right eye corner opening angle.

[0161] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0162] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0163] Figure 11 A schematic block diagram of a terminal device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0164] like Figure 11 As shown, the terminal device 11 of this embodiment includes: a processor 110, a memory 111, and a computer program 112 stored in the memory 111 and executable on the processor 110. When the processor 110 executes the computer program 112, the steps in the above-mentioned sleep detection method embodiments are implemented, for example Figure 2 Alternatively, when the processor 110 executes the computer program 112, the functions of the modules / units in the above-mentioned device embodiments are realized, for example, Figure 10 The functions of modules 1001 to 1004 are shown.

[0165] Exemplarily, the computer program 112 may be divided into one or more modules / units, which are stored in the memory 111 and executed by the processor 110 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 112 in the terminal device 11.

[0166] The terminal device 11 can be a computing device such as a mobile phone, a tablet computer, a desktop computer, a notebook, a PDA, a robot, etc. It will be understood by those skilled in the art that Figure 11 It is only an example of the terminal device 11 and does not constitute a limitation of the terminal device 11. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 11 may also include input and output devices, network access devices, buses, etc.

[0167] The processor 110 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0168] The memory 111 may be an internal storage unit of the terminal device 11, such as a hard disk or memory of the terminal device 11. The memory 111 may also be an external storage device of the terminal device 11, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 11. Furthermore, the memory 111 may include both an internal storage unit of the terminal device 11 and an external storage device. The memory 111 is used to store the computer program and other programs and data required by the terminal device 11. The memory 111 may also be used to temporarily store data that has been output or is about to be output.

[0169] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0170] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0171] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0172] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0173] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0174] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0175] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0176] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A sleep detection method, characterized in that: include: Get the face image to be detected; Using a preset facial key point detection model to detect facial key points in the facial image to obtain a facial key point set; Extracting the coordinates of the left eye key point, the right eye key point, the upper eye key point, and the lower eye key point of the same eye from the set of facial key points; calculating the left eye corner opening angle according to the coordinates of the left eye key point, the upper eye key point, and the lower eye key point; calculating the right eye corner opening angle according to the coordinates of the right eye key point, the upper eye key point, and the lower eye key point; calculating the eye opening angle according to the left eye corner opening angle and the right eye corner opening angle; and determining the eye opening state according to the eye opening angle; The sleeping state is determined according to the eye opening and closing state of the person.

2. The sleep detection method according to claim 1, wherein: The method of using a preset facial key point detection model to detect facial key points in the facial image to obtain a facial key point set includes: Performing a first facial key point detection on the facial image using the facial key point detection model to obtain an initial facial key point set; Determining a face rotation angle according to the initial face key point set; Performing rotation transformation on the face image according to the face rotation angle to obtain a face transformation image; The facial key point detection model is used to perform a second facial key point detection in the face transformation image to obtain the facial key point set.

3. The sleep detection method according to claim 2, wherein: The determining of the face rotation angle according to the initial face key point set includes: Extracting the coordinates of the left eye key point and the right eye key point from the initial face key point set; The face rotation angle is calculated according to the coordinates of the left eye key point and the coordinates of the right eye key point.

4. The sleep detection method according to claim 3, wherein: The calculating the face rotation angle according to the coordinates of the left eye key point and the right eye key point includes: If the abscissa of the right eye key point is greater than the abscissa of the left eye key point, and the ordinate of the right eye key point is greater than or equal to the ordinate of the left eye key point, then the quadrant rotation angle is determined to be zero degrees, and the reference rotation angle is the angle between the straight line between the left eye key point and the right eye key point and the straight line on which the abscissa axis lies; If the abscissa of the right eye key point is less than or equal to the abscissa of the left eye key point, and the ordinate of the right eye key point is greater than the ordinate of the left eye key point, then the quadrant rotation angle is determined to be ninety degrees, and the reference rotation angle is the angle between the straight line between the left eye key point and the right eye key point and the straight line on which the ordinate axis lies; If the abscissa of the right eye key point is smaller than the abscissa of the left eye key point, and the ordinate of the right eye key point is smaller than or equal to the ordinate of the left eye key point, then the quadrant rotation angle is determined to be 180 degrees, and the reference rotation angle is the angle between the straight line between the left eye key point and the right eye key point and the straight line on which the abscissa axis lies; If the abscissa of the right eye key point is greater than or equal to the abscissa of the left eye key point, and the ordinate of the right eye key point is less than the ordinate of the left eye key point, then the quadrant rotation angle is determined to be 270 degrees, and the reference rotation angle is the angle between the straight line between the left eye key point and the right eye key point and the straight line on which the ordinate axis lies; The sum of the reference rotation angle and the quadrant rotation angle is determined as the face rotation angle.

5. A sleep detection device, characterized in that: include: A face image acquisition module is used to acquire the face image to be detected; A facial key point detection module, configured to detect facial key points in the facial image using a preset facial key point detection model to obtain a facial key point set; The module for determining the eye opening and closing state is configured to extract the coordinates of the left eye key point, the right eye key point, the upper eye key point, and the lower eye key point of the same eye from the set of facial key points; calculate the left eye corner opening angle based on the coordinates of the left eye key point, the upper eye key point, and the lower eye key point; calculate the right eye corner opening angle based on the coordinates of the right eye key point, the upper eye key point, and the lower eye key point; calculate the eye opening angle based on the left eye corner opening angle and the right eye corner opening angle; and determine the eye opening and closing state based on the eye opening angle. The sleep state determination module is used to determine the sleep state according to the eye opening and closing state of the person.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the sleep detection method according to any one of claims 1 to 4 are implemented.

7. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the sleep detection method according to any one of claims 1 to 4 are implemented.

Images