Sleep detection warning method and device

Abstract

This application discloses a sleep detection and early warning method and device. The method comprises: obtaining a target monitoring image for monitoring a first subject's sleep; detecting target region information and target key point information in the target monitoring image, wherein the target region information includes at least facial region information, limb region information, and quilt region information, and the target key point information includes human body key point information; determining a target sleep pattern for the first subject based on the target region information and target key point information; processing the target region information and target key point information based on a target processing method corresponding to the target sleep pattern to obtain the first subject's sleep state, and generating early warning information corresponding to the sleep state. This application addresses the technical problem in related technologies of poor performance in infant and toddler sleep detection solutions and difficulty in providing timely and effective early warnings.

Description

Technical Field

[0001] The present application relates to the field of safety monitoring technology, and more specifically, to a sleep detection and early warning method and device. Background Art

[0002] While infants and young children are sleeping, guardians may not be able to constantly monitor their condition, so automatic safety monitoring is a technology that has received considerable attention. Related technologies primarily employ the following three monitoring methods for automatic infant safety monitoring: 1) Subjective video monitoring, which suffers from the fact that guardians may not be able to constantly monitor the camera and receive timely warning information; 2) Template-matching-based intelligent camera video monitoring, which requires manual simulation to generate sample features of infant sleeping positions and professional personnel to establish templates for infant sleeping positions. This method is technically difficult and requires significant human and financial investment; 3) Audio analysis of infants and young children, which triggers an alarm when the infant cries. However, this method cannot detect the crying condition beforehand; the alarm only sounds after the event has occurred.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a sleep detection and early warning method and device to at least solve the technical problem in the related art that solutions for infant sleep detection are ineffective and difficult to provide effective early warnings in a timely manner.

[0005] According to one aspect of an embodiment of the present application, a sleep detection and warning method is provided, including: obtaining a target monitoring image for monitoring the sleep of a first subject; detecting target area information and target key point information in the target monitoring image, wherein the target area information includes at least: face area information, limb area information and quilt area information, and the target key point information includes: human body key point information; determining a target sleep mode of the first subject based on the target area information and the target key point information; processing the target area information and the target key point information based on a target processing method corresponding to the target sleep mode to obtain a sleep state of the first subject, and generating warning information corresponding to the sleep state.

[0006] Optionally, a target monitoring video for monitoring the sleep of the first subject is obtained; redundant frame images and blurred frame images in all frame images of the target monitoring video are deleted to obtain a target monitoring image.

[0007] Optionally, based on the face detection algorithm in deep learning, the face area information in the target monitoring image is detected, and the face area information includes at least: the number of faces, the type of faces, and the face area position; based on the general target detection algorithm in deep learning, the limb area information and quilt area information in the target monitoring image are detected, wherein the limb area information includes at least: the limb area position, and the quilt area information includes at least: the quilt covering area position; based on the key point detection algorithm in deep learning, the human body key point information in the target monitoring image is detected, and the human body key point information includes at least: the number of human body key points and the human body key point positions.

[0008] Optionally, when the number of faces does not exceed a first preset threshold or the face category does not include the face of the second object, the target sleep mode is determined to be the first sleep mode; when the number of faces exceeds the first preset threshold and the face category includes the face of the second object, the target sleep mode is determined to be the second sleep mode.

[0009] Optionally, the first sleep mode includes: a first sub-sleep mode and a second sub-sleep mode, the second sleep mode includes: a third sub-sleep mode and a fourth sub-sleep mode, and the target sleep mode of the first object is determined based on the target area information and the target key point information, including: in the first sleep mode, if the number of human body key points does not exceed the second preset threshold, the target sleep mode is determined to be the first sub-sleep mode; if the number of human body key points exceeds the second preset threshold, the target sleep mode is determined to be the second sub-sleep mode; in the second sleep mode, if the number of human body key points does not exceed the third preset threshold, the target sleep mode is determined to be the third sub-sleep mode; if the number of human body key points exceeds the third preset threshold, the target sleep mode is determined to be the fourth sub-sleep mode.

[0010] Optionally, when the target sleep mode is the first sub-sleep mode, the target area information and the target key point information are processed based on the target processing method corresponding to the target sleep mode to obtain the sleep state of the first object, and generate warning information corresponding to the sleep state, including: determining a first distance between the center position of the face area of ​​the first object and the center position of the quilt area, and the overlap between the limb area position of the first object and the quilt-covered area position; when the first distance is less than a fourth preset threshold, determining that the first object is in a first sleep state, and generating a first warning message, wherein the first sleep state indicates that the first object covers the quilt too high, and the first warning message is used to prompt that the quilt needs to be pulled down; when the overlap is less than a fifth preset threshold, determining that the first object is in a second sleep state, and generating a second warning message, wherein the second sleep state indicates that the first object covers the quilt too low, and the second warning message is used to prompt that the quilt needs to be pulled up.

[0011] Optionally, when the target sleep mode is the second sub-sleep mode, the target area information and the target key point information are processed based on the target processing method corresponding to the target sleep mode to obtain the sleep state of the first object, and generate warning information corresponding to the sleep state, including: determining a first number of head key points and a second number of limb key points in the human body key points of the first object, determining a first angle between the first key point vector and the second key point vector, and determining a second distance between the two-part key points, wherein the first key point vector is a vector from the nose key point to the neck key point, and the second key point vector is a normal vector of the vector of the two-part key points, and the two-part key points include at least one of the following: the two-part key points, the two-part key points on both sides of the waist, and the two-part key points. points, key points of both arms, and key points of both knees; when the first number exceeds the sixth preset threshold and the second number exceeds the seventh preset threshold, it is determined that the first object is in a third sleep state, wherein the third sleep state indicates that the first object is lying on its back; when the first number is less than the eighth preset threshold and the second number exceeds the seventh preset threshold, it is determined that the first object is in a fourth sleep state, and a third warning message is generated, wherein the fourth sleep state indicates that the first object is lying on its stomach, and the third warning message is used to prompt that the sleeping position of the first object needs to be adjusted; when the first angle is less than the ninth preset threshold and the second distance is less than the tenth preset threshold, it is determined that the first object is in a fifth sleep state, and a third warning message is generated, wherein the fifth sleep state indicates that the first object is lying on its side.

[0012] Optionally, when the target sleep mode is the third sub-sleep mode, the target area information and the target key point information are processed based on the target processing method corresponding to the target sleep mode to obtain the sleep state of the first object, and generate warning information corresponding to the sleep state, including: determining a third distance between the center position of the limb area of ​​the second object and the center position of the limb area of ​​the first object; when the third distance is less than the eleventh preset threshold, determining that the first object is in the sixth sleep state, and generating fourth warning information, wherein the sixth sleep state indicates that squeezing occurs between the second object and the first object, and the fourth warning information is used to prompt the second object to stay away from the first object.

[0013] Optionally, when the target sleep mode is the fourth sub-sleep mode, the target area information and the target key point information are processed based on the target processing method corresponding to the target sleep mode to obtain the sleep state of the first object, and generate warning information corresponding to the sleep state, including: determining whether there is an overlapping area between the human body key point position of the second object and the limb area position of the first object; when there is an overlapping area, determining that the first object is in the sixth sleep state, and generating fourth warning information, wherein the sixth sleep state indicates that squeezing occurs between the second object and the first object, and the fourth warning information is used to prompt the second object to stay away from the first object.

[0014] Optionally, based on the classification detection algorithm in deep learning, it is detected whether there is an occlusion area at the facial area of ​​the first object; when there is an occlusion area, a fifth warning information is generated, wherein the fifth warning information is used to prompt that there is occlusion on the face of the first object and needs to be checked and processed.

[0015] According to another aspect of an embodiment of the present application, a sleep detection and warning device is also provided, including: an acquisition module for acquiring a target monitoring image for monitoring the sleep of a first object; a detection module for detecting target area information and target key point information in the target monitoring image, wherein the target area information includes at least: face area information, limb area information and quilt area information, and the target key point information includes: human body key point information; a determination module for determining a target sleep mode of the first object based on the target area information and the target key point information; a processing module for processing the target area information and the target key point information based on a target processing method corresponding to the target sleep mode, obtaining the sleep state of the first object, and generating warning information corresponding to the sleep state.

[0016] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided. The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above-mentioned sleep detection warning method.

[0017] According to another aspect of an embodiment of the present application, a processor is further provided, and the processor is used to run a program, wherein the above-mentioned sleep detection and warning method is executed when the program is running.

[0018] In an embodiment of the present application, after obtaining a target monitoring image for monitoring the sleep of a first object, a depth target detection and human limb key point detection method can be used to detect the target area information and target key point information in the target monitoring image, and the target sleep pattern of the first object is determined based on the above target area information and target key point information, so as to achieve the purpose of intelligent identification of infant sleep safety, and the target area information and target key point information are processed by the processing method corresponding to the target sleep pattern, and finally the corresponding warning information is generated, thereby achieving the technical effect of fully automatic and timely warning of infant sleep, and thus solving the technical problem in the related technology that the solutions for infant sleep detection are not effective and it is difficult to provide effective warnings in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1is a flowchart of a sleep detection and warning method according to an embodiment of the present application;

[0021] Figure 2 This is a schematic diagram of a process for preprocessing a target surveillance video according to an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a process for detecting a target monitoring image according to an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a process for determining a sleep mode of a first object according to an embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of a process for generating corresponding warning information based on the sleep mode of a first object according to an embodiment of the present application;

[0025] Figure 6 2 is a structural diagram of a sleep detection and warning device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Example 1

[0029] According to an embodiment of the present application, a sleep detection and warning method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] Figure 1 FIG. 1 is a flow chart of an optional sleep detection and warning method according to an embodiment of the present application, such as Figure 1 As shown, the method includes at least steps S102-S108, wherein:

[0031] Step S102: Acquire a target monitoring image for monitoring the sleep of the first subject.

[0032] Typically, a data acquisition device such as a webcam can be used to collect a target monitoring video that monitors the sleep of the first object, and the obtained target monitoring video is preprocessed, such as deleting redundant frame images and blurred frame images in all frame images of the target monitoring video to obtain a target monitoring image, wherein the first object can be an infant or a patient who lacks self-care ability. The following content is explained using infants as an example.

[0033] Figure 2 This preprocessing operation performs on the target surveillance video. Specifically, it uses random sampling at intervals to remove redundant frames. It then uses the Fourier transform (FFT) method to detect blur. If the result is blurry, the video frame is discarded; if the result is clear, the data is processed for deep learning. This preprocessing operation is designed to extract key frames from all frames of the target surveillance video to obtain higher-quality video frame data.

[0034] Step S104 , detecting target region information and target key point information in the target monitoring image, wherein the target region information at least includes: face region information, limb region information and quilt region information, and the target key point information includes: human body key point information.

[0035] Specifically, the acquired target surveillance video is subjected to target area detection and target key point detection respectively through deep learning target detection and human key point detection, such as Figure 3 The method uses the face detection algorithm, real-time target detection algorithm, general target detection algorithm, classification detection algorithm and key point detection algorithm of deep learning to detect all key point information of the human body and quilt area information in the target monitoring image. This can be achieved through the following specific process:

[0036] The face detection algorithm in deep learning is used to detect the face area information in the target surveillance image. Specifically, the number, type and face area position of the faces appearing in the target surveillance video within a certain period of time can be obtained. For example, the face position of an infant can be expressed as FU[Face X ,Face y ,Face w ,Face_h].

[0037] The limb region information and quilt region information in the target monitoring image are detected based on the general target detection algorithm in deep learning, wherein the limb region information at least includes: the limb region position, and the quilt region information at least includes: the quilt covering region position. For example, the entire limb region position of an infant can be expressed as LU[limb X ,limb y ,limb w ,limb h ], the position of the sleeping partner's limb area can be expressed as PU [Plimb X ,Plimb y ,Plimb w ,Plimb h ], the location of the quilt can be expressed as BU[b X ,b y ,b w ,b h ], where the position labels [x, y, w, h] represent the center coordinates of the target detection frame and the width and height of the detection frame respectively.

[0038] Based on the classification detection algorithm in deep learning, it is detected whether there is an occlusion area in the facial area of ​​the first object. When there is an occlusion area, a fifth warning information is generated, wherein the fifth warning information is used to prompt that there is occlusion on the face of the first object and needs to be checked and processed. It uses a neural network classification model to identify whether the face of an infant or child is occluded.

[0039] Using a deep learning-based keypoint detection algorithm, human keypoint information is detected in the target surveillance image. This keypoint information includes at least the number and location of keypoints. For example, keypoint detection involves the coordinates of keypoints on each of an infant's limbs. The keypoints include {"nose": 0, "neck": 1, "right shoulder": 2, "right elbow": 3, "right wrist": 4, "left shoulder": 5, "left elbow": 6, "left wrist": 7, "right waist": 8, "right knee": 9, "right ankle": 10, "left waist": 11, "left knee": 12, "left ankle": 13, "right eye": 14, "left eye": 15, "right ear": 16, "left ear": 17, "background": 18}, where the numbers represent the keypoint location retrieval sequence.

[0040] Step S106 : determining the target sleep mode of the first subject based on the target area information and the target key point information.

[0041] Usually, infants and young children can sleep alone or with adults such as their parents. Therefore, their target sleep mode can be determined first in the following way: when the number of faces does not exceed the first preset threshold or the face type does not include the face of the second object, the target sleep mode is determined to be the first sleep mode, also known as the infant and young child sleeping alone mode; when the number of faces exceeds the first preset threshold and the face type includes the face of the second object, the target sleep mode is determined to be the second sleep mode, also known as the adult sleeping together mode.

[0042] Among them, the second person can be an adult guardian such as father, mother, grandfather, grandmother, etc.

[0043] Furthermore, the first sleep mode can be divided into: a first sub-sleep mode and a second sub-sleep mode. In the first sleep mode, if the number of key points on the human body does not exceed the second preset threshold, the target sleep mode is determined to be the first sub-sleep mode, also known as the infant and toddler sleeping alone with a quilt mode; if the number of key points on the human body exceeds the second preset threshold, the target sleep mode is determined to be the second sub-sleep mode, also known as the infant and toddler sleeping alone without a quilt mode.

[0044] Similarly, the second sleep mode can be divided into: a third sub-sleep mode and a fourth sub-sleep mode. In the second sleep mode, if the number of key points on the human body does not exceed the third preset threshold, the target sleep mode is determined to be the third sub-sleep mode, also known as the adult sleeping with a quilt mode; if the number of key points on the human body exceeds the third preset threshold, the target sleep mode is determined to be the fourth sub-sleep mode, also known as the adult sleeping with a quilt mode.

[0045] For example, Figure 4 The target area information and target key point information are analyzed to determine the infant sleep pattern in the target surveillance video. During the infant sleep detection process, the target surveillance video is detected to obtain the current number and type of faces. When the number of faces is less than 2 (the first preset threshold), it does not include adult faces, and the key point detection is for multiple people, it is determined to be A mode (infant sleeping alone mode). When the A mode is activated, the number of key points of the human limbs is counted. When the number of key points of the human body is greater than 8 (the second preset threshold), it is determined to be A-NCQ mode (infant sleeping alone without a quilt mode); when the number of key points of the human body is less than or equal to 8 (the second preset threshold), it is determined to be A-CQ mode (infant sleeping alone with a quilt).

[0046] When the number of faces is greater than or equal to 2 (the first preset threshold), including adult faces, and the key point detection is for multiple people, it is determined to be P mode (adult companion sleeping mode). When the P mode is activated, the number of key points of the human limbs is counted. When the number of key points of the human body is greater than 8 (the third preset threshold), it is determined to be P-NCQ mode (adult companion sleeping without covering the quilt mode); when the number of key points of the human body is less than or equal to 8 (the third preset threshold), it is determined to be P-CQ mode (adult companion sleeping with covering the quilt mode).

[0047] Step S108 : Processing the target area information and the target key point information based on the target processing method corresponding to the target sleep mode to obtain the sleep state of the first subject, and generating warning information corresponding to the sleep state.

[0048] Figure 5 The target area information and target key point information are processed based on the target processing method corresponding to the target sleep mode to obtain the sleep state of the first object and generate a complete flow chart of warning information corresponding to the sleep state. The specific processing methods for different target sleep modes and the generation of corresponding warning information can be achieved by the following methods:

[0049] During infant sleep detection, when the target sleep mode is A-CQ mode, the distance between the center of the infant's face and the center of the quilt, as well as the degree of overlap between the quilt-covered area and the infant's limb area, can be determined. This can be used to determine whether the infant's quilt is too low or too high, whether the infant is kicking off the quilt, and generate corresponding warning information.

[0050] Specifically, when the target sleep mode is the first sub-sleep mode, the first distance between the center position of the face area of ​​the first object and the center position of the quilt area, as well as the overlap between the limb area position of the first object and the quilt-covered area position are determined; when the first distance is less than the fourth preset threshold, it is determined that the first object is in a first sleep state, and a first warning message is generated, wherein the first sleep state indicates that the first object covers the quilt too high, and the first warning message is used to prompt that the quilt needs to be pulled down; when the overlap is less than the fifth preset threshold, it is determined that the first object is in a second sleep state, and a second warning message is generated, wherein the second sleep state indicates that the first object covers the quilt too low, and the second warning message is used to prompt that the quilt needs to be pulled up.

[0051] The specific analysis method is as follows:

[0052] The first step is to determine the coordinates of the center of the infant's face F[Face X ,Face y ] and the coordinates of the center of the area where the quilt is located B[b X ,b y ];

[0053] The second step is to calculate the Euclidean distance between the coordinates of point F and point B. (first distance);

[0054] Step 3: If Face y ≤b y (Fourth preset threshold), take the warning threshold As the critical point threshold, when Dist1∈(α1, α2), it means that the overlap between the quilt-covered area and the infant's limb area is high, indicating that the quilt is too high for the infant. At this time, a first warning message is generated, which prompts that the quilt needs to be pulled down.

[0055] The fourth step is to use the infant limb area position LU[limb X ,limb y ,limb w ,limb_h] and the location of the quilt area BU[b X ,b y ,b w ,b h ] The intersection is divided by the limb area LU to get the limb intersection rate Ω, which is expressed as It can be set that when Ω≤0.3 (the fifth preset threshold), it indicates that the quilt covering the infant is too low, and a second warning message is generated at this time, and the second warning message prompts that the quilt needs to be pulled up.

[0056] During infant sleep detection, when the target sleep mode is the A-NCQ mode, the sleeping posture of the first subject may be analyzed based on key point information.

[0057] Specifically, when the target sleep mode is the second sub-sleep mode, determining a first number of head key points and a second number of limb key points among the human body key points of the first subject, determining a first angle between the first key point vector and the second key point vector, and determining a second distance between the two body key points, wherein the first key point vector is a vector from the nose key point to the neck key point, and the second key point vector is a normal vector of a vector from the two shoulder key points;

[0058] When the first number exceeds the sixth preset threshold and the second number exceeds the seventh preset threshold, it is determined that the first object is in a third sleep state, wherein the third sleep state indicates that the first object is lying on its back; when the first number is less than the eighth preset threshold and the second number exceeds the seventh preset threshold, it is determined that the first object is in a fourth sleep state, and a third warning message is generated, wherein the fourth sleep state indicates that the first object is lying on its stomach, and the third warning message is used to prompt that the sleeping position of the first object needs to be adjusted; when the first angle is less than the ninth preset threshold and the second distance is less than the tenth preset threshold, it is determined that the first object is in a fifth sleep state, and a third warning message is generated, wherein the fifth sleep state indicates that the first object is lying on its side.

[0059] The specific analysis process is as follows:

[0060] The sleeping position of the infant is determined based on the above information, including: when the number of head key points (first number) among the infant's limb key points is greater than or equal to a preset threshold (sixth preset threshold), and the number of limb key points (second number) is greater than or equal to 10 (seventh preset threshold), the infant's sleeping position is determined to be supine (third sleeping state); when the number of head key points (first number) among the infant's limb key points is less than or equal to 2 (eighth preset threshold) and the number of limb key points (second number) is greater than or equal to 10 (seventh preset threshold), the infant's sleeping position is determined to be prone (fourth sleeping state); by calculating the angle (first angle) between the vector from the infant's nose key point to the neck key point (first key point vector) and the horizontal vector of the limb detection midline, the horizontal vector of the limb detection midline is defined as the normal vector of the shoulder vector (second key point vector). When the angle threshold is less than the set threshold (the ninth preset threshold), and the relative Euclidean distance (the second distance) of the key points of two parts (shoulders, waist, arms, and knees) is less than the predetermined threshold (the tenth preset threshold), it is determined that the infant's sleeping position is side sleeping (the fifth sleeping state).

[0061] When the infant's sleeping position is prone or sideways, a third warning message is generated, which prompts that the infant's sleeping position needs to be adjusted.

[0062] During infant sleep detection, when the target sleep mode is P-CQ mode, the distance between the center of the infant detection frame and the center of the sleeping partner detection frame can be judged. If the distance is less than the set threshold, a squeeze warning is generated.

[0063] Specifically, when the target sleep mode is the third sub-sleep mode, the third distance between the center position of the limb area of ​​the second object and the center position of the limb area of ​​the first object is determined; when the third distance is less than the eleventh preset threshold, it is determined that the first object is in the sixth sleep state, and a fourth warning message is generated, wherein the sixth sleep state indicates that squeezing occurs between the second object and the first object, and the fourth warning message is used to prompt the second object to stay away from the first object.

[0064] The specific analysis process is as follows:

[0065] The first step is to obtain the position coordinates P[Plimb of the sleeping partner based on the real-time target detection algorithm. X ,Plimb y ] and the center coordinates of the area where the quilt infant is located C[limb X ,limb y ];

[0066] The second step is to calculate the Euclidean distance between the coordinate point P of the sleeping partner and the coordinate point C of the center of the area where the infant is located. (third distance);

[0067] The third step is when When , it means that there is a safe distance between the sleeping partner and the infant; when (the eleventh preset threshold), it indicates that there is squeezing between the sleeping partner and the infant (the sixth sleeping state), and a fourth warning message is generated, which prompts the sleeping partner to stay away from the infant.

[0068] During infant sleep detection, when the target sleep mode is P-NCQ mode, it is possible to detect whether the key points of the sleeping partner are within the infant detection frame.

[0069] Specifically, when the target sleep mode is the fourth sub-sleep mode, it is determined whether there is an overlapping area between the human body key point position of the second object and the limb area position of the first object; when there is an overlapping area, it is determined that the first object is in the sixth sleep state, and a fourth warning message is generated, wherein the sixth sleep state indicates that squeezing occurs between the second object and the first object, and the fourth warning message is used to prompt the second object to stay away from the first object.

[0070] When it is determined that the positions of the key points of the sleeping partner's body intersect with the skeleton lines connecting the key points of the infant's limbs, it is determined that there is strong squeezing between the infant and the sleeping partner. At this time, the fourth warning information is generated. The fourth warning information is used to remind the sleeping partner to stay away from the infant.

[0071] It should be noted that the various preset thresholds in the embodiments of the present application can be adjusted according to the actual scenario, and all the above values ​​are only for illustrative purposes and do not constitute specific limitations.

[0072] Specifically, the above-mentioned warning information can be prompted by outputting a warning signal through the warning device, for example, through a buzzer, a text message prompt, a vibration of an electronic bracelet, a ringing of a mobile phone, etc.

[0073] The embodiment of the present application uses a neural network algorithm to analyze infant monitoring videos collected by network cameras, and can detect the position of the infant's face in the monitoring video stream, whether the infant's face is blocked by foreign objects, whether the body is covered with a quilt, and the specific information of the quilt's location. By analyzing the key points of the infant's limbs, it can be determined whether the infant's sleeping posture is correct, and the overlap rate of the key points of the sleeping partner's limbs and the key points of the infant can be analyzed to analyze whether the sleeping partner has squeezed the infant. This overcomes the defects of the existing technology in ensuring safe sleep for infants and young children, and adds an early warning system for identifying when the sleeping partner squeezes the infant when sleeping with the child.

[0074] Example 2

[0075] According to an embodiment of the present application, a sleep detection and warning device for implementing the above sleep detection and warning method is also provided. Figure 6 As shown, the device at least includes an acquisition module 61, a detection module 62, a determination module 63 and a processing module 64, wherein:

[0076] The acquisition module 61 is configured to acquire a target monitoring image for monitoring the sleep of a first subject.

[0077] Optionally, the acquisition module may acquire a target monitoring video of the first subject sleeping, and delete redundant frame images and blurred frame images from all frame images of the target monitoring video to obtain the target monitoring image.

[0078] The detection module 62 is used to detect target area information and target key point information in the target monitoring image, wherein the target area information at least includes: face area information, limb area information and quilt area information, and the target key point information includes: human body key point information.

[0079] Optionally, the detection module can detect face area information in the target monitoring image based on the face detection algorithm in deep learning, and the face area information includes at least: the number of faces, the type of faces, and the position of the face area; detect limb area information and quilt area information in the target monitoring image based on the general target detection algorithm in deep learning, wherein the limb area information includes at least: the position of the limb area, and the quilt area information includes at least: the position of the quilt covering area; detect human key point information in the target monitoring image based on the key point detection algorithm in deep learning, and the human key point information includes at least: the number of human key points and the position of human key points

[0080] The determination module 63 is configured to determine the target sleep mode of the first subject based on the target area information and the target key point information.

[0081] Optionally, the determination module may determine the target sleep mode as the first sleep mode when the number of faces does not exceed a first preset threshold or the face category does not include a face of the second object; and determine the target sleep mode as the second sleep mode when the number of faces exceeds the first preset threshold and the face category includes a face of the second object. The first sleep mode includes a first sub-sleep mode and a second sub-sleep mode, and the second sleep mode includes a third sub-sleep mode and a fourth sub-sleep mode.

[0082] Specifically, in the first sleep mode, if the number of key points on the human body does not exceed the second preset threshold, the target sleep mode is determined to be the first sub-sleep mode; if the number of key points on the human body exceeds the second preset threshold, the target sleep mode is determined to be the second sub-sleep mode; in the second sleep mode, if the number of key points on the human body does not exceed the third preset threshold, the target sleep mode is determined to be the third sub-sleep mode; if the number of key points on the human body exceeds the third preset threshold, the target sleep mode is determined to be the fourth sub-sleep mode.

[0083] When the target sleep mode is the first sub-sleep mode, determine a first distance between the center position of the face area of ​​the first object and the center position of the quilt area, as well as a degree of overlap between the limb area position of the first object and the quilt-covered area position; when the first distance is less than a fourth preset threshold, determine that the first object is in a first sleep state, and generate a first warning message, wherein the first sleep state indicates that the first object covers the quilt too high, and the first warning message is used to prompt that the quilt needs to be pulled down; when the degree of overlap is less than a fifth preset threshold, determine that the first object is in a second sleep state, and generate a second warning message, wherein the second sleep state indicates that the first object covers the quilt too low, and the second warning message is used to prompt that the quilt needs to be pulled up.

[0084] When the target sleep mode is the second sub-sleep mode, a first number of head key points and a second number of limb key points among the human body key points of the first subject are determined, a first angle between the first key point vector and the second key point vector is determined, and a second distance between the two body key points is determined, wherein the first key point vector is a vector from the nose key point to the neck key point, and the second key point vector is a normal vector of a vector of the two shoulder key points, and the two body key points include at least one of the following: two shoulder key points, key points on both sides of the waist, two arm key points, and two knee key points. When the first number exceeds a sixth preset threshold and the second number exceeds a seventh preset threshold, the first subject is determined to be in a third sleep state, wherein the third sleep state indicates that the first subject is lying on his back. When the first number is less than an eighth preset threshold and the second number exceeds the seventh preset threshold, the first subject is determined to be in a fourth sleep state, and a third warning message is generated, wherein the fourth sleep state indicates that the first subject is lying on his stomach, and the third warning message is used to prompt the first subject to adjust his sleeping position. When the first angle is less than a ninth preset threshold and the second distance is less than a tenth preset threshold, the first subject is determined to be in a fifth sleep state, and a third warning message is generated, wherein the fifth sleep state indicates that the first subject is lying on his side.

[0085] When the target sleep mode is the third sub-sleep mode, a third distance between the center position of the limb area of ​​the second subject and the center position of the limb area of ​​the first subject is determined; when the third distance is less than an eleventh preset threshold, it is determined that the first subject is in a sixth sleep state, and a fourth warning message is generated, wherein the sixth sleep state indicates that squeezing occurs between the second subject and the first subject, and the fourth warning message is used to prompt the second subject to stay away from the first subject.

[0086] When the target sleep mode is the fourth sub-sleep mode, determine whether there is an overlapping area between the key point positions of the second subject and the limb area positions of the first subject; when there is an overlapping area, determine that the first subject is in a sixth sleep state, and generate a fourth warning message, wherein the sixth sleep state indicates that squeezing occurs between the second subject and the first subject, and the fourth warning message is used to prompt the second subject to stay away from the first subject.

[0087] Optionally, it is also possible to detect whether there is an occlusion area in the facial area of ​​the first object based on the classification detection algorithm in deep learning; when there is an occlusion area, generate a fifth warning information, wherein the fifth warning information is used to prompt that there is occlusion on the face of the first object and needs to be checked and processed.

[0088] The processing module 64 is configured to process the target area information and the target key point information based on a target processing method corresponding to the target sleep mode, obtain the sleep state of the first subject, and generate warning information corresponding to the sleep state.

[0089] It should be noted that the modules in the sleep detection and warning device in the embodiment of the present application correspond one-to-one to the implementation steps of the sleep detection and warning method in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be repeated here.

[0090] Example 3

[0091] According to an embodiment of the present application, a non-volatile storage medium is further provided. The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the sleep detection warning method in Example 1.

[0092] According to an embodiment of the present application, a processor is further provided, which is used to run a program, wherein the sleep detection and warning method in Example 1 is executed when the program is running.

[0093] Optionally, the following steps are performed while the program is running:

[0094] Acquire a target monitoring image for monitoring the sleep of a first subject; detect target area information and target key point information in the target monitoring image, wherein the target area information includes at least: face area information, limb area information and quilt area information, and the target key point information includes: human body key point information; determine a target sleep mode of the first subject based on the target area information and the target key point information; process the target area information and the target key point information based on a target processing method corresponding to the target sleep mode to obtain a sleep state of the first subject, and generate warning information corresponding to the sleep state.

[0095] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0096] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0097] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0098] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0099] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0100] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0101] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A sleep detection and warning method, characterized in that: include: acquiring a target monitoring image for monitoring the sleep of a first subject; Detecting target area information and target key point information in the target surveillance image, wherein the target area information includes at least: face area information, limb area information and quilt area information, the target key point information includes: human body key point information, and the face area information includes: the number of faces, the type of faces and the location of the face area; determining a target sleep mode for the first subject based on the target area information and the target key point information; processing the target area information and the target key point information based on a target processing method corresponding to the target sleep mode to obtain a sleep state of the first subject, and generating warning information corresponding to the sleep state; Among them, determining the target sleep mode of the first object based on the target area information and the target key point information includes: when the number of faces does not exceed a first preset threshold or the face type does not include the face of the second object, determining the target sleep mode to be the first sleep mode; when the number of faces exceeds the first preset threshold and the face type includes the face of the second object, determining the target sleep mode to be the second sleep mode.

2. The method according to claim 1, characterized in that Acquiring a target monitoring image for monitoring the sleep of a first subject, including: Acquire a target monitoring video for monitoring the sleep of the first subject; Redundant frame images and blurred frame images in all frame images of the target monitoring video are deleted to obtain the target monitoring image.

3. The method according to claim 1, characterized in that Detecting target area information and target key point information in the target monitoring image includes: Detecting the face area information in the target monitoring image based on a face detection algorithm in deep learning; Detecting the limb region information and the quilt region information in the target monitoring image based on a general target detection algorithm in deep learning, wherein the limb region information includes at least: a limb region position, and the quilt region information includes at least: a quilt-covered region position; The human body key point information in the target monitoring image is detected based on a key point detection algorithm in deep learning, and the human body key point information includes at least: the number of human body key points and the positions of human body key points.

4. The method according to claim 3, characterized in that The first sleep mode includes: a first sub-sleep mode and a second sub-sleep mode, the second sleep mode includes: a third sub-sleep mode and a fourth sub-sleep mode, and determining the target sleep mode of the first subject based on the target area information and the target key point information includes: In the first sleep mode, if the number of the key points on the human body does not exceed a second preset threshold, the target sleep mode is determined to be the first sub-sleep mode; if the number of the key points on the human body exceeds the second preset threshold, the target sleep mode is determined to be the second sub-sleep mode; In the second sleep mode, if the number of key points on the human body does not exceed a third preset threshold, the target sleep mode is determined to be the third sub-sleep mode; if the number of key points on the human body exceeds the third preset threshold, the target sleep mode is determined to be the fourth sub-sleep mode.

5. The method according to claim 4, characterized in that When the target sleep mode is the first sub-sleep mode, processing the target area information and the target key point information based on a target processing method corresponding to the target sleep mode to obtain a sleep state of the first subject, and generating warning information corresponding to the sleep state, including: Determining a first distance between a center position of a face region of the first subject and a center position of the quilt region, and a degree of overlap between a position of a limb region of the first subject and a position of a region covered by the quilt; When the first distance is less than a fourth preset threshold, determining that the first subject is in a first sleeping state and generating a first warning message, wherein the first sleeping state indicates that the first subject has covered the quilt too high, and the first warning message is used to prompt the first subject to pull the quilt down; When the overlap is less than a fifth preset threshold, it is determined that the first subject is in a second sleep state and a second warning message is generated, wherein the second sleep state indicates that the first subject has covered the quilt too low, and the second warning message is used to prompt the need to pull up the quilt.

6. The method according to claim 4, characterized in that When the target sleep mode is the second sub-sleep mode, processing the target area information and the target key point information based on a target processing method corresponding to the target sleep mode to obtain a sleep state of the first subject, and generating warning information corresponding to the sleep state, including: Determine a first number of head key points and a second number of limb key points among the human body key points of the first object, determine a first angle between the first key point vector and the second key point vector, and determine a second distance between the two-part key points, wherein the first key point vector is a vector from the nose key point to the neck key point, the second key point vector is a normal vector from a vector of two shoulder key points, and the two-part key points include at least one of the following: two shoulder key points, key points on both sides of the waist, two arm key points, and two knee key points; When the first number exceeds a sixth preset threshold and the second number exceeds a seventh preset threshold, determining that the first subject is in a third sleep state, wherein the third sleep state indicates that the first subject is lying on his back; When the first number is less than an eighth preset threshold and the second number exceeds a seventh preset threshold, determining that the first subject is in a fourth sleep state and generating a third warning message, wherein the fourth sleep state indicates that the first subject is lying prone, and the third warning message is used to prompt the first subject to adjust his sleeping position; When the first angle is less than a ninth preset threshold and the second distance is less than a tenth preset threshold, it is determined that the first subject is in a fifth sleep state, and the third warning information is generated, wherein the fifth sleep state indicates that the first subject is lying on his side.

7. The method according to claim 4, characterized in that When the target sleep mode is the third sub-sleep mode, processing the target area information and the target key point information based on a target processing method corresponding to the target sleep mode to obtain a sleep state of the first subject, and generating warning information corresponding to the sleep state, including: determining a third distance between a center position of a limb region of the second subject and a center position of a limb region of the first subject; When the third distance is less than an eleventh preset threshold, it is determined that the first object is in a sixth sleep state, and a fourth warning message is generated, wherein the sixth sleep state indicates that a squeeze occurs between the second object and the first object, and the fourth warning message is used to prompt the second object to stay away from the first object.

8. The method according to claim 4, characterized in that When the target sleep mode is the fourth sub-sleep mode, processing the target area information and the target key point information based on a target processing method corresponding to the target sleep mode to obtain a sleep state of the first subject, and generating warning information corresponding to the sleep state, including: Determine whether there is an overlapping area between the human body key point position of the second object and the limb area position of the first object; When the overlapping area exists, it is determined that the first object is in a sixth sleep state, and a fourth warning message is generated, wherein the sixth sleep state indicates that a squeeze occurs between the second object and the first object, and the fourth warning message is used to prompt the second object to stay away from the first object.

9. The method according to claim 3, characterized in that The method further comprises: Detecting whether there is an occlusion area at the face area of ​​the first object based on a classification detection algorithm in deep learning; When the occlusion area exists, fifth warning information is generated, wherein the fifth warning information is used to prompt that there is occlusion on the face of the first object and needs to be checked and processed.

10. A sleep detection and warning device, characterized in that: include: An acquisition module, configured to acquire a target monitoring image for monitoring the sleep of the first subject; a detection module, configured to detect target region information and target key point information in the target surveillance image, wherein the target region information includes at least: face region information, limb region information, and quilt region information; the target key point information includes: human body key point information; and the face region information includes: the number of faces, the type of faces, and the location of the face region; a determination module, configured to determine a target sleep mode for the first subject based on the target area information and the target key point information, including: determining that the target sleep mode is a first sleep mode when the number of faces does not exceed a first preset threshold or the face type does not include the face of the second subject; and determining that the target sleep mode is a second sleep mode when the number of faces exceeds the first preset threshold and the face type includes the face of the second subject; A processing module is used to process the target area information and the target key point information based on a target processing method corresponding to the target sleep mode, obtain the sleep state of the first object, and generate warning information corresponding to the sleep state.

11. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the sleep detection and warning method according to any one of claims 1 to 9.

12. A processor, characterized in that: The processor is configured to run a program, wherein the program, when running, executes the sleep detection and warning method according to any one of claims 1 to 9.

Images