Sleeping posture monitoring method and device, equipment and readable storage medium

Abstract

The invention discloses a sleeping posture monitoring method and device, equipment and a readable storage medium, and relates to the technical field of sleeping posture monitoring, and the method comprises the steps: obtaining the point cloud data of indoor information through a depth camera; from the point cloud data of the indoor information, separating the point cloud data of the sleeping posture of the human body through preprocessing; taking the point cloud data of the human body sleeping postures as input, and obtaining sleeping posture categories through a trained convolutional neural network classification model; and judging whether the sleeping posture state of the user accords with the setting or not through the sleeping posture category, and sending prompt information when the sleeping posture state of the user does not accord with the setting. According to the sleeping posture monitoring method, the inconvenience caused by using wearable equipment or using a bed pressure sensor in an existing sleeping posture recognition method is overcome, the privacy problem and the light interference problem caused by using an image acquisition camera are also avoided, the sleeping posture of the user can be accurately acquired by using the millimeter-level depth camera, and the user experience is improved. And the practicability of sleeping posture monitoring in a home environment is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of sleeping posture monitoring, and in particular to a sleeping posture monitoring method and device, equipment and readable storage medium. Background Art

[0002] Sleeping posture affects sleep quality and physical health. Correct sleeping posture can help maintain the natural curve of the spine, reduce pressure on the cervical and lumbar spine, and reduce muscle and joint discomfort. In addition, correct sleeping posture can also help breathe smoothly and help reduce snoring and breathing difficulties. In particular, infants, the elderly, the disabled and other groups have special needs for sleeping posture.

[0003] Currently, the publicly available sleeping posture monitoring methods mainly cover three methods of obtaining sleeping posture information. One is to take sleeping posture images based on visual image information. This method has privacy risks and is easily affected by ambient light. The other is to use wearable devices to obtain sleeping posture status. This method is not easily affected by the environment but requires additional wear, which is inconvenient and uncomfortable. The third is to use a sheet-type pressure sensor. The disadvantage of this collection method is that it is impossible to distinguish the sleeping posture contour through pressure data alone, and it is even impossible to determine whether the pressure source is the human body. Summary of the invention

[0004] In view of the above technical problems, the present invention proposes a sleeping posture monitoring method and device, equipment and readable storage medium. The specific technical solutions are as follows: A sleeping posture monitoring method, the method comprising the following steps: S1, obtain point cloud data of indoor information through depth camera; S2, separating the point cloud data of human sleeping posture from the point cloud data of indoor information through preprocessing; S3, using the point cloud data of human sleeping posture as input, obtaining the sleeping posture category through the trained convolutional neural network classification model; S4. Determine whether the user's sleeping posture conforms to the setting according to the sleeping posture category, and issue a prompt message when the user's sleeping posture conforms to the setting.

[0005] Furthermore, the depth camera in S1 uses a structured light depth camera, which projects a coded pattern so as to obtain millimeter-level measurement accuracy even in scenes with insufficient lighting and lack of texture.

[0006] Furthermore, the depth camera in S1 is installed above the bedroom, and the field of view of the depth camera covers the user's sleeping area including the bed, so as to obtain point cloud data of the bed and the user's sleeping posture.

[0007] Furthermore, the preprocessing steps in S2 include: S21, using the camera intrinsic parameter to convert the point cloud data in the pixel coordinate system into the point cloud data in the camera coordinate system; S22, identifying the bed plane in the point cloud data in the camera coordinate system using a random sampling consensus algorithm; S23. Based on the positional relationship between the bed surface and the human body, the point cloud data of the human body is filtered out.

[0008] Furthermore, in S3, the point cloud data of human sleeping posture is used as input, and the sleeping posture categories obtained by the trained convolutional neural network classification model include: supine, prone, left side, and right side.

[0009] Furthermore, in the step S4, when the user sets the sleeping position, he or she may set one or more desired sleeping positions according to his or her own needs, and a prompt sound may be issued when an incompatible sleeping position occurs.

[0010] A sleeping posture monitoring device, comprising: Acquisition module: The acquisition module is used to obtain the user's sleeping posture information detected by the depth camera; Extraction module: The extraction module is used to extract the point cloud data of the user's sleeping posture from the point cloud data of the acquired indoor information; Recognition module: The recognition module is used to input the point cloud data of the user's sleeping posture into the trained convolutional neural network for recognition and matching to obtain the sleeping posture category; Judgment module: The judgment module is used to compare the sleeping posture category obtained by identification and matching with the expected sleeping posture category set by the user, and issue a prompt message when the user's sleeping posture status does not meet the setting.

[0011] A computer device comprises a memory and a processor, wherein the memory stores instructions executable by the processor so that the processor can execute the steps of the sleeping posture monitoring method.

[0012] A computer-readable storage medium stores a computer program, which implements the steps of the sleeping posture monitoring method when executed by a processor.

[0013] The beneficial effects of the present invention compared with the prior art are: the sleeping posture monitoring method of the present application overcomes the inconvenience of using wearable devices or using pressure sensors on the bed in the existing sleeping posture recognition methods, and also avoids the privacy issues and light interference problems caused by using image acquisition cameras. By using a millimeter-level depth camera, the user's sleeping posture can be accurately captured, thereby improving the practicability of sleeping posture monitoring in a home environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the flow of the sleeping posture monitoring method of the present invention.

[0015] Figure 2 Schematic diagram of the process of the sleeping posture monitoring device of the present invention.

[0016] Figure 3 Schematic diagram of the principle of collecting information by the structured light depth camera of the present invention.

[0017] Figure 4 Schematic diagram of the installation position of the structured light depth camera of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with specific embodiments. The illustrative embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0019] Example: Figure 1 As shown, the present invention provides a sleeping posture monitoring method, the method comprising the following steps: S1, obtain point cloud data of indoor information through depth camera; S2, separating the point cloud data of human sleeping posture from the point cloud data of indoor information through preprocessing; S3, using the point cloud data of human sleeping posture as input, obtaining the sleeping posture category through the trained convolutional neural network classification model; S4. Determine whether the user's sleeping posture conforms to the setting according to the sleeping posture category, and issue a prompt message when the user's sleeping posture conforms to the setting.

[0020] like Figure 2 As shown, the present invention also provides a sleeping posture monitoring device, which includes: an acquisition module, an extraction module, an identification module and a judgment module; S1 is implemented by the acquisition module; S2 is implemented by the extraction module; S3 is implemented by the identification module; and S4 is implemented by the judgment module.

[0021] That is: the acquisition module is used to obtain the user's sleeping posture information detected by the depth camera; the extraction module is used to extract the point cloud data of the user's sleeping posture from the point cloud data of the indoor information obtained; the recognition module is used to input the point cloud data of the user's sleeping posture into the trained convolutional neural network for recognition and matching to obtain the sleeping posture category; the judgment module is used to compare the sleeping posture category obtained by recognition and matching with the expected sleeping posture category set by the user, and issue a prompt message when the user's sleeping posture status does not meet the setting.

[0022] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores instructions executable by the processor so that the processor can execute the steps of the sleeping posture monitoring method.

[0023] The present invention also provides a computer-readable storage medium storing a computer program, which implements the steps of the sleeping posture monitoring method when executed by a processor.

[0024] The depth camera in the S1 uses a structured light depth camera, which projects a coded pattern to obtain millimeter-level measurement accuracy even in scenes with insufficient lighting and lack of texture.

[0025] The depth camera in S1 is installed above the bedroom, and the field of view of the depth camera covers the user's sleeping area including the bed, so as to obtain point cloud data of the bed and the user's sleeping posture.

[0026] The schematic diagram of the structured light depth camera collecting the outline of an object is as follows Figure 3 As shown, the infrared projection end projects a beam of infrared laser, which passes through an optical element such as a diffraction grating to form a structured light spot. When this structured light spot is irradiated on the surface of an object, it will deform. The infrared receiving end calculates the depth information of each pixel on the surface of the object by analyzing the shape and size of this distortion, thereby generating a depth map. The data in the depth map is the distance from each pixel to the imaging plane. The structured light depth camera can accurately measure the three-dimensional contour information of objects in the field of view. When performing recognition, only the contour information of the face and torso is used, without other facial features or features of other parts, which can avoid the privacy avoidance problem of image acquisition cameras.

[0027] Examples of installation locations for structured light depth cameras include: Figure 4 As shown in the figure, the depth camera is installed directly above the bed, and its field of view covers the bed and the user's sleeping position. In the camera model, the camera coordinate system takes the optical center of the camera as the origin and the optical axis as the Z axis. In this installation method, the X-axis, Y-axis, and Z-axis directions of the camera coordinate system of the depth camera are as follows: Figure 4 As shown, the planes where the X-axis and the Y-axis are located are parallel to the bed surface, and the Z-axis is perpendicular to the bed surface.

[0028] In S2, the specific steps of separating the point cloud data of the human sleeping posture from the point cloud data of the indoor information through preprocessing include: S21, using the camera intrinsic parameter to convert the point cloud data in the pixel coordinate system into the point cloud data in the camera coordinate system; S22, identifying the bed plane in the point cloud data in the camera coordinate system using a random sampling consensus algorithm; S23. Based on the positional relationship between the bed surface and the human body, the point cloud data of the human body is filtered out.

[0029] In S3, the point cloud data of human sleeping posture is used as input, and the sleeping posture categories obtained by the trained convolutional neural network classification model include: supine, prone, left side, and right side.

[0030] In the step S4, when the user sets the sleeping position, he or she may set one or more desired sleeping positions according to his or her own needs, and a prompt sound may be issued when an incompatible sleeping position occurs.

[0031] The specific conversion formula for converting point cloud data in the pixel coordinate system into point cloud data in the camera coordinate system using the camera intrinsic parameters is: ;

[0032] in, Represents the distance from each pixel to the camera imaging plane, Represents the coordinates in the pixel coordinate system, Represents the internal parameters of the depth camera, Represents the coordinates in the camera coordinate system.

[0033] The principle of using the random sampling consensus algorithm to identify the bed plane in the point cloud data in the camera coordinate system is: the bed is the largest plane in the field of view, and the random sampling consensus algorithm is used to fit the largest plane in the point cloud data of indoor information. The specific operation steps are: randomly select three points as the minimum sample set; calculate a plane model based on these three points; calculate the distance from all other points to the plane; if the distance from a point to the plane is less than the set threshold, mark it as an inner point; if the number of inner points exceeds the previous maximum number of inner points, update the plane model with all current inner points; the above process will be repeated until the predicted number of iterations is reached; the plane model constructed by the sample with the largest number of inner points is the model of the largest plane to be found, and all the inner points belonging to the model are points belonging to the bed plane. The random sampling consensus algorithm can effectively eliminate the interference of other data and obtain a robust plane estimation result.

[0034] The step of filtering out the point cloud data of the human body based on the positional relationship between the bed surface and the human body is to solve the average value of the Z coordinates of the points belonging to the bed surface; in the camera coordinate system, the Z coordinate value of the points belonging to the human body is smaller than the Z coordinate of the points on the bed surface, and the points other than the points belonging to the bed surface whose Z coordinates are smaller than the average value of the Z coordinates of the points on the bed surface are filtered out.

[0035] The step of taking the point cloud data of human sleeping posture as input and obtaining the sleeping posture category through the trained convolutional neural network classification model is as follows: the recognition of the convolutional neural network is divided into two sections, the first section is the matching of the human head, the human head is the least likely to be blocked when sleeping, and the position and angle of all or part of the facial features are located according to the shape of each organ of the head and the distance characteristics between each organ, so as to judge the position and angle of the head. The data set of this section of the convolutional neural network covers the point cloud data of the human head in each posture of supine sleeping, prone sleeping, left side sleeping, and right side sleeping, and the position and angle of the facial features of the person are marked in each piece of data; the second section is the matching of the human body trunk, the data set of this section of the convolutional neural network covers the point cloud data of the human body trunk in each posture of supine sleeping, prone sleeping, left side sleeping, and right side sleeping, and the position and angle of the limbs of the person are marked in each piece of data. The category of sleeping posture is obtained by combining the position and angle of the facial features in the human head and the position and angle of the limbs in the trunk.

[0036] The obtained sleeping position category is compared with the sleeping position category set by the user. When a sleeping position that the user does not expect appears, a prompt sound is issued to remind the user, or a mobile device (such as a mobile phone, tablet, etc.) can be bound to send a message reminder. In addition, sleeping position statistics can be performed every night, and users can see in detail the maintenance time of each sleeping position through a bar chart.

Claims

1. A sleeping posture monitoring method, characterized in that: The method comprises the following steps: S1, obtain point cloud data of indoor information through depth camera; S2, separating the point cloud data of human sleeping posture from the point cloud data of indoor information through preprocessing; S3, using the point cloud data of human sleeping posture as input, obtaining the sleeping posture category through the trained convolutional neural network classification model; S4. Determine whether the user's sleeping posture conforms to the setting according to the sleeping posture category, and issue a prompt message when the user's sleeping posture conforms to the setting.

2. A sleeping posture monitoring method according to claim 1, characterized in that: The depth camera in the S1 uses a structured light depth camera, which projects a coded pattern to obtain millimeter-level measurement accuracy in scenes with insufficient lighting and lack of texture.

3. A sleeping posture monitoring method according to claim 1, characterized in that: The depth camera in S1 is installed above the bedroom, and the field of view of the depth camera covers the user's sleeping area including the bed, so as to obtain point cloud data of the bed and the user's sleeping posture.

4. A sleeping posture monitoring method according to claim 1, characterized in that: The preprocessing steps in S2 include: S21, using the camera intrinsic parameter to convert the point cloud data in the pixel coordinate system into the point cloud data in the camera coordinate system; S22, identifying the bed plane in the point cloud data in the camera coordinate system using a random sampling consensus algorithm; S23. Based on the positional relationship between the bed surface and the human body, the point cloud data of the human body is filtered out.

5. A sleeping posture monitoring method according to claim 1, characterized in that: In S3, the point cloud data of human sleeping posture is used as input, and the sleeping posture categories obtained by the trained convolutional neural network classification model include: supine, prone, left side, and right side.

6. A sleeping posture monitoring method according to claim 1, characterized in that: In the step S4, when the user sets the sleeping position, he or she may set one or more desired sleeping positions according to his or her own needs, and a prompt sound may be issued when an incompatible sleeping position occurs.

7. A sleeping posture monitoring device, characterized in that: The sleeping posture monitoring device comprises: Acquisition module: The acquisition module is used to obtain the user's sleeping posture information detected by the depth camera; Extraction module: The extraction module is used to extract the point cloud data of the user's sleeping posture from the point cloud data of the acquired indoor information; Recognition module: The recognition module is used to input the point cloud data of the user's sleeping posture into the trained convolutional neural network for recognition and matching to obtain the sleeping posture category; Judgment module: The judgment module is used to compare the sleeping posture category obtained by identification and matching with the expected sleeping posture category set by the user, and issue a prompt message when the user's sleeping posture status does not meet the setting.

8. A computer device comprising a memory and a processor, characterized in that: The memory stores instructions executable by the processor, so that the processor can perform the steps of the sleeping posture monitoring method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the sleeping posture monitoring method according to any one of claims 1 to 6 are implemented.

Images