Sleeping posture recognition method and device and storage medium

Abstract

The invention discloses a sleeping posture recognition method and device and a storage medium, and the method comprises the steps: obtaining a current frame pressure image which comprises at least one target user; determining a foreground detection frame corresponding to a foreground area in the current frame pressure image; based on the current frame pressure image, predicting to obtain an initial sleeping posture recognition result of the target user; based on the foreground detection frame, correcting the initial sleeping posture recognition result; and smoothing the corrected initial sleeping posture recognition result to obtain a target sleeping posture recognition result of the target user. The current frame image is obtained, the foreground detection frame is determined, and the initial sleeping posture recognition result is corrected based on the foreground detection frame, so that the recognition accuracy is effectively improved. In the face of a complex scene, the foreground detection frame is used for comprehensive judgment, and interference factors such as noise, body movement and sleeping posture imaging missing can be better handled. And meanwhile, the corrected result is smoothed, so that the stability of the recognition result is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of smart home, and in particular to a sleeping posture recognition method, device and storage medium. Background Art

[0002] With the development of smart home technology, smart mattresses and sleep monitoring systems have gradually become popular. Accurately identifying the sleeping posture of users is of great significance for comprehensively evaluating sleep quality.

[0003] Currently, the existing sleeping posture recognition technologies mainly include two methods: sensor-based and image recognition-based. The sensor-based sleeping posture recognition technology classifies sleeping postures by collecting pressure distribution maps. However, this method can only support classification under relatively ideal conditions, and is greatly affected by body movements, environment, etc., resulting in poor classification accuracy and stability. The image recognition-based technology usually directly relies on an image recognition model to predict the sleeping posture result during sleeping posture recognition. In actual application scenarios, the image may have complex situations such as noise interference, light changes, and human body posture occlusion, resulting in poor accuracy of the recognition result. Summary of the Invention

[0004] Based on this, it is necessary to provide a sleeping posture recognition method, device and storage medium for the above technical problems to solve at least one of the problems existing in the above prior art.

[0005] In a first aspect, a sleeping posture recognition method is provided, including:

[0006] Obtain a current frame pressure image, where the current frame pressure image includes at least one target user;

[0007] Determine a foreground detection box corresponding to the foreground area in the current frame pressure image;

[0008] Based on the current frame pressure image, predict an initial sleeping posture recognition result of the target user;

[0009] Based on the foreground detection box, correct the initial sleeping posture recognition result;

[0010] Perform smoothing processing on the corrected initial sleeping posture recognition result to obtain a target sleeping posture recognition result of the target user.

[0011] In an embodiment of the present application, the correcting the initial sleeping posture recognition result based on the foreground detection box includes:

[0012] Determine the aspect ratio of the foreground detection box and the area ratio of the area of the foreground detection box to the area of the current frame pressure image;

[0013] Determine the foreground pixel ratio of the foreground detection box in the current frame pressure image;

[0014] Determine whether the aspect ratio, area ratio, and foreground pixel ratio of the foreground detection box meet the preset correction conditions respectively;

[0015] If the preset correction conditions are met, correct the initial sleeping posture recognition result according to the preset correction rules.

[0016] In an embodiment of the present application, the step of determining whether the aspect ratio, area ratio, and foreground pixel ratio of the foreground detection box meet the preset correction conditions respectively includes:

[0017] Determine whether the foreground pixel ratio is less than a first preset threshold;

[0018] Determine whether the aspect ratio of the foreground detection box is greater than a second preset threshold;

[0019] Determine whether the area ratio of the foreground detection box to the area of the current frame pressure image is less than a third preset threshold.

[0020] In an embodiment of the present application, after determining whether the aspect ratio, area ratio, and foreground pixel ratio of the foreground detection box meet the preset correction conditions respectively, the method further includes:

[0021] If the preset correction conditions are not met, determine the sleeping posture category with the highest confidence in the initial sleeping posture recognition result;

[0022] If the sleeping posture category with the highest confidence is an unknown category and the confidence of the unknown category is less than the preset confidence threshold, correct the initial sleeping posture recognition result to the sleeping posture category with the second highest confidence, otherwise, correct the initial sleeping posture recognition result to the sleeping posture category with the highest confidence.

[0023] In an embodiment of the present application, the step of smoothing the corrected initial sleeping posture recognition result to obtain the target sleeping posture recognition result includes:

[0024] Save the initial sleeping posture recognition result corresponding to one frame of image at intervals of a preset time;

[0025] Count the number of each sleeping posture category in all the initial sleeping posture recognition results;

[0026] If the number of types of sleeping posture categories is less than a preset number threshold, determine the sleeping posture category with the largest number as the target sleeping posture recognition result;

[0027] If the number of types of sleeping posture categories is greater than or equal to the preset number threshold, set the target sleeping posture recognition result as an unknown category.

[0028] In one embodiment of the present application, before counting the number of each sleep posture category in all initial sleep posture recognition results within a preset time range, the following steps are further included:

[0029] If the number of the initial sleep posture recognition results is greater than a preset number threshold, clear the initial sleep posture recognition result corresponding to the starting frame pressure image, and save the initial sleep posture recognition result corresponding to the current frame pressure image.

[0030] In one embodiment of the present application, predicting the initial sleep posture recognition result of the target user based on the current frame pressure image includes:

[0031] Convert the current frame pressure image into a first grayscale image of a first preset size;

[0032] Input the grayscale image into a pre-trained sleep posture recognition model for prediction processing to obtain the initial sleep posture recognition result of the target user;

[0033] Wherein, the sleep posture recognition model includes a first convolutional layer, a first rectified linear unit, a second convolutional layer, a second rectified linear unit, a pooling layer, and a fully connected layer connected in sequence.

[0034] In one embodiment of the present application, after obtaining the current frame pressure image, the following steps are further included:

[0035] Convert the current frame pressure image into a second grayscale image of a second preset size;

[0036] Convert the second grayscale image into a heat map, and the display area of the heat map is used to display the target sleep posture recognition result.

[0037] In a second aspect, a sleep posture recognition device is provided, including:

[0038] A current frame pressure image acquisition unit, configured to acquire a current frame pressure image;

[0039] A foreground detection box determination unit, configured to determine a foreground detection box corresponding to a foreground area in the current frame pressure image;

[0040] A prediction unit, configured to predict an initial sleep posture recognition result based on the current frame pressure image;

[0041] An identification result correction unit, configured to correct the initial sleep posture recognition result based on the foreground detection box;

[0042] An identification result smoothing unit, configured to perform smoothing processing on the corrected initial sleep posture recognition result to obtain a target sleep posture recognition result.

[0043] In a third aspect, a readable storage medium is provided, which stores computer-readable instructions that, when executed by a processor, implement the steps of the sleeping posture recognition method according to any one of claims 1 to 7.

[0044] The above-mentioned sleeping posture recognition method, device and storage medium, the implementation of the method includes: obtaining a current frame pressure image, where the current frame pressure image includes at least one target user; determining a foreground detection box corresponding to the foreground area in the current frame pressure image; predicting an initial sleeping posture recognition result of the target user based on the current frame pressure image; correcting the initial sleeping posture recognition result based on the foreground detection box; and smoothing the corrected initial sleeping posture recognition result to obtain a target sleeping posture recognition result of the target user. In the embodiments of the present application, by obtaining the current frame image, determining the foreground detection box, and correcting the initial sleeping posture recognition result based on this, the accuracy of recognition is effectively improved. When facing complex scenarios, by comprehensively judging using multi-dimensional information such as the aspect ratio, area, and foreground pixel ratio of the foreground detection box, it is possible to better cope with interference factors such as noise, body movement, and missing sleeping posture imaging. At the same time, smoothing the corrected result further improves the stability of the recognition result. It can provide more comprehensive and accurate reliable sleeping posture data support for sleep health monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments of the present invention. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0046] Figure 1 is a schematic diagram of an application environment of the sleeping posture recognition method in an embodiment of the present invention;

[0047] Figure 2 is a schematic flowchart of the sleeping posture recognition method in an embodiment of the present invention;

[0048] Figure 3 is a schematic diagram of a foreground detection box in an embodiment of the present invention;

[0049] Figure 4 is a schematic flowchart of the sleeping posture result correction method in an embodiment of the present invention;

[0050] Figure 5 is a schematic flowchart of the sleeping posture result smoothing processing method in an embodiment of the present invention;

[0051] Figure 6It is the model architecture diagram of the sleeping posture recognition model in an embodiment of the present invention;

[0052] Figure 7 It is an interface display diagram of the heat map in an embodiment of the present invention;

[0053] Figure 8 It is the sleeping posture classification curve graph before processing in an embodiment of the present invention;

[0054] Figure 9 It is the sleeping posture classification curve graph after processing in an embodiment of the present invention;

[0055] Figure 10 It is a structural schematic diagram of the sleeping posture recognition device in an embodiment of the present invention;

[0056] Figure 11 It is a schematic diagram of a computer device in an embodiment of the present invention. Specific embodiments

[0057] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0058] The sleeping posture recognition method provided in this embodiment can be applied in an application environment such as Figure 1 . Among them, the sleeping posture recognition model can be deployed in an embedded device. The pressure detection device, such as a pressure sensor, can collect pressure data every preset time interval, such as 0.1 second, and upload it to the embedded device. The embedded device can convert the pressure map data into grayscale images of a preset size, such as 32*32 and 32*16. Then, the 32*16 grayscale image can be input into the sleeping posture recognition model pre-deployed in the embedded device for predicting the sleeping posture recognition result, and the predicted sleeping posture recognition result can be sent to the PC side. The PC side generates a folder named after the sleeping posture label, and at the same time, the 32*16 grayscale Figure 1 image is saved. The 32*32 grayscale image can be uploaded to the host computer for heat map conversion and heat map display. At the same time, the sleeping posture prediction result label can be displayed in the heat map to realize label visualization.

[0059] If the sleep posture category in the sleep posture recognition result is greater than a preset threshold, for example, greater than 0, the name of the sleep posture category can be displayed in the heat map display area of the host computer; otherwise, no display is performed. By deploying the sleep posture recognition model in an embedded device, the test results can be visualized, the test efficiency can be effectively improved, and the real-time data and recognition results output by the model can be obtained, achieving the effect of data pre-annotation and reducing the cost of manual data annotation.

[0060] In one embodiment, as Figure 2 shown, a sleep posture recognition method is provided, including the following steps:

[0061] In step S110, obtain the current frame pressure image, where the current frame pressure image includes at least one target user;

[0062] Optionally, a pressure detection device can be set on an object for users to rest, such as a pressure blanket can be set on a smart mattress, a sofa, a bed or other devices for users to lie on, or multiple pressure detection devices arranged in an array can be set. The pressure detection devices can be arranged at equal intervals in an array longitudinally and horizontally, and then the pressure image can be collected in real time, regularly or at intervals of a preset time, for example, once every second.

[0063] It can be understood that the pressure image can include the pressure image of the area where the human body is located and the pressure image of the unoccupied area. Among them, the pressure image of the area where the human body is located can be connected according to the pressure detection devices that obtain the pressure data, and then the image of the area where the human body is located can be obtained. The unoccupied area can refer to the area where the pressure detection device does not detect pressure data, that is, the area without human contact and without heavy object pressure.

[0064] Among them, the pressure detection device can be a pressure sensor.

[0065] It should be noted that the current frame pressure image can include at least one target user. For example, in the scenario of two people in bed or multiple people in bed, the pressure image can include images of multiple target users at this time, so as to perform sleep posture recognition on multiple users simultaneously. If the recognized pressure image does not include a target user, the collected image can be discarded without subsequent operations.

[0066] It should be noted that multiple consecutive image frames can be collected, and the image with the highest clarity and the most complete user area is selected as the current frame image.

[0067] In step S120, determine the foreground detection box corresponding to the foreground area in the current frame pressure image;

[0068] It should be noted that the foreground area is the area in the current frame image that contains the body part of the target user. The pixel value of the foreground area is greater than 0. As Figure 3As shown in the figure, a schematic diagram of a foreground detection frame is provided. In the figure, W represents the width of the foreground display frame, h represents the height of the foreground detection frame, and the coordinates of the leftmost, rightmost, topmost, and bottommost positions of the target user in the figure are (x1, y1), (x2, y2), (x3, y3), and (x4, y4) respectively.

[0069] Optionally, the current frame pressure image can be grayscaled. For example, a color image can be converted into a single-channel grayscale image. Traverse each pixel point in the image. If the grayscale value of the pixel point is greater than a preset pixel threshold, such as 0, it can be determined as a foreground pixel; otherwise, it is determined as a background pixel. Then, a connected component analysis algorithm, such as a marker-based connected component analysis method, can be used to merge the connected foreground pixels into a connected region. For each connected region, calculate its bounding rectangle, and this rectangle is the foreground detection frame.

[0070] Optionally, edge information in the current frame pressure image can be detected through an edge detection algorithm, such as the Canny edge detection algorithm. After obtaining the edge image, use a contour search algorithm, such as the findContours function in OpenCV, to search for the contours in the image. The contours represent the boundaries of different objects or regions in the image. For each contour, calculate its minimum bounding rectangle, and this rectangle is the foreground detection frame. It should be noted that in the scenario where multiple people are in the same bed, multiple contours may be detected. According to actual needs, some screening conditions, such as the rectangle area size, aspect ratio, etc., can be set to determine which rectangles are the foreground detection frames corresponding to the target user.

[0071] Optionally, the determination of the foreground detection frame can also be based on a deep learning object detection model. For example, FasterR-CNN, YOLO, etc. First, collect a large number of pressure image samples containing users and label the foreground regions (i.e., the regions where the users are located) to generate a training dataset. Then, use this dataset to train the object detection model so that the model learns the features of the foreground regions. Then, the current frame pressure image can be input into the trained object detection model, and the model will output the position information of the foreground region, which is usually represented in the form of a bounding box, that is, the foreground detection frame.

[0072] In step S130, based on the current frame pressure image, the initial sleeping posture recognition result of the target user is predicted;

[0073] Optionally, a sleep recognition model can be constructed, such as a convolutional neural network, decision tree, support vector machine (SVM), random forest, etc. Taking the convolutional neural network as an example, the training dataset can be used to iteratively train the sleep recognition model until it meets the preset convergence conditions. For example, when the number of iterations is greater than the preset number or the loss value is less than the preset loss threshold, a trained sleep recognition model can be obtained for subsequent prediction use. After obtaining the current frame pressure image, the current frame pressure image can be input into the trained sleeping posture recognition model for sleeping posture recognition. The sleeping posture recognition model can include multiple cascaded convolutional layers. Through the convolutional layers, corresponding feature maps can be extracted, and the output of each convolutional layer can be used as the input of the next convolutional layer. Before inputting it into the next convolutional layer, the feature map extracted by the convolutional layer can be activated through an activation function. Then, the feature maps extracted through multiple convolutional layers can be input into the pooling layer for pooling processing, such as max pooling or average pooling, to perform dimensionality reduction operations on the feature maps output by the convolutional layer, reduce the amount of data, and retain important feature information at the same time. After the processing of the convolutional layer and the pooling layer, the obtained feature maps contain rich abstract features related to the sleeping posture. The fully connected layer flattens these feature maps into one-dimensional vectors and, through a series of weight matrix multiplications and activation function operations, maps the features to different sleeping posture categories, and finally, the probability distribution prediction results of different sleeping posture categories can be obtained.

[0074] Optionally, pressure images of various known sleeping postures can be collected as template images and accurately labeled with sleeping postures. Feature extraction is performed on each template image, and the extracted features can be the overall pressure distribution pattern, pressure features of key parts, etc. The extracted features are combined into a template vector to construct a template library, and each template vector represents a specific sleeping posture. The same feature extraction operation as the template image is performed on the current frame pressure image to obtain the feature vector to be matched. Then, a preset similarity algorithm, such as Euclidean distance, cosine similarity, etc., can be used to calculate the similarity between the feature vector to be matched and each template vector in the template library. If the similarity is greater than the preset threshold, it can be considered consistent, and thus the sleeping posture category can be determined.

[0075] Among them, the initial sleeping posture recognition result can include the sleeping posture category and the confidence corresponding to each sleeping posture category. The sleeping posture category can include 0 for the unknown category, 1 for supine, 2 for left lateral lying, 3 for left curled, 4 for right lateral lying, 5 for right curled, and 6 for prone.

[0076] It should be noted that the order of step S120 and step S130 is not sequential, that is, step S120 can be executed first and then step S130, or step S130 can be executed first and then step S120, or step S120 and step S130 can be executed simultaneously. That is, after obtaining the current frame pressure image, the generation of the initial sleeping posture recognition result and the determination of the foreground detection frame can be performed respectively.

[0077] In step S140, based on the foreground detection box, the initial sleeping posture recognition result is corrected;

[0078] Optionally, after determining the foreground detection box, data such as the aspect ratio of the foreground detection box, the area ratio of the area occupied by the foreground detection box to the area of the pressure image, and the pixel ratio of the pixels occupied by the foreground detection box to the pixels of the pressure image can be further determined. If the aspect ratio, area ratio, and pixel ratio all meet the preset correction conditions, the initial sleeping posture recognition result can be corrected to the unknown class. If any one of them does not meet the preset correction conditions, it can be determined whether the maximum confidence is less than the preset threshold and whether the corresponding class is the unknown class. If so, the second-highest confidence class is used as the initial sleeping posture recognition result.

[0079] It should be noted that the sleeping posture recognition result can also be corrected by checking the user features within the foreground detection box. For example, the pressure distribution details of various parts of the user's body may be included within the detection box. By analyzing the pressure concentration areas and shapes of key parts such as the head, shoulders, and hips, the sleeping posture can be judged. If it is detected that the pressure points of the head are on one side and the pressure distributions of the shoulders and hips form a certain angle, the initial recognition result can be corrected to the side-lying position.

[0080] Alternatively, the change situation between the foreground detection box of the current frame and adjacent frames can also be analyzed. If, in several consecutive frames, the position and shape of the foreground detection box change relatively smoothly and the initial sleeping posture recognition results are consistent, then this result can be considered relatively reliable; if the initial sleeping posture recognition result of the current frame differs greatly from that of adjacent frames, it may be necessary to correct it according to the situation of adjacent frames. For example, if the previous frame and the next frame are both recognized as the supine position, while the current frame is initially recognized as the side-lying position and the change of the foreground detection box in the current frame is not caused by an obvious action switch, then the sleeping posture of the current frame can be corrected to the supine position.

[0081] In step S150, the corrected initial sleeping posture recognition result is smoothed to obtain the target sleeping posture recognition result of the target user.

[0082] Optionally, since the acquisition of pressure images is a continuous process, the sleep posture recognition can be performed on each acquired frame of pressure image, and the initial sleep posture recognition results corresponding to each frame of pressure image can be saved. When the latest initial sleep posture recognition result is obtained, all the saved initial sleep posture recognition results can be counted to determine the number of each sleep posture category. If the number of sleep posture categories is less than the preset number, for example, 3, the sleep posture category with the largest number is used as the target sleep posture recognition result. For example, if the category of supine appears the most times, then supine is determined as the target sleep posture recognition result. If the number of sleep posture categories is greater than or equal to the preset number, for example, 3, it indicates that the target user may think there is body movement, and the target sleep posture recognition result can be set to the unknown category. By smoothing the corrected initial sleep posture recognition results, it is possible to exclude to a certain extent those minority categories caused by accidental factors or recognition errors, making the final result closer to the real sleep posture situation and improving the recognition accuracy.

[0083] It should be noted that a pressure pattern can be obtained every preset time interval, such as 0.1 second, that is, an initial sleep posture recognition result can be generated every preset time interval. Therefore, the recognition results of one frame of pressure image are saved every 1 second. If the number of saved results exceeds the preset number, for example, 5, the earliest frame needs to be cleared, and at the same time, the initial sleep posture recognition result of the latest frame of pressure image is saved.

[0084] The embodiment of the present application provides a sleep posture recognition method, including: acquiring a current frame of pressure image, where the current frame of pressure image includes at least one target user; determining a foreground detection frame corresponding to the foreground area in the current frame of pressure image; predicting an initial sleep posture recognition result of the target user based on the current frame of pressure image; correcting the initial sleep posture recognition result based on the foreground detection frame; and smoothing the corrected initial sleep posture recognition result to obtain the target sleep posture recognition result of the target user. In the embodiment of the present application, by acquiring the current frame image, determining the foreground detection frame, and correcting the initial sleep posture recognition result based on this, the recognition accuracy is effectively improved. When facing a complex scene, comprehensive judgment is made using multi-dimensional information such as the aspect ratio, area, and foreground pixel ratio of the foreground detection frame, which can better handle interference factors such as noise, body movement, and missing sleep posture imaging. At the same time, smoothing the corrected result further improves the stability of the recognition result. It can provide more comprehensive and accurate reliable sleep posture data support for sleep health monitoring.

[0085] In an embodiment of the present application, the correcting the initial sleep posture recognition result based on the foreground detection frame includes:

[0086] determining the aspect ratio of the foreground detection frame and the area ratio of the area of the foreground detection frame to the area of the current frame of pressure image;

[0087] Determine the foreground pixel ratio of the foreground detection box in the current frame pressure image;

[0088] Respectively determine whether the aspect ratio, area ratio, and the foreground pixel ratio of the foreground detection box meet the preset correction conditions;

[0089] If it meets the preset correction conditions, correct the initial sleeping posture recognition result according to the preset correction rules.

[0090] Optionally, after obtaining the foreground detection box, the length and width of the foreground detection box can be obtained, and then the aspect ratio and area can be calculated. For example, if the width of the foreground detection box is 100 pixels and the height is 200 pixels, the aspect ratio is 0.5. Then the foreground area = 100 * 200 = 20000 square pixels. At the same time, the total area of the pressure image can be calculated, and the area ratio = foreground area / total area of the pressure image.

[0091] By traversing all the pixels within the foreground detection box, count the number of foreground pixels within the foreground detection box. At the same time, count the total number of pixels within the foreground detection box. Through the formula: foreground pixel ratio = number of foreground pixels / total number of pixels, the pixel ratio can be obtained.

[0092] The preset correction conditions can be configured according to the actual application scenario and business requirements. If the aspect ratio, area ratio, and the foreground pixel ratio of the foreground detection box all meet the above preset correction conditions, then correct the initial sleeping posture recognition result according to the preset correction rules.

[0093] In an embodiment of the present application, the step of respectively determining whether the aspect ratio, area ratio, and the foreground pixel ratio of the foreground detection box meet the preset correction conditions includes:

[0094] Determine whether the foreground pixel ratio is less than a first preset threshold;

[0095] Determine whether the aspect ratio of the foreground detection box is greater than a second preset threshold;

[0096] Determine whether the area ratio of the foreground detection box to the area of the current frame pressure image is less than a third preset threshold.

[0097] Optionally, the preset correction conditions can be configured according to the actual application scenario and business requirements. The aspect ratio, area ratio, and the proportion of foreground pixels can respectively correspond to different preset correction conditions. For example, as Figure 4 shown, the foreground pixel ratio is less than a first preset threshold, such as 0.1, the aspect ratio is greater than a second preset threshold, such as 0.7, and the area ratio of the foreground circumscribed rectangle is less than a third preset threshold, such as 0.4. It should be noted that the preset correction conditions can be dynamically adjusted.

[0098] In one embodiment of the present application, after separately determining whether the aspect ratio, area ratio, and foreground pixel ratio of the foreground detection box meet the preset correction conditions, the following steps are further included:

[0099] If it does not meet the preset correction conditions, determine the sleep posture category with the highest confidence in the initial sleep posture recognition result;

[0100] If the sleep posture category with the highest confidence is the unknown category, and the confidence of the unknown category is less than the preset confidence threshold, then correct the initial sleep posture recognition result to the sleep posture category with the second highest confidence; otherwise, correct the initial sleep posture recognition result to the sleep posture category with the highest confidence.

[0101] Optionally, as Figure 4 shown, if the aspect ratio, area ratio, and pixel ratio all meet the preset correction conditions, the initial sleep posture recognition result can be corrected to the unknown category. If any one of them does not meet the preset correction conditions, it can be determined whether the maximum confidence is less than the preset threshold, such as 0.95. If so, the category with the second highest confidence is used as the initial sleep posture recognition result; otherwise, the initial sleep posture recognition result is corrected to the sleep posture category with the highest confidence.

[0102] It should be noted that to avoid misjudgment caused by non-human interference objects such as pillows, quilts, pets, etc. and the noise of the pressure sensor. First, for the aspect ratio, area ratio, and foreground pixel ratio of the foreground detection box, those that meet the preset correction conditions can be recognized as non-human objects and set to the unknown category. Second, if they do not meet the preset correction conditions, it can be further determined whether the sleep posture category with the highest confidence is the unknown category, and whether the confidence of the unknown category is less than the preset confidence threshold. If so, the initial sleep posture recognition result is corrected to the sleep posture category with the second highest confidence; otherwise, the initial sleep posture recognition result is corrected to the sleep posture category with the highest confidence.

[0103] Optionally, the smoothing process of the corrected initial sleep posture recognition result to obtain the target sleep posture recognition result includes:

[0104] Save the initial sleep posture recognition result corresponding to one frame of image at every preset time interval;

[0105] Count the number of each sleep posture category in all the initial sleep posture recognition results;

[0106] If the number of types of sleep posture categories is less than the preset number threshold, determine the sleep posture category with the largest number as the target sleep posture recognition result;

[0107] If the number of types of sleep posture categories is greater than or equal to the preset number threshold, set the target sleep posture recognition result to the unknown category.

[0108] As Figure 5 shown, since the acquisition of the pressure images is a continuous process, the sleeping posture recognition can be performed on each frame of the acquired pressure images, and the initial sleeping posture recognition results corresponding to each frame of the pressure images can be saved. When the latest initial sleeping posture recognition result is obtained, all the saved initial sleeping posture recognition results can be counted to determine the number of each sleeping posture category. If the number of sleeping posture categories is less than the preset number, for example, 3, the sleeping posture category with the largest number is used as the target sleeping posture recognition result at this time. For example, if the category of supine appears the most times, then supine is determined as the target sleeping posture recognition result. If the number of sleeping posture categories is greater than or equal to the preset number, for example, 3, it indicates that the target user may think that there is body movement, and the target sleeping posture recognition result can be set to the unknown category. By performing smoothing processing on the corrected initial sleeping posture recognition results, it is possible to exclude to a certain extent those minority categories caused by accidental factors or recognition errors, making the final result closer to the real sleeping posture situation and improving the recognition accuracy.

[0109] In an embodiment of the present application, before counting the number of each sleeping posture category in all the initial sleeping posture recognition results within a preset time range, it further includes:

[0110] If the number of the initial sleeping posture recognition results is greater than the preset number threshold, the initial sleeping posture recognition result corresponding to the starting frame pressure image is cleared, and the initial sleeping posture recognition result corresponding to the current frame pressure image is saved.

[0111] A pressure pattern can be obtained every preset time interval, such as 1 second, that is, an initial sleeping posture recognition result can be generated every preset time interval. Therefore, the recognition result of one frame of the pressure map is saved every 1 second. If the number of saved results exceeds the preset number, for example, 5, the earliest frame needs to be cleared, and at the same time, the initial sleeping posture recognition result of the latest frame of the pressure image is saved.

[0112] In an embodiment of the present application, the predicting the initial sleeping posture recognition result of the target user based on the current frame pressure image includes:

[0113] Converting the current frame pressure image into a first grayscale image with a first preset size;

[0114] Inputting the grayscale image into a pre-trained sleeping posture recognition model for prediction processing to obtain the initial sleeping posture recognition result of the target user;

[0115] Optionally, refer to Figure 6, the sleeping posture recognition model includes a first convolutional layer, a first rectified linear unit, a second convolutional layer, a second rectified linear unit, a pooling layer, and a fully connected layer connected in sequence. The first convolutional layer (conv) consists of 32 convolutional kernels of size 3*3, with a stride of 2 and a padding of 1 to achieve downsampling. The input is a two-dimensional grayscale image with a resolution of 32*16. After being processed by the first convolutional layer, a feature map of size 16*8*32 can be generated. The feature map is activated by the first rectified linear unit, and a feature map of size 16*8*32 can be output; the second convolutional layer (conv) consists of 32 convolutional kernels of size 3*3, with a stride of 2 and a padding of 1 to achieve downsampling. After the feature map of size 16*8*32 is input into the second convolutional layer, a feature map of size 8*4*32 can be generated. The feature map of size 8*4*32 is activated by the second rectified linear unit, and a feature map of size 8*4*32 can be output; the third max pooling layer (maxpool) has a size of 3*3, a stride of 1, and a padding of 1 to keep the feature Figure 8 *4*32 size unchanged. The fully connected layer (FC) consists of 7 vectors of 1*1024, and the output regression vector has a size of 1*7. Finally, after classification by the softmax function, a confidence vector of 1*7 is output. It should be noted that the sleeping posture recognition model can define 7 sleeping posture category labels, which are 0 for the unknown class, 1 for the supine position, 2 for the left lateral position, 3 for the left curled position, 4 for the right lateral position, 5 for the right curled position, and 6 for the prone position. The category label with the highest confidence in the final output is the initial sleeping posture recognition result.

[0116] It should be noted that the number of convolutional layers can also be 1, 3, etc., which can be specifically set according to the actual situation.

[0117] In an embodiment of the present application, after obtaining the current frame pressure image, the following steps are further included:

[0118] Convert the current frame pressure image into a second grayscale image with a second preset size;

[0119] Convert the second grayscale image into a heat map, and the display area of the heat map is used to display the target sleeping posture recognition result.

[0120] Optionally, the sleeping posture detection system may specifically include a sensor, an embedded device, and a host computer. The sleeping posture recognition model can be deployed in the embedded device. The pressure detection device, such as a pressure sensor, can collect pressure data every preset time interval, such as 0.1 second, and upload it to the embedded device. The embedded device converts the pressure map data into two grayscale images with a preset size, such as 32*32 and 32*16, and then the 32*32 grayscale image can be uploaded to the host computer for heat map conversion and heat map display.

[0121] The 32*16 grayscale image can be input into the sleep posture recognition model pre-deployed in the embedded device to predict the sleep posture recognition result, and the predicted sleep posture recognition result is sent to the PC side. A folder named with the sleep posture label is generated through the PC side, and at the same time, the 32*16 grayscale Figure 1 is saved.

[0122] It should be noted that a mapping relationship can be established between the grayscale image and the heat map, and the grayscale value is mapped to different color ranges. For example, a lower grayscale value can be mapped to a cold color tone (such as blue), indicating that the pressure or related features corresponding to this area in the original pressure image are lower; a higher grayscale value is mapped to a warm color tone (such as red), indicating that the pressure or related features corresponding to this area are higher, so as to realize the conversion of the heat map. Specifically, it can be realized through a linear mapping function or a non-linear mapping function.

[0123] If the sleep posture category of the sleep posture recognition result is greater than the preset threshold, for example, greater than 0, the sleep posture category name can be displayed in the heat map display area of the upper computer, otherwise it is not displayed. As Figure 7 shown in the figure, the heat maps of 2 users are shown. Among them, the sleep posture recognition result of the user on the left is right lateral lying, and the sleep posture recognition result of the user on the right is supine. The sleep posture recognition result is directly marked in the heat map display area, which can be viewed intuitively. And this heat map also includes a real-time test result log area, so as to count and detect the sleep posture changes of users within a preset time range, so as to understand the sleep quality of users. By deploying the sleep posture recognition model in the embedded device, the test result visualization is realized, the test efficiency can be effectively improved, and the real-time data and recognition results output by the model can be obtained, which plays the role of data pre-annotation and reduces the cost of manual data annotation.

[0124] To further illustrate the accuracy and reliability of the prediction results of the present application, see Figure 8 and Figure 9 , where Figure 8 is the sleep posture classification curve graph before processing, that is, by inputting the sleep posture image into the model, such as a CNN model, the sleep posture result classification curve graph is predicted, Figure 9 is the sleep posture classification curve graph after processing, that is, in the present application, the prediction result is corrected by the correction rule, and the classification curve graph after smoothing the correction result. In the figure, the abscissa can be understood as the time axis: Figure 8 in the processing before, the data is processed at 10 frames per second, Figure 9 after adding the business logic: processed at 1 frame per second, and then adding the correction rule and smoothing processing; the ordinate: represents the value. From Figure 8 and 9It can be seen that on the sleep posture classification result labels, the results before processing show more discrete states and irregular changes, and the classification is not clear enough. After processing, the classification result labels change more coherently and reasonably, and can classify the sleep postures more accurately, indicating that the present application significantly improves the accuracy and reliability of the sleep posture recognition prediction results through rule correction and subsequent smoothing processing.

[0125] In the embodiment of the present application, by acquiring the current frame image, determining the foreground detection frame, and correcting the initial sleep posture recognition result based on this, the accuracy of recognition is effectively improved. When facing complex scenes, comprehensive judgment is carried out using multi-dimensional information such as the aspect ratio, area, and foreground pixel ratio of the foreground detection frame, which can better cope with interference factors such as noise, body movement, and missing sleep posture imaging. At the same time, smoothing processing is performed on the corrected result, further improving the stability of the recognition result. It can provide more comprehensive and accurate reliable sleep posture data support for sleep health monitoring.

[0126] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0127] In one embodiment, a sleep posture recognition device is provided, and this sleep posture recognition device corresponds one-to-one with the sleep posture recognition method in the above embodiment. As Figure 10 shown, this sleep posture recognition device includes a current frame pressure image acquisition unit 10, a foreground detection frame determination unit 20, a prediction unit 30, a recognition result correction unit 40, and a recognition result smoothing unit 50. The detailed description of each functional module is as follows:

[0128] The current frame pressure image acquisition unit 10 is used to acquire the current frame pressure image;

[0129] The foreground detection frame determination unit 20 is used to determine the foreground detection frame corresponding to the foreground area in the current frame pressure image;

[0130] The prediction unit 30 is used to predict the initial sleep posture recognition result based on the current frame pressure image;

[0131] The recognition result correction unit 40 is used to correct the initial sleep posture recognition result based on the foreground detection frame;

[0132] The recognition result smoothing unit 50 is used to perform smoothing processing on the corrected initial sleep posture recognition result to obtain the target sleep posture recognition result.

[0133] In an embodiment of the present application, the recognition result correction unit 40 is further used for:

[0134] Determine the aspect ratio of the foreground detection frame and the area ratio of the area of the foreground detection frame to the area of the current frame pressure image;

[0135] Determine the foreground pixel ratio of the foreground detection frame in the current frame pressure image;

[0136] Respectively determine whether the aspect ratio, area ratio and foreground pixel ratio of the foreground detection frame meet the preset correction conditions;

[0137] If it meets the preset correction conditions, correct the initial sleep posture recognition result according to the preset correction rules.

[0138] In an embodiment of the present application, the recognition result correction unit 40 is further configured to:

[0139] Determine whether the foreground pixel ratio is less than a first preset threshold;

[0140] Determine whether the aspect ratio of the foreground detection frame is greater than a second preset threshold;

[0141] Determine whether the area ratio of the area of the foreground detection frame to the area of the current frame pressure image is less than a third preset threshold.

[0142] In an embodiment of the present application, the recognition result correction unit 40 is further configured to:

[0143] If it does not meet the preset correction conditions, determine the sleep posture category with the highest confidence in the initial sleep posture recognition result;

[0144] If the sleep posture category with the highest confidence is an unknown category and the confidence of the unknown category is less than the preset confidence threshold, correct the initial sleep posture recognition result to the sleep posture category with the second highest confidence, otherwise, correct the initial sleep posture recognition result to the sleep posture category with the highest confidence.

[0145] In an embodiment of the present application, the recognition result smoothing unit 50 is further configured to:

[0146] Save the initial sleep posture recognition result corresponding to one frame of image at every preset time interval;

[0147] Count the number of each sleep posture category in all initial sleep posture recognition results;

[0148] If the number of types of sleep posture categories is less than the preset number threshold, determine the sleep posture category with the largest number as the target sleep posture recognition result;

[0149] If the number of types of sleep posture categories is greater than or equal to the preset number threshold, set the target sleep posture recognition result as an unknown category.

[0150] In an embodiment of the present application, the recognition result smoothing unit 50 is further configured to:

[0151] If the number of the initial sleeping posture recognition results is greater than a preset number threshold, clear the initial sleeping posture recognition result corresponding to the starting frame pressure image, and save the initial sleeping posture recognition result corresponding to the current frame pressure image.

[0152] In an embodiment of the present application, the prediction unit 30 is further configured to:

[0153] Convert the current frame pressure image into a first grayscale image with a first preset size;

[0154] Input the grayscale image into a pre-trained sleeping posture recognition model for prediction processing to obtain the initial sleeping posture recognition result of the target user;

[0155] Wherein, the sleeping posture recognition model includes a first convolutional layer, a first rectified linear unit, a second convolutional layer, a second rectified linear unit, a pooling layer, and a fully connected layer connected in sequence.

[0156] In an embodiment of the present application, it further includes a heat map display unit, which is used to:

[0157] Convert the current frame pressure image into a second grayscale image with a second preset size;

[0158] Convert the second grayscale image into a heat map, and the display area of the heat map is used to display the target sleeping posture recognition result.

[0159] In the embodiment of the present application, by acquiring the current frame image, determining the foreground detection frame, and correcting the initial sleeping posture recognition result based on this, the recognition accuracy is effectively improved. When facing complex scenes, using multi-dimensional information such as the aspect ratio, area, and foreground pixel ratio of the foreground detection frame for comprehensive judgment can better cope with interference factors such as noise, body movement, and missing sleeping posture imaging. At the same time, smoothing the corrected result further improves the stability of the recognition result. It can provide more comprehensive and accurate reliable sleeping posture data support for sleep health monitoring.

[0160] For the specific limitations on the sleeping posture recognition device, reference can be made to the limitations on the sleeping posture recognition method in the above text, which will not be elaborated here. Each module in the above sleeping posture recognition device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0161] In one embodiment, a computer device is provided. The computer device can be a terminal device, and its internal structure diagram can be asFigure 11 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium. The readable storage medium stores computer-readable instructions. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer-readable instructions are executed by the processor, a sleeping posture recognition method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0162] In an embodiment of the present application, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the steps of the sleeping posture recognition method as described above are implemented.

[0163] In an embodiment of the application, a readable storage medium is provided. The readable storage medium stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps of the sleeping posture recognition method as described above are implemented.

[0164] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or an external cache. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0165] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0166] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A sleeping posture recognition method, characterized in that, The method includes: Obtain a current frame pressure image, where the current frame pressure image includes at least one target user; Determine a foreground detection box corresponding to the foreground area in the current frame pressure image; Based on the current frame pressure image, predict an initial sleeping posture recognition result of the target user; Based on the foreground detection box, correct the initial sleeping posture recognition result; Perform smoothing processing on the corrected initial sleeping posture recognition result to obtain the target sleeping posture recognition result of the target user.

2. The sleeping posture recognition method according to claim 1, characterized in that, The correcting the initial sleeping posture recognition result based on the foreground detection box includes: Determine the aspect ratio of the foreground detection box and the area ratio of the area of the foreground detection box to the area of the current frame pressure image; Determine the foreground pixel ratio of the foreground detection box in the current frame pressure image; Respectively determine whether the aspect ratio, area ratio, and foreground pixel ratio of the foreground detection box meet preset correction conditions; If the preset correction conditions are met, correct the initial sleeping posture recognition result according to the preset correction rules.

3. The sleep posture recognition method according to claim 2, characterized in that, The respectively determining whether the aspect ratio, area ratio, and foreground pixel ratio of the foreground detection box meet the preset correction conditions includes: Determine whether the foreground pixel ratio is less than a first preset threshold; Determine whether the aspect ratio of the foreground detection box is greater than a second preset threshold; Determine whether the area ratio of the area of the foreground detection box to the area of the current frame pressure image is less than a third preset threshold.

4. The sleeping posture recognition method according to claim 2, characterized in that After respectively determining whether the aspect ratio, area ratio, and foreground pixel ratio of the foreground detection box meet the preset correction conditions, it further includes: If the preset correction conditions are not met, determine the sleeping posture category with the highest confidence in the initial sleeping posture recognition result; If the sleeping posture category with the highest confidence is an unknown category and the confidence of the unknown category is less than a preset confidence threshold, correct the initial sleeping posture recognition result to the sleeping posture category with the second highest confidence, otherwise, correct the initial sleeping posture recognition result to the sleeping posture category with the highest confidence.

5. The sleep posture recognition method according to claim 1, characterized in that, The performing smoothing processing on the corrected initial sleeping posture recognition result to obtain the target sleeping posture recognition result includes: Save the initial sleeping posture recognition result corresponding to one frame of image at every preset time interval; Count the number of each sleeping posture category in all the initial sleeping posture recognition results; If the number of types of sleeping posture categories is less than a preset number threshold, determine the sleeping posture category with the largest number as the target sleeping posture recognition result; If the number of types of sleeping posture categories is greater than or equal to the preset number threshold, set the target sleeping posture recognition result as an unknown category.

6. The sleep posture recognition method according to claim 5, characterized in that, Before counting the number of each sleeping posture category in all the initial sleeping posture recognition results within a preset time range, it further includes: If the number of the initial sleeping posture recognition results is greater than a preset number threshold, clear the initial sleeping posture recognition result corresponding to the starting frame pressure image and save the initial sleeping posture recognition result corresponding to the current frame pressure image.

7. The sleeping posture recognition method according to claim 1, wherein The predicting the initial sleeping posture recognition result of the target user based on the current frame pressure image includes: Convert the current frame pressure image into a first grayscale image of a first preset size; Input the grayscale image into a pre-trained sleeping posture recognition model for prediction processing to obtain the initial sleeping posture recognition result of the target user; Among them, the sleeping posture recognition model includes a first convolutional layer, a first activation function, a second convolutional layer, a second activation function, a pooling layer, and a fully connected layer connected in sequence.

8. The sleeping posture recognition method according to any one of claims 1-7, characterized in that, After obtaining the current frame pressure image, it further includes: Convert the current frame pressure image into a second grayscale image of a second preset size; Convert the second grayscale image into a heat map, and the display area of the heat map is used to display the target sleeping posture recognition result.

9. A sleeping posture recognition device, characterized in that, The device includes: A current frame pressure image acquisition unit for acquiring a current frame pressure image; A foreground detection box determination unit for determining a foreground detection box corresponding to the foreground area in the current frame pressure image; A prediction unit for predicting an initial sleeping posture recognition result based on the current frame pressure image; An identification result correction unit for correcting the initial sleeping posture recognition result based on the foreground detection box; An identification result smoothing unit for smoothing the corrected initial sleeping posture recognition result to obtain a target sleeping posture recognition result.

10. A readable storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by a processor, the steps of the sleeping posture recognition method according to any one of claims 1 to 8 are implemented.

Images