Sleep monitoring method and device, computer equipment and storage medium

Abstract

The invention discloses a sleep monitoring method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a frame of pressure diagram of a target user at preset time intervals in a sleep cycle; determining whether the pressure diagram is a first frame pressure diagram; if the pressure diagram is the first frame pressure diagram, respectively taking the pressure diagram as a current frame pressure diagram and a reference frame pressure diagram; based on the current frame pressure diagram and the reference frame pressure diagram, predicting a sleeping posture and a body movement level of the target user through a pre-trained prediction model; and counting the duration of each sleeping posture in the sleeping cycle, the body movement duration and the body movement level corresponding to each sleeping posture change so as to evaluate the sleeping quality of the target user. By constructing a multi-task learning framework, the current frame pressure diagram and the reference frame pressure diagram are predicted, the sleeping posture and the body movement level can be predicted at the same time, the data utilization rate and monitoring comprehensiveness are improved, and generalization and accuracy of the model are improved.

Description

Technical Field

[0001] The present application relates to the field of smart home, and in particular to a sleep monitoring method, device, computer equipment and storage medium. Background Art

[0002] In sleep monitoring technology, sleeping posture recognition is an important means to evaluate sleep quality and prevent sleep-related diseases. Traditional sleeping posture monitoring methods mostly rely on video monitoring or acceleration sensors, which have problems such as high cost, high risk of privacy leakage, and recognition accuracy affected by the environment. With the development of sensor technology and artificial intelligence algorithms, sleep monitoring technology based on pressure distribution maps has gradually attracted attention. By identifying human pressure data, analyzing the effective imaging area of ​​the pressure value, dividing the human body contour area on the mattress, and then analyzing the human body contour area and the patient's personalized sleeping posture database, the sleeping posture state is determined, thereby achieving an assessment of sleep quality.

[0003] However, most existing pressure image-based methods focus on a single sleeping posture classification task, and fail to fully utilize the advantages of deep learning multi-task learning to improve overall recognition efficiency and accuracy. A single sleeping posture cannot fully evaluate sleep quality from the sleeping posture dimension. In addition, the rich information in these data is not fully utilized, resulting in limited generalization ability, difficulty in adapting to changes in the sleeping environment, and low accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a sleep monitoring method, device, computer equipment and storage medium to address the above technical issues, so as to solve at least one problem existing in the above-mentioned prior art.

[0005] In a first aspect, the embodiment of the present application is implemented as follows: a sleep monitoring method is provided, comprising the following steps:

[0006] During the sleep cycle, a frame of the target user's stress map is obtained at each preset time interval;

[0007] Determine whether the pressure map is a first frame pressure map;

[0008] If the pressure map is a first frame pressure map, the pressure map is used as a current frame pressure map and a reference frame pressure map respectively;

[0009] Based on the current frame pressure map and the reference frame pressure map, the sleeping posture and body movement level of the target user are predicted by a pre-trained prediction model;

[0010] The duration of each sleeping posture, the duration of body movement, and the body movement level corresponding to each sleeping posture change during the sleep cycle are counted, so as to evaluate the sleep quality of the target user based on the duration of sleeping posture, the duration of body movement, and the body movement level corresponding to each sleeping posture change.

[0011] In one embodiment, after determining whether the pressure map is a first frame pressure map, the method further includes:

[0012] If the pressure map is not the first frame pressure map, the pressure map is used as the current frame pressure map, and the previous frame pressure map of the pressure map is used as the reference frame pressure map.

[0013] In one embodiment, based on the current frame pressure map and the reference frame pressure map, the sleeping posture and body movement level of the target user are predicted by a pre-trained prediction model, including:

[0014] Performing feature extraction on the current frame pressure map and the reference frame pressure map respectively to obtain a current frame feature map and a reference frame feature map;

[0015] Performing differential processing on the current frame feature map and the reference frame feature map to obtain a differential feature map;

[0016] Splicing the current frame feature map with the differential feature map to obtain a spliced ​​feature map;

[0017] Based on the current frame feature map and the spliced ​​feature map, the sleeping posture and body movement level of the target user are predicted.

[0018] In one embodiment, the predicting of the sleeping posture and body movement level of the target user based on the current frame feature map and the spliced ​​feature map includes:

[0019] Performing average pooling processing on the current frame feature map, and obtaining the sleeping posture of the target user based on the current frame features after the average pooling processing and a preset activation function; and

[0020] Average pooling is performed on the spliced ​​feature map, and the body movement level of the target user is obtained based on the spliced ​​feature map after the average pooling process.

[0021] In one embodiment, after acquiring a frame of the target user's stress map at each preset time interval during the sleep cycle, the method further includes:

[0022] Performing noise reduction and data enhancement processing on the pressure map;

[0023] The pressure map after noise reduction and data enhancement is converted into a grayscale sleeping posture map, wherein the pixel value of the grayscale sleeping posture map is within a preset interval, so that the grayscale sleeping posture map is predicted by the pre-trained prediction model.

[0024] In one embodiment, the pre-trained prediction model includes a differential network, the differential network includes an input layer, a backbone network and an output layer connected in sequence, the backbone network includes a convolutional layer, a first pooling layer, a first residual module, a second residual module, a second pooling layer, a third residual module and a fourth residual module connected in sequence, the output layer includes a first output head and a second output head, the first output head is used to output the sleeping posture of the target user, and the second output head is used to output the body movement level of the target user.

[0025] In one embodiment, the pre-trained prediction model is trained in the following manner:

[0026] Step a: obtaining training sample data, wherein the training sample data is annotated with labels;

[0027] Step b: In the training sample data, a frame of standard sleeping posture picture is selected as a reference frame picture, and a frame of sleeping posture picture with body movement changes is selected as a current frame picture;

[0028] Step c: inputting the reference frame image and the current frame image into an initial prediction model for prediction processing to obtain a sleeping posture category and a body movement level value;

[0029] Step d: Based on a preset loss function and the label, respectively calculating the loss values ​​corresponding to the sleeping posture category and the body movement level value;

[0030] When the loss value is greater than a preset loss threshold, the initial prediction model is updated and the above steps b to d are repeated until the preset convergence condition is met to obtain the pre-trained prediction model.

[0031] In a second aspect, a sleep monitoring device is provided, comprising:

[0032] A pressure map acquisition unit, used to acquire a frame of the target user's pressure map at each preset time interval during a sleep cycle;

[0033] A first frame pressure map determining unit, used to determine whether the pressure map is a first frame pressure map;

[0034] An input picture construction unit is used for, if the pressure map is a first frame pressure map, using the pressure map as a current frame pressure map and a reference frame pressure map respectively;

[0035] A prediction unit, configured to predict the sleeping posture and body movement level of the target user through a pre-trained prediction model based on the current frame pressure map and the reference frame pressure map;

[0036] The sleep quality assessment unit is used to count the duration of each sleeping posture, the duration of body movement, and the body movement level corresponding to each change of sleeping posture during the sleep cycle, so as to assess the sleep quality of the target user based on the duration of sleeping posture, the duration of body movement, and the body movement level corresponding to each change of sleeping posture.

[0037] In one embodiment, a computer device includes a memory, a processor, and computer-readable instructions stored in the memory and running on the processor, and the processor implements the sleep monitoring method as described above when executing the computer-readable instructions.

[0038] In one embodiment, a readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the sleep monitoring method described above is implemented.

[0039] The above-mentioned sleep monitoring method, device, computer equipment and storage medium, and its implementation method include: obtaining a frame of pressure map of the target user at every preset time interval during the sleep cycle; determining whether the pressure map is the first frame pressure map; if the pressure map is the first frame pressure map, using the pressure map as the current frame pressure map and the reference frame pressure map respectively; based on the current frame pressure map and the reference frame pressure map, predicting the sleeping posture and body movement level of the target user through a pre-trained prediction model; counting the duration of each sleeping posture, the duration of body movement, and the body movement level corresponding to each change in sleeping posture during the sleep cycle, so as to evaluate the sleep quality of the target user based on the duration of sleeping posture, the duration of body movement, and the body movement level corresponding to each change in sleeping posture. In an embodiment of the present application, a pressure map of the user is collected once at every interval during the sleep cycle, and each collected pressure map is combined with the reference frame pressure map to obtain the target user's sleep quality. Figure 1 Through the multi-task learning framework, the sleeping posture and body movement level can be predicted simultaneously, which improves data utilization and monitoring comprehensiveness, and improves the generalization and accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 is a flowchart of a sleep monitoring method in one embodiment of the present application;

[0042] Figure 2 is a flow chart of a method for predicting sleeping posture and body movement level in one embodiment of the present application;

[0043] Figure 3 is a network architecture diagram of a differential network in an embodiment of the present application;

[0044] Figure 4 is a structural schematic diagram of a sleep monitoring device in one embodiment of the present application;

[0045] Figure 5 It is a schematic diagram of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] In one embodiment, if Figure 1 As shown, a sleep monitoring method is provided, comprising the following steps:

[0048] In step S110, during the sleep cycle, a frame of the target user's pressure map is acquired at every preset time interval;

[0049] In the embodiment of the present application, a pressure detection device can be set on an object for users to rest, for example, a pressure blanket can be set on a smart mattress, sofa, bed or other equipment for users to lie down, or multiple pressure detection devices arranged in an array can be set. The pressure detection device can be arranged in an array equidistantly in the longitudinal and transverse directions. When the user is on the device, the pressure detection device can collect the pressure map data, analyze the effective imaging area of ​​the pressure value, and divide the human body contour area on the mattress to obtain the pressure map. It should be noted that the target user can include at least one user, that is, the sleep monitoring of one user can be performed, and the sleep monitoring of multiple users can be performed at the same time, that is, the pressure map can include only the sleeping posture of one user, or the sleeping postures of multiple users at the same time, so as to realize the monitoring of the sleep quality of multiple users at the same time.

[0050] Among them, the pressure detection device can be a pressure sensor, a piezoelectric film sensor, etc.

[0051] Optionally, the target user's pressure map may be collected once at a preset time interval, such as 1 second, 2 seconds, etc., and one frame or multiple frames of pressure maps may be collected each time. If multiple frames of pressure maps are collected, the pressure map with the highest definition may be selected for subsequent prediction.

[0052] In step S120, determining whether the pressure map is a first frame pressure map;

[0053] Optionally, each frame of the pressure map may be marked with an acquisition time or an acquisition number, and the number may be used to characterize the order in which the pressure map acquisition time is obtained. Therefore, after the pressure map is obtained, it may be determined whether it is the first frame of the pressure map in this sleep cycle based on the acquisition time or number marked thereon. Alternatively, each time a pressure map is obtained, it may be stored. After the pressure map is obtained, it may be determined whether other pressure maps are obtained in this sleep cycle. If not, the pressure map may be considered to be the first frame of the pressure map in this sleep cycle.

[0054] In step S130, if the pressure map is a first frame pressure map, the pressure map is used as a current frame pressure map and a reference frame pressure map respectively;

[0055] It can be understood that the current frame pressure map refers to the latest pressure map currently collected. The reference frame pressure map is used to compare with the current frame pressure map to determine the difference between the current frame pressure map and the previous frame pressure map, so as to determine the pressure map of the target user's sleeping posture change.

[0056] In an embodiment of the present application, if the pressure map is a first frame pressure map, the pressure map is used as a current frame pressure map and a reference frame pressure map, respectively, and is input into a pre-trained prediction model for prediction processing.

[0057] like Figure 1 As shown, in step S160, if the pressure map is not the first frame pressure map, the pressure map can be used as the current frame pressure map, and the previous frame pressure map of the pressure map can be used as the reference frame pressure map, and input into the pre-trained prediction model for prediction processing.

[0058] The current frame pressure map and the reference frame pressure map can both be two-dimensional grayscale images.

[0059] In step S140, based on the current frame pressure map and the reference frame pressure map, the sleeping posture and body movement level of the target user are predicted by a pre-trained prediction model;

[0060] Optionally, the pre-trained prediction model can be a deep learning model, which includes a differential network, and the differential network includes an input layer, a backbone network and an output layer connected in sequence. The current frame pressure map and the reference frame pressure map can enter the backbone network through the input layer for feature extraction to obtain the current frame feature map and the reference frame feature map. Then, the current frame feature map and the reference frame feature map can be differentially processed, and then feature spliced ​​with the current frame feature map, and the body movement level is output through the fully connected layer. The current frame feature map can obtain the sleeping posture category after being processed by the global connection layer and the activation function. By adopting a multi-task neural network framework, it is possible to monitor sleeping posture and body movement at the same time, which improves the utilization rate of data and the comprehensiveness of monitoring. By adding a differential network to the deep learning model, the weight of the difference feature in the feature map can be increased, and weak body movements can be more accurately monitored, thereby improving recognition accuracy.

[0061] It should be noted that the sleeping position categories may include supine, prone, left side, left curled up, right side, right curled up, etc. Body movement level: It can be divided into 5 levels 0-4, where level 0 represents the definition of six standard sleeping positions: supine, prone, left side, left curled up, right side, and right curled up; level 1 represents that the body does not move, only the head, hands, and feet move slightly; level 2 represents that the body does not move, and the head, arms, hands, legs and feet move slightly; level 3 represents that the body moves slightly, and the head, arms, hands, legs and feet move slightly; level 4 represents that the body moves slightly, and the head, arms, hands, legs and feet move more significantly.

[0062] In step S150, the duration of each sleeping posture, the duration of body movement, and the body movement level corresponding to each change of sleeping posture during the sleep cycle are counted, so as to evaluate the sleep quality of the target user based on the duration of sleeping posture, the duration of body movement, and the body movement level corresponding to each change of sleeping posture.

[0063] In an embodiment of the present application, the duration of each sleeping position, the duration of body movement, and the level of body movement each time the sleeping position changes during a sleep cycle can be counted. The user's sleep quality can be determined by analyzing the duration of sleeping position, the duration of body movement, and the level of body movement each time the sleeping position changes.

[0064] Exemplarily, sleeping posture duration = number of times sleeping posture is identified * 1 second, body movement duration = number of times body movement is identified * 1 second, then the deep sleep duration = total sleep duration - total sleeping posture duration - total movement duration is roughly estimated. Then, according to the deep sleep duration, it can be divided into three sleep levels: > 4 hours is very good, 2 hours < deep sleep duration < 4 hours is good, and < 2 hours is general, so that the sleep quality of the target user can be determined. Then, the sleep quality of the target user can be further determined according to the corresponding body movement level each time the sleeping posture changes. For example, if the body movement level of the target user is 4 each time the sleeping posture changes and the deep sleep duration is less than 2 hours, the target user's sleep quality is considered to be poor. If the body movement level of each sleeping posture change of the target user is 1, and the deep sleep duration is > 4 hours, the sleep quality is considered to be very good. If the body movement level of each sleeping posture change of the target user is 2, and 2 hours < deep sleep duration < 4 hours, the sleep quality is considered to be general.

[0065] Among them, the user's sleeping posture in the adjacent frame pressure map can be matched for similarity. If they are the same, it means that the sleeping posture has not changed. If they are different, it means that the sleeping posture has changed, so that the duration of the sleeping posture can be determined as the sleeping posture duration. For the body movement duration, it can be determined according to the changes in the user's movements in the adjacent frame pressure map. Or it can also be determined according to the body movement level. For example, if the body movement level is large, the body movement duration is long. If the body movement level is small, the body movement duration is short. For example, if the body movement level is 4, the body movement duration is 2 seconds. If the body movement level is 1, the body movement duration is 1 second.

[0066] It should be noted that the sleep level can be determined by the duration of sleeping posture, the duration of body movement, and the level of body movement each time the sleeping posture changes. Then the user's physiological characteristics such as breathing rate and heart rate can be obtained to comprehensively evaluate the overall sleep quality of the target user and generate a sleep quality report. Exemplarily, each feature can be assigned a corresponding weight, and all weights and values ​​are 1. For example, sleep quality = deep sleep duration * a + heart rate * b + breathing rate * c, where a, b, c are weights, and a + b + c = 1. When the sleep quality is greater than a preset threshold, for example 0.8, it means that the sleep quality is good.

[0067] An embodiment of the present application provides a sleep monitoring method, including: obtaining a pressure map of a target user at a preset time interval during a sleep cycle; determining whether the pressure map is the first frame pressure map; if the pressure map is the first frame pressure map, using the pressure map as the current frame pressure map and the reference frame pressure map respectively; based on the current frame pressure map and the reference frame pressure map, predicting the sleeping position and body movement level of the target user through a pre-trained prediction model; counting the duration of each sleeping position, the duration of body movement, and the body movement level corresponding to each change in sleeping position during the sleep cycle, so as to evaluate the sleep quality of the target user based on the duration of sleeping position, the duration of body movement, and the body movement level corresponding to each change in sleeping position. In an embodiment of the present application, a pressure map of the user is collected at a preset time interval during a sleep cycle, and each collected pressure map is combined with the reference frame pressure map to obtain a sleep quality evaluation result. Figure 1 Through the multi-task learning framework, the sleeping posture and body movement level can be predicted simultaneously, which improves data utilization and monitoring comprehensiveness, and improves the generalization and accuracy of the model.

[0068] In one embodiment of the present application, based on the current frame pressure map and the reference frame pressure map, the sleeping posture and body movement level of the target user are predicted by a pre-trained prediction model, including:

[0069] Performing feature extraction on the current frame pressure map and the reference frame pressure map respectively to obtain a current frame feature map and a reference frame feature map;

[0070] Performing differential processing on the current frame feature map and the reference frame feature map to obtain a differential feature map;

[0071] Splicing the current frame feature map with the differential feature map to obtain a spliced ​​feature map;

[0072] Based on the current frame feature map and the spliced ​​feature map, the sleeping posture and body movement level of the target user are predicted.

[0073] See also Figure 2, the pre-trained prediction model includes a differential network, and the differential network includes an input layer, a backbone network and an output layer connected in sequence. The backbone network includes a convolutional layer, a first pooling layer, a first residual module, a second residual module, a second pooling layer, a third residual module and a fourth residual module connected in sequence. The output layer includes a first output head and a second output head, the first output head is used to output the sleeping posture of the target user, and the second output head is used to output the body movement level of the target user. The current frame pressure map and the reference frame pressure map are respectively feature extracted by the backbone network to obtain a current frame feature map and a reference frame feature map, and then the current frame feature map and the reference frame feature map are differentially processed to obtain a differential feature map, and the current frame feature map and the differential feature map are spliced ​​to obtain a spliced ​​feature map, and the body movement level is predicted based on the spliced ​​feature map. Based on the current frame feature map, the sleeping posture category of the target user is predicted.

[0074] The backbone network may include a first backbone network and a second backbone network, wherein the first backbone network is used to perform feature extraction on a current frame pressure map to obtain a current frame feature map, and the second backbone network is used to perform feature extraction on a reference frame pressure map to obtain a reference frame feature map.

[0075] See also Figure 3, combined with the specific architecture of the above prediction model, the prediction process is as follows: Input layer: two 64*16 two-dimensional grayscale images, namely the reference frame pressure map and the current frame pressure map, need to be input at the same time. Backbone network: The first layer of convolution conv, consists of 32 convolution kernels of size 3*3, with a step of 2 and padding of 1, to achieve downsampling and generate a feature map size of 32*8*32; the second layer of maximum pooling maxpool, with a size of 3*3, a step of 1, padding 1, and keeping the feature map size unchanged at 32*8*32; the third layer of residual block 1 (resnet block1), consists of two layers of 64 convolution kernels of size 3*3, with a step of 1 and padding 1, to generate a feature map size of 32*8*64; the fourth layer of residual block 2 (resnet The first layer is the resnet block2, which consists of two layers of 64 convolution kernels of size 3*3, with a step of 1 and padding of 1, and the generated feature map size is 32*8*64; the fifth layer is the maximum pooling maxpool, with a size of 3*3, a step of 2, and padding of 1, which realizes downsampling, and the feature map size remains unchanged at 16*4*64; the sixth layer is the residual block 3 (resnetblock3), which consists of two layers of 128 convolution kernels of size 3*3, with a step of 1 and padding of 1, and the generated feature map size is 16*4*128; the seventh layer is the residual block 4 (resnet block4), which consists of two layers of 128 convolution kernels of size 3*3, with a step of 1 and padding of 1, and the size of the output reference frame feature map and the current frame feature map is 16*4*128. Then through the output layer, the sleeping posture category and body movement level can be output.

[0076] In one embodiment of the present application, the step of predicting the sleeping posture and body movement level of the target user based on the current frame feature map and the spliced ​​feature map includes:

[0077] Performing average pooling processing on the current frame feature map, and obtaining the sleeping posture of the target user based on the current frame features after the average pooling processing and a preset activation function; and

[0078] Average pooling is performed on the spliced ​​feature map, and the body movement level of the target user is obtained based on the spliced ​​feature map after the average pooling process.

[0079] See also Figure 3, optionally, the output layer may include a first output head and a second output head. The first output head can be used to output the sleeping posture category, which includes average pooling (average pool), size 3*3, step 1, padding 1, and feature map size 16*4*128; fully connected layer (FC), composed of 6 1*8192 vectors, the output regression vector size is 1*6; finally, it is classified by a preset activation function, such as a normalized exponential function (softmax), and the confidence probability vector 1*6 is output. The second output head is used for body movement level regression, wherein the differential part uses the feature map of the reference frame and the current frame extracted by the backbone network to calculate the pixel-level difference between the feature map of the current frame and the reference frame to obtain a differential feature map, whose size is 16*4*128. The feature map of the current frame is concatenated with the differential feature map to obtain a concatenated feature map, whose size is 16*4*256. Then, the concatenated feature map is average pooled with a size of 3*3, a step of 1, and a padding of 1 to obtain a feature map with a size of 16*4*256; finally, the regression vector size is output through the fully connected layer (FC) with a size of 1*5, and the fully connected layer can be composed of 5 1*16384 vectors.

[0080] It should be noted that after the reference frame pressure map and the current frame pressure map are subjected to feature extraction through the backbone network, two reference frame feature maps and current frame feature maps with dimensions of 16*4*128 can be generated. Then, pixel-by-pixel subtraction can be performed, that is, the pixel values ​​of the reference frame feature map and the current frame feature map at the same coordinate position can be subtracted one by one, and finally a differential feature map with the same dimension of 16*4*128 can be obtained. The dimension of the differential feature map is 16*4*128, and the dimension of the current frame feature map is 16*4*128. Then, the size of the two differential feature maps and the current frame feature map can be kept unchanged at 16*4, and they can be spliced ​​front and back on the 128-dimensional channel to generate a spliced ​​feature map with a dimension of 16*4*(128+128)=16*4*256.

[0081] In one embodiment of the present application, the pre-trained prediction model can be trained in the following manner:

[0082] Step a: obtaining training sample data, wherein the training sample data is annotated with labels;

[0083] Step b: In the training sample data, a frame of standard sleeping posture picture is selected as a reference frame picture, and a frame of sleeping posture picture with body movement changes is selected as a current frame picture;

[0084] Step c: inputting the reference frame image and the current frame image into an initial prediction model for prediction processing to obtain a sleeping posture category and a body movement level value;

[0085] Step d: Based on a preset loss function and the label, respectively calculating the loss values ​​corresponding to the sleeping posture category and the body movement level value;

[0086] When the loss value is greater than a preset loss threshold, the initial prediction model is updated and the above steps b to d are repeated until the preset convergence condition is met to obtain the pre-trained prediction model.

[0087] Optionally, the pressure distribution map data can be collected by pre-setting a pressure detection device on an object for the user to rest, such as a pressure blanket or multiple pressure detection devices arranged in an array, such as a pressure sensor, on a smart mattress, sofa, bed, or other equipment for the user to lie down. And after pre-processing such as noise reduction and enhancement, the pressure distribution map data is converted into a grayscale sleeping posture map with a pixel value between 0-255. The grayscale sleeping posture map may include standard sleeping postures such as supine, prone, left side lying, left curled up, right side lying, and right curled up. And sleeping postures with small body movements. For example, in the case of a standard sleeping posture, the arms, head, legs, feet, etc., have a certain degree of movement and change. Therefore, each sleeping posture map can be manually labeled with two labels, namely, a sleeping posture category label and a body movement amplitude coefficient label. The body movement amplitude coefficient is divided into levels of 0-4 according to the body movement amplitude. The larger the value, the greater the body movement. For example, if a picture is marked as (supine, 0), it means that the picture is a standard sleeping posture with no body movement. If the picture is marked as (supine, 4), it means that the body movement is the maximum body movement within the defined range and the level is 4. In this way, a training sample data set can be constructed.

[0088] In the labeled training sample data, a standard sleeping position picture is randomly selected as a reference frame pressure map and a sleeping position picture with body movement is randomly selected as a current frame pressure map. For example, one picture with a label of prone and body movement 0 is used as a reference frame, and another picture with prone and body movement 1 is used as a current frame. The initial prediction model is input, and the model is used for prediction processing to output the sleeping position category and body movement level respectively.

[0089] Then, for the sleeping posture category, a preset loss function, such as the cross entropy loss function, can be used, as shown below:

[0090]

[0091] Among them, p is the true value, q is the predicted value, and x i is a discrete random variable. Then the network model parameters are updated using stochastic gradient.

[0092] For the body motion level output by the model, a preset loss function can be used, for example, the L1 loss function, as shown below:

[0093]

[0094] Where N is the number of samples, y i is the true value of the i-th sample, is the predicted value of the i-th sample. Stochastic gradient is also used to update the network model parameters.

[0095] Based on the two loss values ​​calculated above, determine whether each loss value is greater than the corresponding preset loss threshold. If so, repeat the above steps and select the next training sample for prediction processing until the loss value is less than or equal to the preset loss threshold, or the number of iterations is greater than the preset number, or after all the training samples in the entire training data set are trained, output the final model as the trained prediction model.

[0096] In an embodiment of the present application, after obtaining a frame of the target user's stress map at each preset time interval during the sleep cycle, the method further includes:

[0097] Performing noise reduction and data enhancement processing on the pressure map;

[0098] The pressure map after noise reduction and data enhancement is converted into a grayscale sleeping posture map, wherein the pixel value of the grayscale sleeping posture map is within a preset interval, so that the grayscale sleeping posture map is predicted by the pre-trained prediction model.

[0099] Optionally, after obtaining the pressure map, a preset filtering algorithm, such as Gaussian filtering, may be used to filter the pressure map. Then, the pressure map may be enhanced by image rotation, translation and other enhancement operations. Then, the pixel values ​​of the pressure map after noise reduction and data enhancement are normalized to the range of 0-255, which is conducive to the convergence of the prediction model training.

[0100] In the embodiment of the present application, the pressure map of the user is collected at intervals during the sleep cycle, and each collected pressure map is combined with the reference frame pressure map. Figure 1 Through the multi-task learning framework, the sleeping posture and body movement level can be predicted simultaneously, which improves data utilization and monitoring comprehensiveness, and improves the generalization and accuracy of the model.

[0101] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0102] In one embodiment, a sleep monitoring device is provided, which corresponds one-to-one to the sleep monitoring method in the above embodiment. Figure 4As shown, the sleep monitoring device includes a pressure map acquisition unit 10, a first frame pressure map determination unit 20, an input image construction unit 30, a prediction unit 40 and a sleep quality assessment unit 50. Each functional module is described in detail as follows:

[0103] The pressure map acquisition unit 10 is used to acquire a frame of the pressure map of the target user at each preset time interval during the sleep cycle;

[0104] A first frame pressure map determining unit 20, used to determine whether the pressure map is a first frame pressure map;

[0105] An input picture construction unit 30 is used to use the pressure map as a current frame pressure map and a reference frame pressure map respectively if the pressure map is a first frame pressure map;

[0106] A prediction unit 40 is used to predict the sleeping posture and body movement level of the target user through a pre-trained prediction model based on the current frame pressure map and the reference frame pressure map;

[0107] The sleep quality evaluation unit 50 is used to count the duration of each sleeping posture, the duration of body movement, and the body movement level corresponding to each change of sleeping posture during the sleep cycle, so as to evaluate the sleep quality of the target user based on the duration of sleeping posture, the duration of body movement, and the body movement level corresponding to each change of sleeping posture.

[0108] In one embodiment of the present application, the input picture construction unit 30 is further used to:

[0109] If the pressure map is not the first frame pressure map, the pressure map is used as the current frame pressure map, and the previous frame pressure map of the pressure map is used as the reference frame pressure map.

[0110] In an embodiment of the present application, the prediction unit 40 is further configured to:

[0111] Performing feature extraction on the current frame pressure map and the reference frame pressure map respectively to obtain a current frame feature map and a reference frame feature map;

[0112] Performing differential processing on the current frame feature map and the reference frame feature map to obtain a differential feature map;

[0113] Splicing the current frame feature map with the differential feature map to obtain a spliced ​​feature map;

[0114] Based on the current frame feature map and the spliced ​​feature map, the sleeping posture and body movement level of the target user are predicted.

[0115] In an embodiment of the present application, the prediction unit 40 is further configured to:

[0116] Performing average pooling processing on the current frame feature map, and obtaining the sleeping posture of the target user based on the current frame features after the average pooling processing and a preset activation function; and

[0117] Average pooling is performed on the spliced ​​feature map, and the body movement level of the target user is obtained based on the spliced ​​feature map after the average pooling process.

[0118] In one embodiment of the present application, the device further includes a preprocessing unit configured to:

[0119] Performing noise reduction and data enhancement processing on the pressure map;

[0120] The pressure map after noise reduction and data enhancement is converted into a grayscale sleeping posture map, wherein the pixel value of the grayscale sleeping posture map is within a preset interval, so that the grayscale sleeping posture map is predicted by the pre-trained prediction model.

[0121] In one embodiment of the present application, the pre-trained prediction model includes a differential network, the differential network includes an input layer, a backbone network and an output layer connected in sequence, the backbone network includes a convolutional layer, a first pooling layer, a first residual module, a second residual module, a second pooling layer, a third residual module and a fourth residual module connected in sequence, the output layer includes a first output head and a second output head, the first output head is used to output the sleeping posture of the target user, and the second output head is used to output the body movement level of the target user.

[0122] In one embodiment of the present application, the pre-trained prediction model is trained in the following manner:

[0123] Step a: obtaining training sample data, wherein the training sample data is annotated with labels;

[0124] Step b: In the training sample data, a frame of standard sleeping posture picture is selected as a reference frame picture, and a frame of sleeping posture picture with body movement changes is selected as a current frame picture;

[0125] Step c: inputting the reference frame image and the current frame image into an initial prediction model for prediction processing to obtain a sleeping posture category and a body movement level value;

[0126] Step d: Based on a preset loss function and the label, respectively calculating the loss values ​​corresponding to the sleeping posture category and the body movement level value;

[0127] When the loss value is greater than a preset loss threshold, the initial prediction model is updated and the above steps b to d are repeated until the preset convergence condition is met to obtain the pre-trained prediction model.

[0128] In the embodiment of the present application, the pressure map of the user is collected at intervals during the sleep cycle, and each collected pressure map is combined with the reference frame pressure map. Figure 1 Through the multi-task learning framework, the sleeping posture and body movement level can be predicted simultaneously, which improves data utilization and monitoring comprehensiveness, and improves the generalization and accuracy of the model.

[0129] For the specific definition of the sleep monitoring device, please refer to the definition of the sleep monitoring method above, which will not be repeated here. Each module in the above sleep monitoring device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0130] In one embodiment, a computer device is provided. The computer device may be a terminal device, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium. The readable storage medium stores computer-readable instructions. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instructions are executed by the processor, a sleep monitoring method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0131] In an embodiment of the present application, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, and when the processor executes the computer-readable instructions, the steps of the sleep monitoring method described above are implemented.

[0132] In an embodiment of the application, a readable storage medium is provided, wherein the readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the sleep monitoring method described above are implemented.

[0133] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0134] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0135] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A sleep monitoring method, characterized in that: The method comprises: During the sleep cycle, a frame of the target user's stress map is obtained at each preset time interval; Determine whether the pressure map is a first frame pressure map; If the pressure map is a first frame pressure map, the pressure map is used as a current frame pressure map and a reference frame pressure map respectively; Based on the current frame pressure map and the reference frame pressure map, the sleeping posture and body movement level of the target user are predicted by a pre-trained prediction model; The duration of each sleeping posture, the duration of body movement, and the body movement level corresponding to each sleeping posture change during the sleep cycle are counted, so as to evaluate the sleep quality of the target user based on the duration of sleeping posture, the duration of body movement, and the body movement level corresponding to each sleeping posture change.

2. The sleep monitoring method according to claim 1, characterized in that: After determining whether the pressure map is the first frame pressure map, the method further includes: If the pressure map is not the first frame pressure map, the pressure map is used as the current frame pressure map, and the previous frame pressure map of the pressure map is used as the reference frame pressure map.

3. The sleep monitoring method according to claim 1 or 2, characterized in that: The step of predicting the sleeping posture and body movement level of the target user based on the current frame pressure map and the reference frame pressure map by using a pre-trained prediction model includes: Performing feature extraction on the current frame pressure map and the reference frame pressure map respectively to obtain a current frame feature map and a reference frame feature map; Performing differential processing on the current frame feature map and the reference frame feature map to obtain a differential feature map; Splicing the current frame feature map with the differential feature map to obtain a spliced ​​feature map; Based on the current frame feature map and the spliced ​​feature map, the sleeping posture and body movement level of the target user are predicted.

4. The sleep monitoring method according to claim 3, characterized in that: The predicting the sleeping posture and body movement level of the target user based on the current frame feature map and the spliced ​​feature map includes: Performing average pooling processing on the current frame feature map, and obtaining the sleeping posture of the target user based on the current frame feature map after the average pooling processing and a preset activation function; and Average pooling is performed on the spliced ​​feature map, and the body movement level of the target user is obtained based on the spliced ​​feature map after the average pooling process.

5. The sleep monitoring method according to claim 1, characterized in that: After obtaining a frame of the target user's stress map at each preset time interval during the sleep cycle, the method further includes: Performing noise reduction and data enhancement processing on the pressure map; The pressure map after noise reduction and data enhancement processing is converted into a grayscale sleeping posture map, wherein the pixel value of the grayscale sleeping posture map is within a preset interval, so as to perform prediction processing on the grayscale sleeping posture map through the pre-trained prediction model.

6. The sleep monitoring method according to claim 1, characterized in that: The pre-trained prediction model includes a differential network, the differential network includes an input layer, a backbone network and an output layer connected in sequence, the backbone network includes a convolutional layer, a first pooling layer, a first residual module, a second residual module, a second pooling layer, a third residual module and a fourth residual module connected in sequence, the output layer includes a first output head and a second output head, the first output head is used to output the sleeping posture of the target user, and the second output head is used to output the body movement level of the target user.

7. The sleep monitoring method according to any one of claims 1 to 6, characterized in that: The pre-trained prediction model is trained in the following way: Step a: obtaining training sample data, wherein the training sample data is annotated with labels; Step b: In the training sample data, a frame of standard sleeping posture picture is selected as a reference frame picture, and a frame of sleeping posture picture with body movement changes is selected as a current frame picture; Step c: inputting the reference frame image and the current frame image into an initial prediction model for prediction processing to obtain a sleeping posture category and a body movement level value; Step d: Based on a preset loss function and the label, respectively calculating the loss values ​​corresponding to the sleeping posture category and the body movement level value; When the loss value is greater than a preset loss threshold, the initial prediction model is updated and the above steps b to d are repeated until the preset convergence condition is met to obtain the pre-trained prediction model.

8. A sleep monitoring device, characterized in that: The device comprises: A pressure map acquisition unit, used to acquire a frame of the target user's pressure map at each preset time interval during a sleep cycle; A first frame pressure map determining unit, used to determine whether the pressure map is a first frame pressure map; An input picture construction unit is used for, if the pressure map is a first frame pressure map, using the pressure map as a current frame pressure map and a reference frame pressure map respectively; A prediction unit, configured to predict the sleeping posture and body movement level of the target user through a pre-trained prediction model based on the current frame pressure map and the reference frame pressure map; The sleep quality assessment unit is used to count the duration of each sleeping posture, the duration of body movement, and the body movement level corresponding to each change of sleeping posture during the sleep cycle, so as to assess the sleep quality of the target user based on the duration of sleeping posture, the duration of body movement, and the body movement level corresponding to each change of sleeping posture.

9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executed on the processor, characterized in that: When the processor executes the computer-readable instructions, the sleep monitoring method according to any one of claims 1 to 7 is implemented.

10. A readable storage medium having computer readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the sleep monitoring method according to any one of claims 1 to 7 is implemented.

Images