A sleep quality evaluation method and device, electronic equipment and readable storage medium

Abstract

The application discloses a sleep quality evaluation method and device, electronic equipment and readable storage medium, and belongs to the technical field of sleep quality evaluation. The method comprises the following steps: obtaining a target image sequence and target audio data of a target user; performing facial expression feature extraction on the target image sequence to obtain a facial expression feature sequence, and performing text feature extraction on the target audio data to obtain a first text feature; inputting the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep evaluation result output by the target neural network model. The facial expression feature and the text feature can be analyzed through the neural network model, and a sleep quality evaluation result with high accuracy can be output.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of sleep quality evaluation, and particularly relates to a sleep quality evaluation method and device, an electronic device and a readable storage medium. BACKGROUND

[0002] Sleep plays a vital role for the human body, and problems in sleep will affect the quality of people's life. In recent years, with the continuous improvement of people's living standards, people pay more and more attention to their sleep quality.

[0003] In the related art, physiological indexes (such as heart rate) and motion data of a user are usually collected by a wearable device, and the physiological indexes and the motion data are subjected to rule judgment, for example, if the heart rate of the user is lower than 60 and there is no motion within 60 seconds, it is judged that the user is in a sleep state. In combination with sleep data such as the highest heart rate and motion frequency of the user during sleep, a sleep quality score of the user is calculated.

[0004] However, the accuracy of the above sleep quality evaluation method is poor. SUMMARY

[0005] The embodiments of the present application provide a sleep quality evaluation method, device, electronic device and readable storage medium, which can solve the problem of poor accuracy of the sleep quality evaluation method in the related art.

[0006] In a first aspect, the embodiments of the present application provide a sleep quality evaluation method, which comprises:

[0007] obtaining a target image sequence and target audio data of a target user;

[0008] performing facial expression feature extraction on the target image sequence to obtain a facial expression feature sequence, and performing text feature extraction on the target audio data to obtain a first text feature;

[0009] inputting the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep evaluation result output by the target neural network model.

[0010] Optionally, the inputting the facial expression feature sequence and the first text feature into the target neural network model to obtain the first sleep evaluation result output by the target neural network model comprises:

[0011] obtaining a second text feature obtained by performing text feature extraction on sleep quality description information of the target user;

[0012] fusing the facial expression feature sequence, the first text feature and the second text feature to obtain a target fusion feature;

[0013] inputting the target fusion feature into the target neural network model to obtain the first sleep evaluation result output by the target neural network model.

[0014] Optionally, the fusing the facial expression feature sequence, the first text feature, and the second text feature to obtain a target fusion feature comprises:

[0015] determining a sleep index of the target user based on the facial expression feature sequence, wherein the sleep index comprises at least one of a sleep duration, a wake-up frequency, a wake-up duration, and a sleep stage duration;

[0016] extracting a sleep index feature based on the sleep index;

[0017] fusing the sleep index feature, the facial expression feature sequence, the first text feature, and the second text feature to obtain the target fusion feature.

[0018] Optionally, the fusing the facial expression feature sequence, the first text feature, and the second text feature to obtain a target fusion feature comprises:

[0019] generating an expression statistical index of the target user based on the facial expression feature sequence;

[0020] fusing the expression statistical index, the first text feature, and the second text feature to obtain the target fusion feature.

[0021] Optionally, the sleep quality description information comprises voice description information and text description information, and the obtaining a second text feature extracted from sleep quality description information of the target user comprises:

[0022] extracting a first semantic feature based on the voice description information and a second semantic feature based on the text description information;

[0023] in a case where a similarity between the first semantic feature and the second semantic feature is less than or equal to a first threshold, determining information weights corresponding to the voice description information and the text description information respectively;

[0024] fusing a third text feature corresponding to the voice description information and a fourth text feature corresponding to the text description information based on the information weights to obtain the second text feature.

[0025] Optionally, the first text features include segment text features obtained by performing feature extraction on target audio segments corresponding to the target images, the target audio segments being obtained by cutting the target audio data based on the capture times of the target images, the number of the target audio segments being the same as the number of the target images, and the inputting of the facial expression feature sequence and the first text features into the target neural network model to obtain the first sleep evaluation result output by the target neural network model includes:

[0026] fusing the facial expression features corresponding to the target images in the facial expression feature sequence and the segment text features corresponding to the target images to obtain target segment features corresponding to the target images;

[0027] inputting the target segment features into the target neural network model to obtain segment sleep evaluation results corresponding to the target images output by the target neural network model;

[0028] averaging the segment sleep evaluation results corresponding to the target images to obtain the first sleep evaluation result of the target user.

[0029] Optionally, the fusing of the facial expression features corresponding to the target images in the facial expression feature sequence and the segment text features corresponding to the target images to obtain the target segment features corresponding to the target images includes:

[0030] obtaining physiological indicator data of the target user;

[0031] cutting physiological indicator segments corresponding to the target images from the physiological indicator data based on the capture times of the target images, the number of the physiological indicator segments being the same as the number of the target images;

[0032] fusing the facial expression features corresponding to the target images in the facial expression feature sequence, the segment text features corresponding to the target images, and the physiological indicator segments corresponding to the target images to obtain the target segment features.

[0033] Optionally, the method further includes:

[0034] obtaining physiological indicator data of the target user;

[0035] generating a second sleep evaluation result based on the physiological indicator data;

[0036] correcting the first sleep evaluation result based on the second sleep evaluation result to obtain a target sleep evaluation result.

[0037] Optionally, the method further includes:

[0038] generate sleep improvement suggestion information based on the first sleep assessment result;

[0039] send the sleep improvement suggestion information to the target user and / or an associated user corresponding to the target user.

[0040] In a second aspect, an embodiment of the present application provides a sleep quality assessment device, the device comprising:

[0041] an acquisition module configured to acquire a target image sequence and target audio data of a target user;

[0042] an extraction module configured to perform facial expression feature extraction on the target image sequence to obtain a facial expression feature sequence, and perform text feature extraction on the target audio data to obtain first text features;

[0043] a first sleep assessment result module configured to input the facial expression feature sequence and the first text features into a target neural network model to obtain a first sleep assessment result output by the target neural network model.

[0044] Optionally, the first sleep assessment result module comprises:

[0045] a text feature sub-module configured to acquire second text features obtained by performing text feature extraction on sleep quality description information of the target user;

[0046] a fusion feature sub-module configured to fuse the facial expression feature sequence, the first text features, and the second text features to obtain target fusion features;

[0047] a fusion feature evaluation sub-module configured to input the target fusion features into the target neural network model to obtain the first sleep assessment result output by the target neural network model.

[0048] Optionally, the fusion feature sub-module comprises:

[0049] a sleep index unit configured to determine sleep indexes of the target user based on the facial expression feature sequence, wherein the sleep indexes comprise at least one of sleep duration, wake-up times, wake-up duration, and sleep stage duration;

[0050] a sleep index feature unit configured to extract sleep index features based on the sleep indexes;

[0051] a first fusion unit configured to fuse the sleep index features, the facial expression feature sequence, the first text features, and the second text features to obtain the target fusion features.

[0052] Optionally, the fusion feature sub-module comprises:

[0053] an expression statistical index unit, configured to generate an expression statistical index of the target user based on the facial expression feature sequence;

[0054] a second fusion unit, configured to fuse the expression statistical index, the first text feature, and the second text feature to obtain the target fusion feature.

[0055] Optionally, the sleep quality description information includes voice description information and text description information, and the text feature submodule includes:

[0056] a semantic feature unit, configured to extract a first semantic feature based on the voice description information and extract a second semantic feature based on the text description information;

[0057] an information weight unit, configured to determine information weights corresponding to the voice description information and the text description information respectively in a case where a similarity between the first semantic feature and the second semantic feature is less than or equal to a first threshold value;

[0058] a text feature unit, configured to fuse a third text feature corresponding to the voice description information and a fourth text feature corresponding to the text description information based on the information weights to obtain the second text feature.

[0059] Optionally, the first text feature includes a segment text feature obtained by performing feature extraction on a target audio segment corresponding to the target image, the target audio segment is obtained by cutting from the target audio data based on a collection time of the target image, the number of target audio segments is the same as the number of target images, and the first sleep assessment result module includes:

[0060] a target segment feature submodule, configured to fuse a facial expression feature corresponding to the target image in the facial expression feature sequence and the segment text feature corresponding to the target image to obtain a target segment feature corresponding to the target image;

[0061] a segment sleep assessment result submodule, configured to input the target segment feature into the target neural network model to obtain a segment sleep assessment result corresponding to the target image output by the target neural network model;

[0062] a first sleep assessment result determination submodule, configured to average the segment sleep assessment results corresponding to the target images to obtain a first sleep assessment result of the target user.

[0063] Optionally, the target segment feature submodule includes:

[0064] a physiological index data unit, configured to acquire physiological index data of the target user;

[0065] a physiological index segment unit, configured to extract, from the physiological index data, a physiological index segment corresponding to the target image based on a collection time of the target image, the number of the physiological index segments being the same as the number of the target images;

[0066] a target segment feature fusion unit, configured to fuse a facial expression feature corresponding to the target image in the facial expression feature sequence, a segment text feature corresponding to the target image, and a physiological index segment corresponding to the target image, to obtain the target segment feature.

[0067] Optionally, the apparatus further includes:

[0068] a physiological index data module, configured to acquire physiological index data of the target user;

[0069] a second sleep evaluation result module, configured to generate a second sleep evaluation result based on the physiological index data;

[0070] a target sleep evaluation result module, configured to correct the first sleep evaluation result based on the second sleep evaluation result, to obtain a target sleep evaluation result.

[0071] Optionally, the apparatus further includes:

[0072] a sleep improvement suggestion information module, configured to generate sleep improvement suggestion information based on the first sleep evaluation result;

[0073] an information sending module, configured to send the sleep improvement suggestion information to the target user and / or an associated user corresponding to the target user.

[0074] In a third aspect, an embodiment of the present application provides an electronic device, including the sleep quality evaluation apparatus as described above, to implement the sleep quality evaluation method of the first aspect as described above.

[0075] In a fourth aspect, an embodiment of the present application provides a readable storage medium, the readable storage medium storing a program or instructions, the program or instructions being executed by a processor to implement the sleep quality evaluation method of the first aspect as described above.

[0076] In the embodiment of the present application, a sleep quality evaluation method is provided, comprising: obtaining a target image sequence and target audio data of a target user; performing facial expression feature extraction on the target image sequence to obtain a facial expression feature sequence, and performing text feature extraction on the target audio data to obtain a first text feature; inputting the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep evaluation result output by the target neural network model. The facial expression feature and the text feature can be extracted according to the image sequence and the audio data during the user's sleep. Since the facial expression feature can reflect the user's body expression and the text feature can reflect the user's language expression, the degree of stability of the user during the sleep process can be more accurately reflected through the facial expression feature and the text feature, so that the sleep quality evaluation result with higher accuracy can be output by analyzing the facial expression feature and the text feature through the neural network model. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 is a step flowchart of a sleep quality evaluation method provided by the embodiment of the present application;

[0078] Figure 2 is a sleep quality evaluation flowchart provided by the embodiment of the present application;

[0079] Figure 3 is a step flowchart of another sleep quality evaluation method provided by the embodiment of the present application;

[0080] Figure 4 is a feature fusion schematic diagram provided by the embodiment of the present application;

[0081] Figure 5 is a step flowchart of still another sleep quality evaluation method provided by the embodiment of the present application;

[0082] Figure 6 is a logic block diagram of a sleep quality evaluation device provided by the embodiment of the present application;

[0083] Figure 7 is a structural schematic diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0084] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0085] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a class, not limited to the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in a "or" relationship.

[0086] The sleep quality evaluation method provided by the embodiments of the present application will be described in detail below in conjunction with the drawings, specific embodiments and application scenarios.

[0087] Referring to Figure 1 , Figure 1 A step flowchart of a sleep quality evaluation method provided by an embodiment of the present application is shown, as Figure 1 shown, specifically comprising the following steps:

[0088] Step 101, obtaining a target image sequence of a target user and target audio data.

[0089] In the embodiments of the present application, a target image sequence containing a target user can be collected, and target audio data recorded near the target user. The above-mentioned target image sequence and target audio data can be collected during the sleep period of the target user, and the sleep period of the target user can be set by the target user, or obtained through the sleep monitoring function of the wearable device or other electronic devices carried by the target user, which is not limited in the embodiments of the present application.

[0090] The target image sequence can be in the form of a video, and the target image in the target image sequence is a video frame; the target image sequence can also be composed of a series of target images taken at a preset interval, for example, target images can be taken at an interval of 1 minute during the sleep period of the target user, which can constitute the above-mentioned target image sequence. The target image sequence can be collected by a camera set or placed near the target user, such as a surveillance camera, a mobile phone camera, etc., which is not limited in the embodiments of the present application.

[0091] The target audio data can be in the form of continuous recording of audio during the sleep period of the target user. The target audio data can be collected by an electronic device with a microphone set or placed around the target user, such as a monitor, a mobile phone, a notebook computer, a smart bracelet, etc., which is not limited in the embodiments of the present application.

[0092] In step 102, facial expression features of the target image sequence are extracted to obtain a facial expression feature sequence, and text features of the target audio data are extracted to obtain first text features.

[0093] In the embodiments of the present application, facial expression features of the target image sequence can be extracted to obtain a facial expression feature sequence. The facial expression features can include, but are not limited to, at least one of an eye opening degree feature, a mouth opening degree feature, a brow position feature, etc., and the embodiments of the present application are not limited specifically. Since the target image sequence can extract the facial expression feature sequence, and the facial expression feature sequence can reflect the expression change of the target user during the sleep process, for example, the target user frequently opens eyes and frequently opens and closes mouth during the sleep process, which indicates that the target user is not stable during the sleep process, and the sleep quality is low.

[0094] Exemplarily, the target images in the target image sequence can be extracted by the trained facial expression feature extraction model. Each target image in the target image sequence is input into the facial expression feature extraction model to obtain the facial expression feature corresponding to each target image output by the facial expression feature extraction model. The facial expression features can include, but are not limited to, happiness, sadness, anger, etc. The facial expression extraction model can detect and locate the face region in the image, extract low-level features (such as edge and texture features in the face region) through the convolution layer, then reduce the dimension and calculation amount through the pooling layer, extract higher-level face features, then gradually capture complex face features through the stacking of multiple convolution and pooling, and finally map the extracted features to specific expression categories through the fully connected layer to realize the recognition and extraction of facial expression features.

[0095] It should be noted that before the facial expression features of the target image sequence are extracted, the target images in the target image sequence can also be preprocessed to improve the accuracy of the facial expression feature extraction. The above preprocessing can include, but is not limited to, image scaling and cropping, grayscale, face detection and alignment, image noise reduction, etc., and the embodiments of the present application are not limited specifically.

[0096] In the embodiments of the present application, text features can also be extracted from the target audio data to obtain first text features. The first text features can include, but are not limited to, at least one of keyword features, semantic features, word frequency features, tone features, etc. Since the target audio data is collected during the target user's sleep period, the target audio data can include the target user's delirium, dream talk and other unconscious speech content during sleep, so that the first text features extracted can reflect the target user's sleep condition during sleep. If the first text features indicate that the target user has negative emotions such as panic and tension, or some negative keyword features frequently appear in the first text features, etc., it means that the target user's sleep process is not stable and the sleep quality is low.

[0097] Specifically, the target audio data can be processed and analyzed by natural language processing technology (NLP) to extract first text features. First, an application programming interface (API) that integrates language recognition, such as Google Speech API, can be used to convert the target audio data into text data, and then the text data can be input into a trained text feature extraction model to obtain first text features output by the text feature extraction model. The text feature extraction model can include, but is not limited to, BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), TF-IDF (Term Frequency-Inverse Document Frequency), etc., and the embodiments of the present application are not limited in this regard.

[0098] It should be noted that before the text features of the target audio data are extracted, the target audio data can also be preprocessed to improve the accuracy of speech recognition and the effect of subsequent feature extraction. The above-mentioned preprocessing can include, but is not limited to, audio sampling rate adjustment, noise removal, gain adjustment, speech enhancement, etc., and the embodiments of the present application are not limited in this regard.

[0099] Step 103: inputting the facial expression feature sequence and the first text features into a target neural network model to obtain first sleep assessment results output by the target neural network model.

[0100] In the embodiments of the present application, the target neural network model for sleep quality evaluation can be trained, the target neural network model can take the facial expression feature sequence and the first text feature as model input, and evaluate the sleep quality of the target user based on the facial expression feature sequence and the first text feature, and output the first sleep evaluation result of the target user. The target neural network model can include but is not limited to convolutional neural network (CNN), recurrent neural network (RNN), deep neural network (DNN), long short-term memory network (LSTM), time convolution network (TCN), etc., and the embodiments of the present application are not limited specifically.

[0101] Exemplarily, the target neural network model can include a time sequence feature extraction layer, a multi-modal fusion layer, a full connection layer and an output layer. The time sequence feature extraction layer can be used to capture the time sequence features of the facial expression feature sequence, the multi-modal fusion layer can be used to splice the time sequence features and the first text features to obtain the fusion features, the full connection layer is used to map the fusion features to the final output space, and the output layer is used to output the final first sleep evaluation result using an activation function. The first sleep evaluation result can include a sleep quality score (for example, 90 points), or a sleep quality rating (for example, poor, good, general, etc.), and the embodiments of the present application are not limited specifically.

[0102] It should be noted that the target neural network model can also not include the multi-modal fusion layer, and the fusion features can be obtained outside the model and directly input into the target neural network model, thereby improving the inference efficiency of the model.

[0103] Further, in order to improve the accuracy of the target neural network model, the target neural network model can be trained based on model-agnostic meta-learning (MAML). Exemplarily, a plurality of tasks can be first constructed, each task containing different training samples and test samples. Each task corresponds to the sleep data of a user. The data in the task includes the fused feature vector (F_combined) and the corresponding sleep quality score. Then, the model parameters θ are initialized, and for each task T i , the training data set and the test data set are selected respectively. Inner loop stage: Gradient descent is performed on the training data set of each task to update the model parameters to obtain task-specific parameters θ i : Wherein, the loss function represents the loss on task T i , and the learning rate α controls the update step. Outer loop stage: On the test data set The model performance is evaluated, the loss is calculated and the meta-gradient is accumulated, and the meta-model parameters are updated: where β is the meta learning rate for adjusting the parameter update step. Through multiple rounds of meta training, the initial parameters of the model are continuously optimized so that they can quickly adapt and generalize on different tasks. The model performance is evaluated in the final test set, and evaluation indicators such as mean square error, R 2 , accuracy, etc. are used to measure the prediction effect of the model. Through the above steps, MAML can effectively utilize the fusion data for user sleep quality evaluation. Through the meta learning process, the model significantly improves the adaptability and prediction accuracy on different user data.

[0104] In addition to MAML, the above target neural network model can also be trained using supervised learning (Supervised Learning), unsupervised learning (Unsupervised Learning), semi-supervised learning (Semi-Supervised Learning), reinforcement learning (Reinforcement Learning), etc. The skilled person can flexibly select the appropriate learning method according to the actual business needs, and the embodiments of the present application are not limited in detail.

[0105] Specifically, sample sleep data of a sample user can be obtained, where the sample sleep data can include a sample image sequence and sample audio data, facial expression feature extraction is performed on the sample image sequence to obtain a facial expression feature sequence of the sample image sequence, text feature extraction is performed on the sample audio data to obtain a first text feature of the sample audio data, and then the facial expression feature sequence and the first text feature are input into an initial neural network model to obtain a first evaluation result output by the initial neural network model; based on the first evaluation result and a sleep evaluation label corresponding to the sample user, a model loss value is determined, and the model parameters of the initial neural network model are adjusted based on the model loss value to obtain a target neural network model.

[0106] It should be noted that the above steps of performing facial expression feature extraction on the target image sequence to obtain a facial expression feature sequence, and performing text feature extraction on the target audio data to obtain a first text feature can be performed outside the model or inside the model. For example, a feature extraction neural network model can be used to extract features from the target image sequence and the target audio data, so that the feature extraction neural network model and the above target neural network model can constitute a complete neural network model. During model training, the sample sleep data and the sample sleep quality label of the sample user can be used to jointly train the feature extraction neural network model and the target neural network model. The embodiments of the present application do not limit the model structure in detail, and the skilled person can set the model structure according to actual needs.

[0107] Referring to Figure 2 , Figure 2 A sleep quality evaluation flowchart provided by an embodiment of the present application is shown as follows. Figure 2 As shown, first, sleep data is acquired through a user client, facial expression recognition is performed on target images in the sleep data, speech content recognition is performed on target audio, feature results are obtained, the feature results are encrypted and uploaded to a cloud server, the cloud server calls a target neural network model to analyze the feature results, sleep evaluation results are obtained, and the sleep evaluation results are encrypted and sent to the user client for display. It should be noted that the sleep quality evaluation function provided by the embodiment of the present application is running after being started by the user, and the sleep data is acquired after being authorized to collect by the user.

[0108] To sum up, the sleep quality evaluation method provided by the embodiment of the present application comprises: acquiring a target image sequence and target audio data of a target user; performing facial expression feature extraction on the target image sequence to obtain a facial expression feature sequence, and performing text feature extraction on the target audio data to obtain a first text feature; inputting the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep evaluation result output by the target neural network model. The facial expression feature and the text feature can be extracted according to the image sequence and the audio data during the sleep of the user. Since the facial expression feature can reflect the body expression of the user and the text feature can reflect the language expression of the user, the degree of stability of the user during the sleep process can be more accurately reflected through the facial expression feature and the text feature, so that the facial expression feature and the text feature are analyzed through the neural network model, and a sleep quality evaluation result with higher accuracy can be output.

[0109] As Figure 3 shown, Figure 3 is a step flowchart of another sleep quality evaluation method provided by an embodiment of the present application. The method can comprise:

[0110] Step 201: acquiring a target image sequence and target audio data of a target user.

[0111] This step can refer to step 101, and the embodiment of the present application will not be described again.

[0112] Optionally, the target user can wear a wearable device, such as a smart bracelet, etc., and the physiological indicators of the target user can be acquired through the wearable device. Whether the target user is in a sleep state can be determined through the physiological indicators, and in the case that the target user is in a sleep state, the target image sequence and the target audio data can be collected through the corresponding sensors, so that the target image sequence and the target audio data more accurately reflect the sleep process of the target user. The physiological indicators can include, but are not limited to, heart rate, respiratory rate, blood oxygen saturation, blood pressure, etc., and the embodiments of the present application are not limited specifically.

[0113] In step 202, facial expression feature extraction is performed on the target image sequence to obtain a facial expression feature sequence, and text feature extraction is performed on the target audio data to obtain a first text feature.

[0114] This step can refer to the above-mentioned step 102, and the embodiments of the present application will not be described again.

[0115] In step 203, a second text feature obtained by performing text feature extraction on the sleep quality description information of the target user is acquired.

[0116] In order to further improve the accuracy of sleep quality evaluation, in the embodiments of the present application, the sleep quality description information provided by the target user can also be acquired, and the sleep quality description information can be used to participate in the sleep quality evaluation process. The sleep quality description information can include voice description information and / or text description information.

[0117] The voice description information can be pre-recorded by the target user. For example, after the user logs in the system, the user can be asked to fill in a questionnaire about sleep quality, and the user can reply to the questionnaire through voice, text, options, etc. to obtain the sleep quality description information of the target user. Through the sleep quality description information, the subjective feeling of the user about the sleep quality of the user can be understood, so that the information of the subjective feeling dimension is added in the subsequent sleep quality evaluation process, which helps to improve the accuracy of sleep quality evaluation for the target user.

[0118] Specifically, text feature extraction can be performed on the sleep quality description information of the target user to obtain a second text feature. The second text feature can include at least one of keyword features, semantic features, word frequency features, tone features, etc. For example, if some negative keyword features appear in the second text feature, it indicates that the subjective feeling of the target user about the sleep quality is low.

[0119] Optionally, in the case that the sleep quality description information includes both voice description information and text description information, step 203 can include:

[0120] In the embodiment of the present application, if the sleep quality description information comprises both the voice description information and the text description information, the first semantic feature can be extracted based on the voice description information and the second semantic feature can be extracted based on the text description information. For the voice description information, the voice description information can be first converted into text information, and then the text information can be input into a semantic recognition model to obtain the first semantic feature thereof; for the text description information, the text description information can be directly input into the semantic recognition model to obtain the second semantic feature thereof.

[0121] In the embodiment of the present application, if the sleep quality description information comprises both the voice description information and the text description information, the first semantic feature can be extracted based on the voice description information and the second semantic feature can be extracted based on the text description information. For the voice description information, the voice description information can be first converted into text information, and then the text information can be input into a semantic recognition model to obtain the first semantic feature thereof; for the text description information, the text description information can be directly input into the semantic recognition model to obtain the second semantic feature thereof.

[0122] In the embodiment of the present application, if the sleep quality description information comprises both the voice description information and the text description information, the first semantic feature can be extracted based on the voice description information and the second semantic feature can be extracted based on the text description information. For the voice description information, the voice description information can be first converted into text information, and then the text information can be input into a semantic recognition model to obtain the first semantic feature thereof; for the text description information, the text description information can be directly input into the semantic recognition model to obtain the second semantic feature thereof.

[0123] In the embodiment of the present application, considering that the content of the voice description information and the text description information can be inconsistent or even conflicting, in order to improve the accuracy of determining the second text feature, the first semantic feature and the second semantic feature can be first compared to obtain the similarity therebetween, if the similarity is high, it indicates that the voice description information and the text description information express similar meanings, and then the third text feature corresponding to the voice description information and the fourth text feature corresponding to the text description information can be directly fused based on the information weight to obtain the second text feature; if the similarity is low, it indicates that the voice description information and the text description information express different meanings, or even completely opposite meanings, and then the information weight corresponding to the voice description information and the information weight corresponding to the text description information can be calculated, and the third text feature and the fourth text feature can be weightedly fused to avoid the inaccurate problem caused by direct fusion.

[0124] Specifically, in an implementation manner, since the text description information is input by the user in written form and usually has high accuracy, the information weight corresponding to the voice description information and the information weight corresponding to the text description information can be determined according to the similarity, so that the lower the similarity is, the lower the information weight corresponding to the voice description information is, and the higher the information weight corresponding to the text description information is. Exemplarily, the similarity can be normalized to obtain a similarity normalized value, the similarity normalized value can be taken as the information weight of the voice description information, and "1-similarity normalized value" can be taken as the information weight of the text description information.

[0125] In another implementation, the sleep quality described by the voice description information and the text description information can be determined according to the first semantic feature and the second semantic feature, respectively, and then the historical sleep evaluation result of the target user can be obtained, and the information weight corresponding to the voice description information and the text description information can be determined according to the similarity between the sleep quality described by the voice description information and the text description information and the historical sleep evaluation result.

[0126] In sub-step 2033, the third text feature corresponding to the voice description information and the fourth text feature corresponding to the text description information are fused based on the information weight to obtain the second text feature.

[0127] In the embodiments of the present application, the third text feature corresponding to the voice description information and the fourth text feature corresponding to the text description information can be fused by weighted average to obtain the second text feature.

[0128] The first semantic feature is extracted based on the voice description information, the second semantic feature is extracted based on the text description information, the information weight corresponding to the voice description information and the text description information is determined in the case that the similarity between the first semantic feature and the second semantic feature is less than or equal to the first threshold, and the third text feature corresponding to the voice description information and the fourth text feature corresponding to the text description information are fused based on the information weight to obtain the second text feature. When the voice description information and the text description information are inconsistent, the weight corresponding to each description information can be determined, the second text feature is obtained by weighted fusion, which helps to improve the accuracy of the second text feature.

[0129] In step 204, the face expression feature sequence, the first text feature and the second text feature are fused to obtain a target fusion feature.

[0130] In the embodiments of the present application, the face expression feature sequence, the first text feature and the second text feature can be fused to obtain the target fusion feature. Specifically, the feature vectors corresponding to the face expression feature sequence, the first text feature and the second text feature can be spliced to obtain the feature vector of the target fusion feature. Other ways of fusing the above features can also be used, such as fusion by a feature fusion neural network model, fusion according to the preset fusion weight corresponding to each feature, etc., which are not limited in the embodiments of the present application.

[0131] Referring to Figure 4 , Figure 4 A feature fusion schematic diagram provided by the embodiments of the present application is shown as Figure 4As shown, after the user logs in, it is judged whether the user logs in for the first time. If the user logs in for the first time, it is inquired whether the user opens the sleep quality evaluation function. After the user opens the function and authorizes to obtain sleep data, the sleep data is collected. The face feature of the target image in the sleep data is captured, and the facial expression feature sequence is recognized according to the face feature. The target audio in the sleep data can be recognized by voice recognition API to obtain the first text feature. The sleep quality description information can be recognized by natural language processing technology to obtain the second text feature. Then, the facial expression feature sequence, the first text feature and the second text feature are fused to obtain the target fusion feature. The target fusion feature is input into the target neural network model to obtain the sleep evaluation result output by the target neural network model. If the user logs in for the second time, the physiological index data collected by the wearable device and / or the sleep quality description information can be used to preliminarily judge whether the user's sleep quality has a problem. If the judgment result is that the sleep quality may have a problem, the user is notified and inquired whether to open the sleep evaluation function. After the user authorizes, the sleep data is collected.

[0132] Optionally, step 204 can include:

[0133] Step A1, determining a sleep indicator of the target user based on the facial expression feature sequence; wherein the sleep indicator includes at least one of sleep duration, wake-up times, wake-up duration and sleep stage duration.

[0134] In the embodiment of the present application, since the facial expression feature sequence can reflect the expression of the target user at each time during sleep, the facial expression feature sequence can be analyzed to determine the sleep indicator of the target user, wherein the sleep indicator can include but is not limited to at least one of sleep duration, wake-up times, wake-up duration and sleep stage duration, and the sleep stage can include but is not limited to non-rapid eye movement sleep stage (NREM) and rapid eye movement sleep stage (REM), and the non-rapid eye movement sleep stage can be further divided into sleep-onset stage, light sleep stage and deep sleep stage.

[0135] Specifically, the sleep state of the target user at the corresponding time can be determined through the facial expression feature. For example, if the number of continuous changes of the facial expression feature of the target user within a preset time exceeds a preset number, it can be judged that the target user is in the sleep-onset stage. If the number of continuous changes is less than or equal to the preset number and greater than 0 times, it can be judged that the target user is in the light sleep stage. If the number of continuous changes is equal to 0 times, it can be judged that the target user is in the deep sleep stage. If the angle of the open eyes of the target user is greater than a preset angle, it can be judged that the target user is in the wake-up stage. The sleep indicators such as the duration of each sleep stage, the total sleep duration, the wake-up times and the wake-up duration can be counted through the sleep stage judgment strategy of the above examples.

[0136] Step A3, extracting a sleep index feature based on the sleep index.

[0137] In the embodiments of the present application, the sleep index can be subjected to feature extraction to obtain a sleep index feature. Specifically, each sleep index can be vectorized to obtain a feature vector of each sleep index, and the feature vectors of each sleep index can be spliced to obtain the sleep index feature.

[0138] Step A2, fusing the sleep index feature, the facial expression feature sequence, the first text feature and the second text feature to obtain the target fusion feature.

[0139] In the embodiments of the present application, the sleep index feature, the facial expression feature sequence, the first text feature and the second text feature can be fused to obtain the target fusion feature. Specifically, the feature vectors corresponding to the sleep index feature, the facial expression feature sequence, the first text feature and the second text feature can be spliced to obtain the feature vector of the target fusion feature. Other ways of fusing the above features can also be used, such as fusing through a feature fusion neural network model, fusing according to the preset fusion weights corresponding to various features, etc., which are not limited in the embodiments of the present application.

[0140] Based on the facial expression feature sequence, a sleep index of the target user is determined; wherein the sleep index comprises at least one of sleep duration, wake-up times, wake-up duration and sleep stage duration; a sleep index feature is extracted based on the sleep index; and the sleep index feature, the facial expression feature sequence, the first text feature and the second text feature are fused to obtain the target fusion feature. The sleep index feature can be introduced into the target fusion feature, and the sleep index feature is generated based on the more intuitive facial expression feature sequence, has high accuracy, so that the target fusion feature can more accurately reflect the sleep situation of the target user, and helps to improve the accuracy of determining the sleep quality of the user based on the target fusion feature.

[0141] Optionally, the facial expression feature sequence can be further processed to extract an expression statistical index, and the information contained in the front and rear facial expression features in the facial expression feature sequence is fully utilized. Step 204 can include:

[0142] Step B1, generating an expression statistical index of the target user based on the facial expression feature sequence.

[0143] In the embodiments of the present application, the expression statistical index of the target user can be generated based on the facial expression feature sequence. The expression statistical index can include but is not limited to the number of expression changes, the frequency of expression changes, the duration of each type of expression, etc., which are not limited in the embodiments of the present application.

[0144] Specifically, the number of inconsistent adjacent facial expression feature pairs in the facial expression feature sequence can be counted to obtain the number of expression changes; the expression change frequency can be determined according to the ratio of the total duration of the facial expression feature sequence and the number of expression changes; and the duration of each type of user facial expression can be determined according to the product of the collection interval duration between the target images corresponding to adjacent facial expression features in the facial expression feature sequence and the number of occurrences of each type of facial expression feature in the facial expression feature sequence.

[0145] In step B2, the expression statistical indicator, the first text feature and the second text feature are fused to obtain the target fusion feature.

[0146] In the embodiments of the present application, the expression statistical indicator, the first text feature and the second text feature can be fused to obtain the target fusion feature. Specifically, the expression statistical indicator can be first vectorized into a feature vector, and the feature vector of the expression statistical indicator can be spliced with the vectors of the first text feature and the second text feature to obtain the target fusion feature.

[0147] It should be noted that in the schemes in which the facial expression feature sequence participates in feature fusion, the expression statistical indicator can be generated based on the facial expression feature sequence, and then the expression statistical indicator can be used to replace the facial expression feature sequence to participate in feature fusion to obtain the corresponding target fusion feature. Through the expression statistical indicator, the key features in the facial expression feature sequence can be extracted, the use of the facial expression feature sequence with a relatively long length to participate in feature fusion can be avoided, the feature length of the target fusion feature can be effectively reduced, and the efficiency of subsequent sleep quality judgment based on the target fusion feature can be improved.

[0148] In step 205, the target fusion feature is input into the target neural network model to obtain the first sleep evaluation result output by the target neural network model.

[0149] In the embodiments of the present application, the target fusion feature can be input into the target neural network model to obtain the first sleep evaluation result output by the target neural network model.

[0150] It should be noted that the model structure and training method of the target neural network model can be different due to different input data. For example, if the input data of the target neural network model is a facial expression feature sequence and a first text feature, the target neural network model can include a feature fusion layer and an output layer. If the input data of the target neural network model is a target fusion feature, the target neural network model can include an output layer. The above process of feature extraction and feature fusion to obtain the target fusion feature can be realized by a feature extraction module and a feature fusion module outside the target neural network model, or the target neural network model can be designed to have a feature extraction layer and a feature fusion layer, and the target neural network model can directly receive the sample image sequence, the sample audio data and the sleep quality description information, and perform feature extraction and feature fusion inside the model. The embodiments of the present application are not limited specifically.

[0151] The second text feature obtained by performing text feature extraction on the sleep quality description information of the target user is obtained. The facial expression feature sequence, the first text feature and the second text feature are fused to obtain a target fusion feature. The target fusion feature is input into the target neural network model to obtain a first sleep evaluation result output by the target neural network model. The target fusion feature can be obtained according to the subjective sleep feeling of the target user, so that the target fusion feature contains feature information related to the subjective sleep feeling of the user, and the sleep quality analysis based on the target fusion feature can achieve higher accuracy.

[0152] In step 206, physiological index data of the target user is obtained.

[0153] In the embodiments of the present application, the physiological indicators of the target user can also be collected by the wearable device of the target user, and the physiological index data corresponding to the target image sequence and the target audio data collection period of the target user is obtained. The first sleep evaluation result is corrected by these physiological index data to improve the accuracy of sleep quality evaluation. The above-mentioned physiological indicators can include but are not limited to heart rate, respiratory rate, blood oxygen saturation, blood pressure, etc. The wearable device can include but is not limited to a smart bracelet, a vital sign monitor, smart glasses, etc. The embodiments of the present application are not limited specifically.

[0154] In step 207, a second sleep evaluation result is generated based on the physiological index data.

[0155] In the embodiments of the present application, the second sleep evaluation result can be generated based on the physiological index data. Specifically, in one implementation, physiological statistical data can be generated according to the physiological index data, and then the physiological statistical data is input into a physiological index sleep evaluation model to obtain the second sleep evaluation result output by the physiological index sleep evaluation model. The physiological statistical data can include but is not limited to: average heart rate, maximum heart rate, minimum heart rate, heart rate change rate, average respiratory rate, maximum respiratory rate, minimum respiratory rate, respiratory rate change rate, average blood oxygen saturation, maximum blood pressure, etc. The physiological index sleep evaluation model can include but is not limited to a mathematical model, a conditional judgment model, a decision tree model, a neural network model, etc., and the embodiments of the present application are not limited in this regard.

[0156] In another implementation, the physiological index data can be directly input into the physiological index sleep evaluation model, the physiological index sleep evaluation model can extract features from the physiological index data to obtain physiological index features, and the second sleep evaluation result can be output according to the physiological index features. The physiological index sleep evaluation model can include a neural network model.

[0157] In step 208, the first sleep evaluation result is corrected based on the second sleep evaluation result to obtain a target sleep evaluation result.

[0158] In the embodiments of the present application, the second sleep evaluation result can be used to correct the first sleep evaluation result to obtain a target sleep evaluation result. Specifically, the second sleep evaluation result and the first sleep evaluation result can be averaged to obtain the target sleep evaluation result. The second sleep evaluation result and the first sleep evaluation result can also be weighted and averaged to obtain the target sleep evaluation result, wherein the weights of the weighted average can be preset values, for example, the weight of the second sleep evaluation result can be set to 0.2 and the weight of the first sleep evaluation result can be set to 1. The weights of the weighted average can also be determined according to the number of index types contained in the physiological index data. If the physiological index data contains more index types, the weight of the second sleep evaluation result relative to the weight of the first sleep evaluation result can be set to be higher. If the physiological index data contains fewer index types, the weight of the second sleep evaluation result relative to the weight of the first sleep evaluation result can be set to be lower.

[0159] It should be noted that in the embodiments of the present application, the physiological index data can be extracted to obtain physiological index features, and the physiological index features can be used in the above feature fusion process to obtain the above target fusion features, so that the target neural network model outputs a first sleep evaluation result with higher accuracy based on the target fusion features.

[0160] The physiological index data of the target user is acquired, a second sleep evaluation result is generated based on the physiological index data, and the first sleep evaluation result is corrected based on the second sleep evaluation result to obtain a target sleep evaluation result. The second sleep evaluation result determined based on the physiological index features in the prior art can be used to correct the first sleep evaluation result to obtain the target sleep evaluation result, which helps to improve the accuracy of the target sleep evaluation result.

[0161] Optionally, for the sleep evaluation result, the following steps C1 to C2 can also be performed:

[0162] In step C1, sleep improvement suggestion information is generated based on the first sleep evaluation result.

[0163] In the embodiments of the present application, sleep improvement suggestion information corresponding to the target user can also be generated based on the sleep evaluation result (e.g., the first sleep evaluation result or the target sleep evaluation result). Specifically, in one implementation, a corresponding relationship between different sleep evaluation results and preset suggestion information can be established, for example, the sleep evaluation result can include a sleep quality score or a sleep quality level, and preset suggestion information corresponding to different sleep quality scores or sleep quality levels can be preset. After obtaining the sleep evaluation result of the target user, the corresponding preset suggestion information can be queried to obtain the sleep improvement suggestion information.

[0164] In another implementation, an improvement suggestion model can be pre-trained. The improvement suggestion model can be trained according to sample sleep quality scores set by expert experience and corresponding sample sleep improvement suggestion information. The sleep evaluation result of the target user can be input into the improvement suggestion model to obtain the sleep improvement suggestion information output by the improvement suggestion model.

[0165] It should be noted that the specific content of the sleep improvement suggestion information can include but is not limited to sleep time suggestion, sleep duration suggestion, etc., and the presentation form can include but is not limited to text information, voice information, graphical information, etc., and the embodiments of the present application are not limited in this regard.

[0166] In step C2, the sleep improvement suggestion information is sent to the target user and / or an associated user corresponding to the target user.

[0167] In the embodiments of the present application, after obtaining the sleep improvement suggestion information, the sleep improvement suggestion information can be sent to the target user and / or an associated user corresponding to the target user. The associated user can include but is not limited to a guardian user of the target user, an authorized user of the target user, etc., and the embodiments of the present application are not limited in this regard. In addition, the target user first sleep evaluation result or the target sleep evaluation result can also be sent to the target user and / or the associated user.

[0168] By generating sleep improvement suggestions based on the first sleep assessment results and sending the sleep improvement suggestions to the target user and / or the associated users corresponding to the target user, sleep improvement suggestions can be provided based on more accurate sleep assessment results, making the sleep improvement suggestions equally accurate. Furthermore, the suggestions can be synchronized to the target user and / or associated users, thereby facilitating the target user to adjust their sleep habits according to the sleep improvement suggestions.

[0169] In summary, another sleep quality assessment method provided in this application includes: acquiring a target image sequence and target audio data of a target user; extracting facial expression features from the target image sequence to obtain a facial expression feature sequence; extracting text features from the target audio data to obtain a first text feature; and inputting the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep assessment result output by the target neural network model. This method can extract facial expression features and text features from image sequences and audio data during a user's sleep. Since facial expression features can reflect a user's body language and text features can reflect a user's verbal language, facial expression features and text features can accurately reflect the user's level of rest during sleep. Therefore, by analyzing facial expression features and text features through a neural network model, a highly accurate sleep quality assessment result can be output.

[0170] like Figure 5 As shown, Figure 5 A flowchart illustrating the steps of another sleep quality assessment method provided in this application embodiment, the method may include:

[0171] Step 301: Obtain the target image sequence and target audio data of the target user.

[0172] This step can be found in step 101, and will not be repeated in this embodiment.

[0173] Step 302: Extract facial expression features from the target image sequence to obtain a facial expression feature sequence; extract text features from the target audio data to obtain a first text feature.

[0174] This step can be found in step 102, and will not be repeated in this embodiment.

[0175] Step 303: Fuse the facial expression features corresponding to the target image in the facial expression feature sequence with the fragment text features corresponding to the target image to obtain the target fragment features corresponding to the target image.

[0176] In the embodiment of the present application, the first text feature can include a segment text feature obtained by performing feature extraction on a target audio segment corresponding to the target image, the target audio segment being obtained by cutting from target audio data based on the capture time of the target image, and the number of target audio segments being the same as the number of target images. For example, the shooting time of the target image sequence can be from 23:00 to 7:00 the next day, a total of 8 hours, and the shooting interval of the target images in the target image sequence can be 1 minute. Correspondingly, the target audio data can include continuous audio recorded from 23:00 to 7:00 the next day. The target audio data can be segmented into 1-minute segments based on the capture time of the target image, so as to obtain a target audio segment corresponding to each target image. Thus, the number of target audio segments corresponding to the target images can be obtained. By performing text feature extraction on the target audio segment, the segment text feature corresponding to the target audio segment can be obtained. That is, the segment text features of all target audio segments can constitute the first text feature corresponding to the target audio data.

[0177] The facial expression feature corresponding to the target image in the facial expression feature sequence and the segment text feature of the target audio segment corresponding to the target image can be fused to obtain a target segment feature corresponding to the target image. The above steps are repeated until target segment features corresponding to all target images in the target image sequence are obtained.

[0178] Optionally, step 303 can include:

[0179] Sub-step 3031, acquiring physiological indicator data of the target user.

[0180] In the embodiment of the present application, the physiological indicators of the target user can also be collected by the wearable device of the target user, and the physiological indicator data corresponding to the target image sequence and the target audio data collection period of the target user can be acquired. The physiological indicators can include, but are not limited to, heart rate, respiratory rate, blood oxygen saturation, blood pressure, etc. The wearable device can include, but is not limited to, a smart bracelet, a vital sign monitor, smart glasses, etc., and the embodiment of the present application is not limited in this regard.

[0181] Sub-step 3032, based on the capture time of the target image, cutting a physiological indicator segment corresponding to the target image from the physiological indicator data, the number of physiological indicator segments being the same as the number of target images.

[0182] In the embodiment of the present application, the physiological indicator segment corresponding to the target image can be intercepted from the physiological indicator data based on the acquisition time of the target image, and the number of physiological indicator segments is the same as the number of target images. For example, the shooting time of the target image sequence can be from 23:00 to 7:00 the next day, for a total of 8 hours, and the shooting interval of the target image in the target image sequence can be 1 minute. Correspondingly, the physiological indicator data can be collected from 23:00 to 7:00 the next day, and the collection time interval can also be 1 minute. The physiological indicator data can be segmented into 1-minute segments based on the acquisition time of the target image to obtain the physiological indicator segment corresponding to each target image, so that the number of physiological indicator segments is equal to the number of target images.

[0183] In substep 3033, the facial expression feature corresponding to the target image in the facial expression feature sequence, the segment text feature corresponding to the target image, and the physiological indicator segment corresponding to the target image are fused to obtain the target segment feature.

[0184] In the embodiment of the present application, the physiological indicator segment can be subjected to feature extraction to obtain a physiological indicator segment feature, and the physiological indicator segment feature can be used to participate in the above feature fusion process to obtain the above target fusion feature, that is, the target segment feature is obtained by fusing the facial expression feature corresponding to the target image in the facial expression feature sequence, the segment text feature corresponding to the target image, and the physiological indicator segment feature corresponding to the target image, so that the target neural network model outputs a more accurate segment sleep evaluation result based on the target segment feature.

[0185] By obtaining the physiological indicator data of the target user, intercepting the physiological indicator segment corresponding to the target image from the physiological indicator data based on the acquisition time of the target image, and fusing the facial expression feature corresponding to the target image in the facial expression feature sequence, the segment text feature corresponding to the target image, and the physiological indicator segment corresponding to the target image to obtain the target segment feature, the feature dimension of the target segment feature can be improved, the physiological indicator feature can be introduced into the target segment feature, the accuracy of the segment sleep evaluation result obtained based on the target segment feature can be improved, and the accuracy of the sleep quality evaluation for the target user can be improved.

[0186] In step 304, the target segment feature is input into the target neural network model to obtain the segment sleep evaluation result corresponding to the target image output by the target neural network model.

[0187] In this embodiment, the target neural network model can receive target segment features as input and output segment sleep assessment results. The target segment features corresponding to each target image can be input into the target neural network model to obtain segment sleep assessment results corresponding to each target image. The segment sleep assessment results are similar to the first sleep assessment results in the aforementioned embodiments and may include sleep quality scores or sleep quality ratings.

[0188] It should be noted that the above-mentioned target fragment features can also be obtained by fusing other features extracted from other types of data. The feature types participating in feature fusion to obtain target fragment features can be the same as the feature types participating in target fusion features in the aforementioned embodiments, which will not be repeated in this application embodiment.

[0189] Step 305: Average the sleep assessment results of the segments corresponding to each target image to obtain the first sleep assessment result of the target user.

[0190] After obtaining the sleep assessment results for each segment corresponding to each target image, the sleep assessment results for each segment can be averaged to obtain the first sleep assessment result for the target user.

[0191] By fusing facial expression features corresponding to the target image from the facial expression feature sequence, and segment text features corresponding to the target image, target segment features corresponding to the target image are obtained. These target segment features are then input into a target neural network model to obtain the segment sleep assessment results output by the model. The average of these segment sleep assessment results for each target image is then used to obtain the target user's first sleep assessment result. This method can generate target segment features for multiple sleep segments during a target user's sleep. The target neural network model outputs corresponding segment sleep assessment results for each sleep segment, refining the target user's sleep process into multiple segments for separate assessment. Finally, the assessment results for all sleep segments are aggregated to obtain the target user's first sleep assessment result, which helps improve the accuracy of the first sleep assessment result.

[0192] like Figure 6 As shown, Figure 6 A logic block diagram of a sleep quality assessment device provided in this application embodiment, the sleep quality assessment device 600 may include:

[0193] The acquisition module 601 is used to acquire the target image sequence and target audio data of the target user;

[0194] Extraction module 602 is used to extract facial expression features from the target image sequence to obtain a facial expression feature sequence, and to extract text features from the target audio data to obtain a first text feature;

[0195] The first sleep evaluation result module 603 is configured to input the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep evaluation result output by the target neural network model.

[0196] Optionally, the first sleep evaluation result module comprises:

[0197] The text feature submodule is configured to obtain second text features extracted from sleep quality description information of the target user;

[0198] The fusion feature submodule is configured to fuse the facial expression feature sequence, the first text feature and the second text feature to obtain target fusion features;

[0199] The fusion feature evaluation submodule is configured to input the target fusion features into the target neural network model to obtain the first sleep evaluation result output by the target neural network model.

[0200] Optionally, the fusion feature submodule comprises:

[0201] The sleep index unit is configured to determine sleep indexes of the target user based on the facial expression feature sequence, wherein the sleep indexes comprise at least one of sleep duration, wake-up times, wake-up duration and sleep stage duration;

[0202] The sleep index feature unit is configured to extract sleep index features based on the sleep indexes;

[0203] The first fusion unit is configured to fuse the sleep index features, the facial expression feature sequence, the first text feature and the second text feature to obtain the target fusion features.

[0204] Optionally, the fusion feature submodule comprises:

[0205] The expression statistical index unit is configured to generate expression statistical indexes of the target user based on the facial expression feature sequence;

[0206] The second fusion unit is configured to fuse the expression statistical indexes, the first text feature and the second text feature to obtain the target fusion features.

[0207] Optionally, the sleep quality description information comprises voice description information and text description information, and the text feature submodule comprises:

[0208] The semantic feature unit is configured to extract first semantic features based on the voice description information and second semantic features based on the text description information;

[0209] an information weight unit, configured to determine information weights corresponding to the voice description information and the text description information respectively, in a case where a similarity between the first semantic feature and the second semantic feature is less than or equal to a first threshold value;

[0210] a text feature unit, configured to fuse a third text feature corresponding to the voice description information and a fourth text feature corresponding to the text description information based on the information weights, to obtain the second text feature.

[0211] Optionally, the first text feature includes a segment text feature obtained by performing feature extraction on a target audio segment corresponding to the target image, the target audio segment being obtained by cutting from the target audio data based on a collection time of the target image, and the number of the target audio segments being the same as the number of the target images.

[0212] a target segment feature submodule, configured to fuse a facial expression feature corresponding to the target image in the facial expression feature sequence and the segment text feature corresponding to the target image, to obtain a target segment feature corresponding to the target image;

[0213] a segment sleep assessment result submodule, configured to input the target segment feature into the target neural network model, to obtain a segment sleep assessment result corresponding to the target image output by the target neural network model;

[0214] a first sleep assessment result determination submodule, configured to average the segment sleep assessment results corresponding to the target images, to obtain a first sleep assessment result of the target user.

[0215] Optionally, the target segment feature submodule includes:

[0216] a physiological index data unit, configured to acquire physiological index data of the target user;

[0217] a physiological index segment unit, configured to cut a physiological index segment corresponding to the target image from the physiological index data based on a collection time of the target image, and the number of the physiological index segments being the same as the number of the target images.

[0218] a target segment feature fusion unit, configured to fuse the facial expression feature corresponding to the target image in the facial expression feature sequence, the segment text feature corresponding to the target image, and a physiological index segment corresponding to the target image, to obtain the target segment feature.

[0219] Optionally, the apparatus further includes:

[0220] a physiological index data module configured to acquire physiological index data of the target user;

[0221] a second sleep evaluation result module configured to generate a second sleep evaluation result based on the physiological index data;

[0222] a target sleep evaluation result module configured to correct the first sleep evaluation result based on the second sleep evaluation result to obtain a target sleep evaluation result.

[0223] Optionally, the device further comprises:

[0224] a sleep improvement suggestion information module configured to generate sleep improvement suggestion information based on the first sleep evaluation result;

[0225] an information sending module configured to send the sleep improvement suggestion information to the target user and / or an associated user corresponding to the target user.

[0226] In summary, the sleep quality evaluation device provided by the embodiments of the present application comprises an acquisition module configured to acquire a target image sequence and target audio data of a target user; an extraction module configured to perform facial expression feature extraction on the target image sequence to obtain a facial expression feature sequence, and perform text feature extraction on the target audio data to obtain a first text feature; and a first sleep evaluation result module configured to input the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep evaluation result output by the target neural network model. The facial expression feature and the text feature can be extracted from the image sequence and the audio data during the sleep of the user. Since the facial expression feature can reflect the body expression of the user and the text feature can reflect the language expression of the user, the degree of stability of the user during the sleep process can be accurately reflected through the facial expression feature and the text feature. Therefore, the sleep quality evaluation result with high accuracy can be output by analyzing the facial expression feature and the text feature through the neural network model.

[0227] The sleep quality evaluation apparatus in the embodiments of the present applicationapplicationbe an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic deviceapplicationbe a terminal or other device than a terminal. For example, the electronic deviceapplicationbe a GPU BOX, a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and the like, andapplicationbe a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, a self-service machine, and the like. The embodiments of the present application do not make a specific limitation.

[0228] The sleep quality evaluation apparatus in the embodiments of the present applicationapplicationbe a device with an operating system. The operating systemapplicationbe an Android operating system, a Linux operating system, a Windows operating system, or the like, or other possible operating system. The embodiments of the present application do not make a specific limitation.

[0229] The sleep quality evaluation apparatus provided in the embodiments of the present applicationapplicationbe capable of realizing the method embodiments. Figures 1 to 5 The method embodiments realize various processes, which will not be repeated here to avoid repetition.

[0230] Optionally, as shown in Figure 7 The embodiments of the present application further provide an electronic device M00, whichapplicationinclude a processor M01 and a memory M02. The memory M02applicationstore programs or instructions thatapplicationbe run on the processor M01. When the programs or instructions are executed by the processor M01, various steps of the sleep quality evaluation method embodiments described above are realized, and the same technical effects are achieved. To avoid repetition, the various steps will not be repeated here.

[0231] In the embodiments of the present application, the memory M02 can be used to store software programs and various data. The memory M02 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory M02 can include a volatile memory or a non-volatile memory, or the memory x09 can include both volatile and non-volatile memories. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory M02 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0232] The processor M01 can include one or more processing units; optionally, the processor M01 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor M01.

[0233] The embodiments of the present application also provide an electronic device including the sleep quality evaluation device and the refrigerant circulation device as described above, to realize each process of the sleep quality evaluation method embodiments and achieve the same technical effects. To avoid repetition, details are not described here.

[0234] The embodiment of the present application further provides a readable storage medium, wherein the readable storage medium stores a program or instructions, and the program or instructions are executed by a processor to realize the processes of the sleep quality evaluation method and achieve the same technical effects. To avoid repetition, details are not described herein.

[0235] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0236] The embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface is coupled with the processor. The processor is used to run a program or instructions to realize the processes of the sleep quality evaluation method and achieve the same technical effects. To avoid repetition, details are not described herein.

[0237] It should be understood that the chip involved in the embodiment of the present application can also be referred to as a system chip, a system chip, a chip system or a system on chip, etc.

[0238] The embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to realize the processes of the sleep quality evaluation method and achieve the same technical effects. To avoid repetition, details are not described herein.

[0239] It should be noted that, in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to the order of functions shown or discussed, but can also include functions performed in a substantially simultaneous manner or in a reverse order, for example, the described method can be performed in an order different from that described, and various steps can be added, omitted or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0240] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned example methods can be realized by means of software and a necessary general hardware platform, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer software product in essence or in the form of a contribution to the related art. The computer software product is stored in a storage medium (such as a ROM / RAM, a magnetic disc, an optical disc), and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0241] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative and not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. A sleep quality assessment method, characterized by, The method includes: Acquire the target user's target image sequence and target audio data; Facial expression features are extracted from the target image sequence to obtain a facial expression feature sequence, and text features are extracted from the target audio data to obtain a first text feature; The facial expression feature sequence and the first text feature are input into the target neural network model to obtain the first sleep assessment result output by the target neural network model. The step of inputting the facial expression feature sequence and the first text feature into the target neural network model to obtain the first sleep assessment result output by the target neural network model includes: The second text feature is obtained by extracting text features from the sleep quality description information of the target user; The target fusion feature is obtained by fusing the facial expression feature sequence, the first text feature, and the second text feature; The target fusion features are input into the target neural network model to obtain the first sleep assessment result output by the target neural network model; The process of fusing the facial expression feature sequence, the first text feature, and the second text feature to obtain the target fused feature includes: Based on the facial expression feature sequence, generate the facial expression statistics of the target user; The target fusion feature is obtained by fusing the facial expression statistics, the first text feature, and the second text feature; The sleep quality description information includes voice description information and text description information. The step of obtaining the second text feature obtained by extracting text features from the sleep quality description information of the target user includes: A first semantic feature is extracted based on the speech description information, and a second semantic feature is extracted based on the text description information; If the similarity between the first semantic feature and the second semantic feature is less than or equal to a first threshold, the information weights corresponding to the speech description information and the text description information are determined respectively. The second text feature is obtained by fusing the third text feature corresponding to the speech description information and the fourth text feature corresponding to the text description information based on the information weight.

2. The method of claim 1, wherein, The process of fusing the facial expression feature sequence, the first text feature, and the second text feature to obtain the target fused feature includes: Based on the facial expression feature sequence, the sleep index of the target user is determined; wherein, the sleep index includes at least one of sleep duration, number of awakenings, awakening duration, and duration of sleep stage. Sleep index features are extracted based on the aforementioned sleep indexes; The target fusion feature is obtained by fusing the sleep index features, the facial expression feature sequence, the first text feature, and the second text feature.

3. The method of claim 1, wherein, The first text feature includes a segment text feature obtained by performing feature extraction on a target audio segment corresponding to the target image, the target audio segment being obtained by cutting the target audio data based on a collection time of the target image, a number of the target audio segment being the same as a number of the target image, the face expression feature sequence and the first text feature being input into a target neural network model to obtain a first sleep evaluation result output by the target neural network model, including: fusing a face expression feature corresponding to the target image in the face expression feature sequence and a segment text feature corresponding to the target image to obtain a target segment feature corresponding to the target image; inputting the target segment feature into the target neural network model to obtain a segment sleep evaluation result corresponding to the target image output by the target neural network model; averaging the segment sleep evaluation result corresponding to each target image to obtain a first sleep evaluation result of the target user.

4. The method of claim 3, wherein, The fusing a face expression feature corresponding to the target image in the face expression feature sequence and a segment text feature corresponding to the target image to obtain a target segment feature corresponding to the target image includes: obtaining physiological index data of the target user; cutting a physiological index segment corresponding to the target image from the physiological index data based on a collection time of the target image, a number of the physiological index segment being the same as a number of the target image; fusing the face expression feature corresponding to the target image in the face expression feature sequence, the segment text feature corresponding to the target image, and the physiological index segment corresponding to the target image to obtain the target segment feature.

5. The method of claim 1, wherein, The method further includes: obtaining physiological index data of the target user; generating a second sleep evaluation result based on the physiological index data; correcting the first sleep evaluation result based on the second sleep evaluation result to obtain a target sleep evaluation result.

6. The method of claim 1, wherein, The method further includes: generating sleep improvement suggestion information based on the first sleep evaluation result; sending the sleep improvement suggestion information to the target user and / or an associated user corresponding to the target user.

7. A sleep quality assessment apparatus, characterized by comprising: The apparatus includes: an obtaining module configured to obtain a target image sequence and target audio data of a target user; an extracting module configured to perform face expression feature extraction on the target image sequence to obtain a face expression feature sequence, and perform text feature extraction on the target audio data to obtain a first text feature; a first sleep evaluation result module configured to input the face expression feature sequence and the first text feature into a target neural network model to obtain a first sleep evaluation result output by the target neural network model; the first sleep evaluation result module includes: a text feature submodule configured to obtain a second text feature obtained by performing text feature extraction on sleep quality description information of the target user; a fusion feature submodule configured to fuse the face expression feature sequence, the first text feature, and the second text feature to obtain a target fusion feature; The fusion feature evaluation submodule is configured to input the target fusion feature into the target neural network model to obtain the first sleep evaluation result output by the target neural network model; The fusion feature submodule comprises: An expression statistical index unit configured to generate an expression statistical index of the target user based on the facial expression feature sequence; A second fusion unit configured to fuse the expression statistical index, the first text feature, and the second text feature to obtain the target fusion feature; The sleep quality description information comprises voice description information and text description information, and the text feature submodule comprises: A semantic feature unit configured to extract a first semantic feature based on the voice description information and a second semantic feature based on the text description information; An information weight unit configured to determine information weights corresponding to the voice description information and the text description information respectively in a case where a similarity between the first semantic feature and the second semantic feature is less than or equal to a first threshold value; A text feature unit configured to fuse a third text feature corresponding to the voice description information and a fourth text feature corresponding to the text description information based on the information weights to obtain the second text feature.

8. An electronic device, comprising: The sleep quality evaluation device of claim 7 is used to implement the sleep quality evaluation method of any one of claims 1 to 6.

9. A readable storage medium, characterized by, The program or instruction is stored on the readable storage medium, and the program or instruction is executed by the processor to implement the sleep quality evaluation method of any one of claims 1 to 6.

Images