Sleep quality evaluation method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a sleep quality evaluation method and device, electronic equipment and a readable storage medium, and belongs to the technical field of sleep quality evaluation.The sleep quality evaluation method comprises the steps that a target image sequence and target audio data of a target user are obtained; performing facial expression feature extraction on the target image sequence to obtain a facial expression feature sequence, and performing text feature extraction on the target audio data to obtain a first text feature; and inputting the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep evaluation result output by the target neural network model. Facial expression features and text features can be analyzed through the neural network model, and a sleep quality evaluation result with high accuracy can be output.

Description

Technical Field

[0001] The present application belongs to the technical field of sleep quality assessment, and specifically relates to a sleep quality assessment method, device, electronic device and readable storage medium. Background Art

[0002] Sleep plays a vital role in the human body, and sleep problems will affect people's quality of life. In recent years, with the continuous improvement of people's living standards, people are paying more and more attention to their sleep quality.

[0003] In related technologies, wearable devices are usually used to collect physiological indicators (such as heart rate, etc.) and exercise data of users, and then make rule judgments on physiological indicators and exercise data. For example, if the user's heart rate is lower than 60 and there is no exercise within 60 seconds, the user is judged to be in a sleeping state. Then, the user's sleep quality score is calculated by combining the user's highest heart rate, exercise frequency and other sleep data during sleep.

[0004] However, the accuracy of the above sleep quality assessment methods is poor. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a sleep quality assessment method, device, electronic device and readable storage medium, which can solve the problem of poor accuracy of sleep quality assessment methods in related technologies.

[0006] In a first aspect, an embodiment of the present application provides a method for evaluating sleep quality, the method comprising:

[0007] Acquire a target image sequence and target audio data of a target user;

[0008] Performing facial expression feature extraction on the target image sequence to obtain a facial expression feature sequence, and performing text feature extraction on the target audio data to obtain a first text feature;

[0009] The facial expression feature sequence and the first text feature are input into a target neural network model to obtain a first sleep assessment result output by the target neural network model.

[0010] Optionally, inputting the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep assessment result output by the target neural network model includes:

[0011] Acquire a second text feature obtained by performing text feature extraction on the sleep quality description information of the target user;

[0012] fusing the facial expression feature sequence, the first text feature, and the second text feature to obtain a target fused feature;

[0013] The target fusion feature is input into the target neural network model to obtain the first sleep assessment result output by the target neural network model.

[0014] Optionally, the fusing the facial expression feature sequence, the first text feature and the second text feature to obtain a target fusion feature includes:

[0015] Determine the sleep index of the target user based on the facial expression feature sequence; wherein the sleep index includes at least one of the sleep duration, the number of awakenings, the awakening duration, and the duration of the sleep stage;

[0016] Extracting sleep index features based on the sleep index;

[0017] The sleep index feature, the facial expression feature sequence, the first text feature, and the second text feature are fused to obtain the target fusion feature.

[0018] Optionally, the fusing the facial expression feature sequence, the first text feature and the second text feature to obtain a target fusion feature includes:

[0019] Based on the facial expression feature sequence, generating expression statistical indicators of the target user;

[0020] The expression statistical index, the first text feature and the second text feature are fused to obtain the target fusion feature.

[0021] Optionally, the sleep quality description information includes voice description information and text description information, and the obtaining of a second text feature obtained by performing text feature extraction on the sleep quality description information of the target user includes:

[0022] Extracting a first semantic feature based on the speech description information, and extracting a second semantic feature based on the text description information;

[0023] When the similarity between the first semantic feature and the second semantic feature is less than or equal to a first threshold, determining information weights corresponding to the voice description information and the text description information respectively;

[0024] The third text feature corresponding to the speech description information and the fourth text feature corresponding to the text description information are fused based on the information weight to obtain the second text feature.

[0025] Optionally, the first text feature includes a segment text feature obtained by extracting features from a target audio segment corresponding to the target image, the target audio segment is obtained by intercepting the target audio data based on the acquisition time of the target image, and the number of the target audio segments is the same as the number of the target images, and the facial expression feature sequence and the first text feature are input into a target neural network model to obtain a first sleep assessment result output by the target neural network model, including:

[0026] Fusion of the facial expression features corresponding to the target image in the facial expression feature sequence and the segment text features corresponding to the target image to obtain a target segment feature corresponding to the target image;

[0027] Inputting the target segment feature into the target neural network model to obtain a segment sleep assessment result corresponding to the target image output by the target neural network model;

[0028] The segment sleep assessment results corresponding to the target images are averaged to obtain a first sleep assessment result of the target user.

[0029] Optionally, fusing the facial expression features corresponding to the target image in the facial expression feature sequence and the segment text features corresponding to the target image to obtain the target segment features corresponding to the target image includes:

[0030] Acquiring physiological indicator data of the target user;

[0031] Based on the acquisition time of the target image, intercepting physiological indicator segments corresponding to the target image from the physiological indicator data, the number of the physiological indicator segments being the same as the number of the target images;

[0032] The facial expression features corresponding to the target image in the facial expression feature sequence, the segment text features corresponding to the target image, and the physiological indicator segment corresponding to the target image are integrated to obtain the target segment features.

[0033] Optionally, the method further comprises:

[0034] Acquiring physiological indicator data of the target user;

[0035] generating a second sleep assessment result based on the physiological indicator data;

[0036] The first sleep evaluation result is corrected based on the second sleep evaluation result to obtain a target sleep evaluation result.

[0037] Optionally, the method further comprises:

[0038] generating sleep improvement suggestion information based on the first sleep assessment result;

[0039] The sleep improvement suggestion information is sent to the target user and / or an associated user corresponding to the target user.

[0040] In a second aspect, an embodiment of the present application provides a sleep quality assessment device, the device comprising:

[0041] An acquisition module, used to acquire a target image sequence and target audio data of a target user;

[0042] An extraction module is used to extract facial expression features from the target image sequence to obtain a facial expression feature sequence, and to extract text features from the target audio data to obtain a first text feature;

[0043] The first sleep assessment result module is used to input the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep assessment result output by the target neural network model.

[0044] Optionally, the first sleep assessment result module includes:

[0045] A text feature submodule, used to obtain a second text feature obtained by performing text feature extraction on the sleep quality description information of the target user;

[0046] A fusion feature submodule, used for fusing the facial expression feature sequence, the first text feature and the second text feature to obtain a target fusion feature;

[0047] The fusion feature evaluation submodule is used to input the target fusion feature into the target neural network model to obtain the first sleep evaluation result output by the target neural network model.

[0048] Optionally, the fusion feature submodule includes:

[0049] A sleep index unit, used to determine the sleep index of the target user based on the facial expression feature sequence; wherein the sleep index includes at least one of the sleep duration, the number of awakenings, the awakening duration, and the duration of the sleep stage;

[0050] A sleep index feature unit, used for extracting sleep index features based on the sleep index;

[0051] The first fusion unit is used to fuse the sleep index feature, the facial expression feature sequence, the first text feature and the second text feature to obtain the target fusion feature.

[0052] Optionally, the fusion feature submodule includes:

[0053] An expression statistical index unit, used for generating an expression statistical index of the target user based on the facial expression feature sequence;

[0054] The second fusion unit is used to fuse the expression statistical index, the first text feature and the second text feature to obtain the target fusion feature.

[0055] Optionally, the sleep quality description information includes voice description information and text description information, and the text feature submodule includes:

[0056] A semantic feature unit, configured to extract a first semantic feature based on the speech description information and a second semantic feature based on the text description information;

[0057] an information weight unit, configured to determine information weights corresponding to the speech description information and the text description information respectively when the similarity between the first semantic feature and the second semantic feature is less than or equal to a first threshold;

[0058] The text feature unit is used to fuse the third text feature corresponding to the speech description information and the fourth text feature corresponding to the text description information based on the information weight to obtain the second text feature.

[0059] Optionally, the first text feature includes a segment text feature obtained by extracting features of a target audio segment corresponding to the target image, the target audio segment is intercepted from the target audio data based on the acquisition time of the target image, and the number of the target audio segments is the same as the number of the target images. The first sleep assessment result module includes:

[0060] A target segment feature submodule, used for fusing the facial expression features corresponding to the target image in the facial expression feature sequence and the segment text features corresponding to the target image to obtain a target segment feature corresponding to the target image;

[0061] A fragment sleep assessment result submodule, used for inputting the target fragment feature into the target neural network model to obtain a fragment sleep assessment result corresponding to the target image output by the target neural network model;

[0062] The first sleep assessment result determination submodule is used to average the segment sleep assessment results corresponding to each of the target images to obtain a first sleep assessment result of the target user.

[0063] Optionally, the target segment feature submodule includes:

[0064] A physiological indicator data unit, used to obtain the physiological indicator data of the target user;

[0065] A physiological indicator segment unit, used for intercepting physiological indicator segments corresponding to the target image from the physiological indicator data based on the acquisition time of the target image, wherein the number of the physiological indicator segments is the same as the number of the target images;

[0066] The target segment feature fusion unit is used to fuse the facial expression features corresponding to the target image, the segment text features corresponding to the target image, and the physiological indicator segment corresponding to the target image in the facial expression feature sequence to obtain the target segment feature.

[0067] Optionally, the device further comprises:

[0068] A physiological indicator data module, used to obtain the physiological indicator data of the target user;

[0069] A second sleep assessment result module, configured to generate a second sleep assessment result based on the physiological indicator data;

[0070] The target sleep assessment result module is used to correct the first sleep assessment result based on the second sleep assessment result to obtain a target sleep assessment result.

[0071] Optionally, the device further comprises:

[0072] A sleep improvement suggestion information module, configured to generate sleep improvement suggestion information based on the first sleep assessment result;

[0073] An information sending module is used to send the sleep improvement suggestion information to the target user and / or an associated user corresponding to the target user.

[0074] In a third aspect, an embodiment of the present application provides an electronic device, comprising the sleep quality assessment device as described above, for implementing the sleep quality assessment method as described in the first aspect.

[0075] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the sleep quality assessment method of the first aspect as described above is implemented.

[0076] In an embodiment of the present application, a sleep quality assessment method is provided, including: obtaining a target image sequence and target audio data of a target user; extracting facial expression features from the target image sequence to obtain a facial expression feature sequence, extracting text features from the target audio data to obtain a first text feature; inputting the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep assessment result output by the target neural network model. Facial expression features and text features can be extracted from the image sequence and audio data of the user during sleep. Since facial expression features can reflect the user's body expression and text features can reflect the user's language expression, the user's stability during sleep can be more accurately reflected through facial expression features and text features. Therefore, by analyzing facial expression features and text features through a neural network model, a sleep quality assessment result with high accuracy can be output. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 is a flowchart of the steps of a sleep quality assessment method provided in an embodiment of the present application;

[0078] Figure 2 is a sleep quality assessment flow chart provided in an embodiment of the present application;

[0079] Figure 3 is a flowchart of another sleep quality assessment method provided by an embodiment of the present application;

[0080] Figure 4 It is a feature fusion schematic diagram provided in an embodiment of the present application;

[0081] Figure 5 is a flowchart of the steps of another sleep quality assessment method provided in an embodiment of the present application;

[0082] Figure 6 It is a logic block diagram of a sleep quality assessment device provided in an embodiment of the present application;

[0083] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0084] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0085] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0086] The sleep quality assessment method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0087] Reference Figure 1 , Figure 1 A flowchart of a sleep quality assessment method provided by an embodiment of the present application is shown. Figure 1 As shown, the specific steps include:

[0088] Step 101: Acquire a target image sequence and target audio data of a target user.

[0089] In the embodiment of the present application, a target image sequence including the target user and target audio data recorded near the target user can be collected. The above target image sequence and target audio data can be collected during the target user's sleep period. The target user's sleep period can be set by the target user independently, or can be obtained through the sleep monitoring function of the target user's wearable device or other electronic device, which is not specifically limited in the embodiment of the present application.

[0090] The target image sequence can be presented in the form of a video, and the target images in the target image sequence are video frames; the target image sequence can also be composed of a series of target images shot at preset intervals, for example, the target images can be shot at intervals of 1 minute during the target user's sleep period, and these target images can constitute the above-mentioned target image sequence. The target image sequence can be collected by a camera set or placed near the target user, such as a surveillance camera, a mobile phone camera, etc., and the embodiments of the present application are not specifically limited.

[0091] The target audio data may be in the form of audio recorded continuously during the target user's sleep period. The target audio data may be collected by an electronic device with a microphone that is set or placed around the target user, such as a monitor, a mobile phone, a laptop computer, a smart bracelet, etc., which is not specifically limited in the embodiments of the present application.

[0092] Step 102: extract facial expression features from the target image sequence to obtain a facial expression feature sequence, and extract text features from the target audio data to obtain a first text feature.

[0093] In an embodiment of the present application, facial expression features can be extracted from a target image sequence to obtain a facial expression feature sequence. The facial expression features may include but are not limited to at least one of the features of eye opening degree, mouth opening degree, eyebrow position, etc., which are not specifically limited in the embodiment of the present application. Since a facial expression feature sequence can be extracted from the target image sequence, and the facial expression feature sequence can reflect the changes in the expression of the target user during sleep, for example, if the target user frequently opens his eyes and opens and closes his mouth during sleep, it means that the target user's sleep process is not stable and the sleep quality is low.

[0094] Exemplarily, the trained facial expression feature extraction model can be used to extract features from the target images in the target image sequence, and each target image in the target image sequence is input into the facial expression feature extraction model to obtain facial expression features corresponding to each target image output by the facial expression feature extraction model. Facial expression features may include but are not limited to happiness, sadness, anger, etc. The facial expression extraction model can detect and locate the face area in the image, extract low-level features (such as edge and texture features in the face area) through the convolution layer, and then reduce the dimension and calculation amount through the pooling layer to extract higher-level facial features, and then gradually capture complex facial features through multi-layer convolution and pooling superposition, and finally use the fully connected layer to map the extracted features to specific expression categories to achieve recognition and extraction of facial expression features.

[0095] It should be noted that before extracting facial expression features from the target image sequence, the target images in the target image sequence may be preprocessed to improve the accuracy of facial expression feature extraction. The preprocessing may include but is not limited to image scaling and cropping, graying, face detection and alignment, image noise reduction, etc., which are not specifically limited in the embodiments of the present application.

[0096] In an embodiment of the present application, text feature extraction can also be performed on the target audio data to obtain a first text feature. The first text feature may include but is not limited to at least one of keyword features, semantic features, word frequency features, and tone features. Since the target audio data is collected during the target user's sleep period, the target audio data may include unconscious speech content such as the target user's mumbling and dream talk during sleep, so that the extracted first text feature can reflect the target user's sleep condition during sleep. If the first text feature indicates that the target user has negative emotions such as panic and tension, or certain negative keyword features frequently appear in the first text feature, it means that the target user's sleep process is not stable and the sleep quality is low.

[0097] Specifically, the target audio data can be processed and analyzed by natural language processing technology (NLP) to extract the first text feature. First, the target audio data can be converted into text data using an application programming interface (Application Programming Interface, API) with integrated language recognition, such as Google Speech API, and then the text data is input into a trained text feature extraction model to obtain the first text feature output by the text feature extraction model. Among them, the text feature extraction model may include but is not limited to BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), TF-IDF (Term Frequency-Inverse Document Frequency), etc., which are not specifically limited in the embodiments of the present application.

[0098] It should be noted that before extracting text features from the target audio data, the target audio data may also be preprocessed to improve the accuracy of speech recognition and the effect of subsequent feature extraction. The preprocessing may include but is not limited to audio sampling rate adjustment, noise elimination, gain adjustment, speech enhancement, etc., which are not specifically limited in the embodiments of the present application.

[0099] Step 103: input the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep assessment result output by the target neural network model.

[0100] In an embodiment of the present application, a target neural network model for sleep quality assessment can be trained, and the target neural network model can use a facial expression feature sequence and a first text feature as model input, and assess the sleep quality of the target user based on the facial expression feature sequence and the first text feature, and output the first sleep assessment result of the target user. The above-mentioned target neural network model may include but is not limited to a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), a long short-term memory network (LSTM), a temporal convolutional network (TCN), etc., and the embodiment of the present application does not specifically limit it.

[0101] Exemplarily, the target neural network model may include a temporal feature extraction layer, a multimodal fusion layer, a fully connected layer, and an output layer. Among them, the temporal feature extraction layer can be used to capture the temporal features of the facial expression feature sequence, the multimodal fusion layer can be used to splice the above-mentioned temporal features and the first text features to obtain fusion features, the fully connected layer is used to map the fusion features to the final output space, and the output layer is used to output the final first sleep assessment result using an activation function. Among them, the first sleep assessment result may include a sleep quality score (for example, 90 points), and may also include a sleep quality rating (for example, poor, good, general, etc.), which is not specifically limited in the embodiments of the present application.

[0102] It should be noted that the target neural network model may not include a multimodal fusion layer. The above-mentioned fusion features can be obtained outside the model and directly input into the target neural network model, thereby improving the model reasoning efficiency.

[0103] Furthermore, in order to improve the accuracy of the target neural network model, the target neural network model can be trained based on model-agnostic meta-learning (MAML). For example, multiple tasks can be constructed first, each task containing different training samples and test samples. Each task corresponds to the sleep data of a user. The data in the task includes the fused feature vector (F_combined) and the corresponding sleep quality score. Then initialize the model parameter θ, for each task T i , respectively select the training data set And the test dataset Inner loop stage: training data set for each task Perform gradient descent on the model to update the model parameters to obtain task-specific parameters θ i : Among them, the loss function Represents task T i The learning rate α controls the update step. Outer loop phase: In the test data set of each task Evaluate model performance, calculate loss and accumulate meta-gradients, and update meta-model parameters: Among them, β is the meta-learning rate, which is used to adjust the parameter update step size. Through multiple rounds of meta-training, the initial parameters of the model are continuously optimized so that it can quickly adapt and generalize on different tasks. The model performance is evaluated in the final test set, and evaluation indicators (such as mean square error, R 2 Through the above steps, MAML can effectively use the fused data to evaluate the sleep quality of users. Through the meta-learning process, the model significantly improves its adaptability and prediction accuracy on different user data.

[0104] In addition to using MAML for training, the above-mentioned target neural network model can also use supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc. Technical personnel can flexibly select the appropriate learning method according to actual business needs, and the embodiments of this application are not specifically limited.

[0105] Specifically, sample sleep data of a sample user can be obtained, wherein the sample sleep data may include a sample image sequence and sample audio data, facial expression feature extraction is performed on the sample image sequence to obtain a facial expression feature sequence of the sample image sequence, text feature extraction is performed on the sample audio data to obtain a first text feature of the sample audio data, and then the facial expression feature sequence and the first text feature are input into an initial neural network model to obtain a first evaluation result output by the initial neural network model; a model loss value is determined based on the first evaluation result and a sleep evaluation label corresponding to the sample user, and model parameters of the initial neural network model are adjusted based on the model loss value to obtain a target neural network model.

[0106] It should be noted that the steps of extracting facial expression features from the target image sequence to obtain a facial expression feature sequence, and extracting text features from the target audio data to obtain a first text feature can be performed outside the model or inside the model. For example, feature extraction can be performed on the target image sequence and the target audio data through a feature extraction neural network model, so that the feature extraction neural network model and the target neural network model can constitute a complete neural network model. During the model training process, the sample sleep data and sample sleep quality labels of sample users can be used to jointly train the feature extraction neural network model and the target neural network model. The embodiments of the present application do not make specific limitations on this, and technicians can set the model structure according to actual needs.

[0107] Reference Figure 2 , Figure 2 A sleep quality assessment flow chart provided in an embodiment of the present application is shown. Figure 2 As shown, first, sleep data is obtained through the user client, facial expression recognition is performed on the target image in the sleep data, and voice content recognition is performed on the target audio therein to obtain feature results, and then the feature results are encrypted and uploaded to the cloud server. The cloud server calls the target neural network model to analyze the above feature results to obtain sleep evaluation results, and then the sleep evaluation results are encrypted and sent to the user client for display. It should be noted that the sleep quality evaluation function provided in the embodiment of the present application runs after the user turns it on, and the above sleep data is obtained after the user authorizes the collection.

[0108] In summary, a sleep quality assessment method provided by an embodiment of the present application includes: obtaining a target image sequence and target audio data of a target user; extracting facial expression features from the target image sequence to obtain a facial expression feature sequence, extracting text features from the target audio data to obtain a first text feature; inputting the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep assessment result output by the target neural network model. Facial expression features and text features can be extracted from the image sequence and audio data of the user during sleep. Since facial expression features can reflect the user's body expression and text features can reflect the user's language expression, the facial expression features and text features can more accurately reflect the user's stability during sleep. Therefore, by analyzing the facial expression features and text features through a neural network model, a sleep quality assessment result with a higher accuracy can be output.

[0109] like Figure 3 As shown, Figure 3 A flowchart of another method for evaluating sleep quality provided in an embodiment of the present application, the method may include:

[0110] Step 201: Acquire a target image sequence and target audio data of a target user.

[0111] This step can refer to step 101 and will not be described in detail in this embodiment of the present application.

[0112] Optionally, the target user may wear a wearable device, such as a smart bracelet, etc., through which the physiological indicators of the target user can be obtained, and whether the target user is in a sleeping state can be judged by the physiological indicators, and when the target user is in a sleeping state, the target image sequence and target audio data are collected by corresponding sensors, so that the target image sequence and target audio data more accurately reflect the target user's sleep process. Among them, the above-mentioned physiological indicators may include but are not limited to heart rate, respiratory rate, blood oxygen saturation, blood pressure, etc., and are not specifically limited in the embodiments of the present application.

[0113] Step 202: extract facial expression features from the target image sequence to obtain a facial expression feature sequence, and extract text features from the target audio data to obtain a first text feature.

[0114] This step can refer to the above-mentioned step 102, and will not be described in detail in this embodiment of the present application.

[0115] Step 203: Acquire a second text feature obtained by performing text feature extraction on the sleep quality description information of the target user.

[0116] In order to further improve the accuracy of sleep quality assessment, in the embodiment of the present application, sleep quality description information provided by the target user can also be obtained, and the sleep quality description information can be used to participate in the sleep quality assessment process. The sleep quality description information can include voice description information and / or text description information.

[0117] The above-mentioned voice description information can be pre-entered by the target user. For example, after the user logs in to the system, a questionnaire asking about sleep quality can be sent to the user, and the user can reply to the questionnaire through voice, text, options, etc. to obtain the sleep quality description information of the target user. Through the sleep quality description information, the user's subjective feelings about his or her own sleep quality can be understood, thereby adding information on the subjective feeling dimension in the subsequent sleep quality assessment process, which helps to improve the accuracy of sleep quality assessment for the target user.

[0118] Specifically, text feature extraction can be performed on the sleep quality description information of the target user to obtain a second text feature. The second text feature may include but is not limited to at least one of a keyword feature, a semantic feature, a word frequency feature, a tone feature, etc. Exemplarily, if some negative keyword features appear in the second text feature indication, it indicates that the target user's subjective sleep quality is low.

[0119] Optionally, when the sleep quality description information includes both voice description information and text description information, step 203 may include:

[0120] Sub-step 2031: extracting a first semantic feature based on the speech description information, and extracting a second semantic feature based on the text description information.

[0121] In an embodiment of the present application, if the sleep quality description information includes both voice description information and text description information, the first semantic feature can be extracted based on the voice description information, and the second semantic feature can be extracted based on the text description information. For the voice description information, the voice description information can be first converted into text information, and then the text information can be input into the semantic recognition model to obtain its first semantic feature; for the text description information, it can be directly input into the semantic recognition model to obtain its second semantic feature.

[0122] Sub-step 2032: when the similarity between the first semantic feature and the second semantic feature is less than or equal to a first threshold, determine the information weights corresponding to the voice description information and the text description information respectively.

[0123] In an embodiment of the present application, taking into account that the contents of the voice description information and the text description information may be inconsistent or even conflicting, in order to improve the accuracy of determining the second text feature, the first semantic feature and the second semantic feature can be first compared to obtain the similarity between the two. If the similarity is high, it means that the voice description information and the text description information express similar meanings. In this case, the third text feature corresponding to the voice description information and the fourth text feature corresponding to the text description information can be directly fused based on the information weight to obtain the second text feature. If the similarity is low, it means that the meanings expressed by the voice description information and the text description information are quite different, or even completely opposite. At this time, the information weights corresponding to the voice description information and the text description information can be calculated respectively, and the third text feature and the fourth text feature can be weightedly fused to avoid the inaccuracy caused by direct fusion.

[0124] Specifically, in one embodiment, since the text description information is input by the user in written form and generally has a high accuracy, the information weights corresponding to the voice description information and the text description information can be determined according to the above similarity, so that the lower the similarity, the lower the information weight corresponding to the voice description information, and the higher the information weight corresponding to the text description information. Exemplarily, the above similarity can be normalized to obtain a normalized similarity value, and the normalized similarity value is used as the information weight of the voice description information, and "1-normalized similarity value" is used as the information weight of the text description information.

[0125] In another embodiment, the description of sleep quality corresponding to the voice description information and the text description information can be determined based on the first semantic feature and the second semantic feature, and then the historical sleep assessment results of the target user are obtained, and the information weights corresponding to the voice description information and the text description information can be determined based on the similarity between the description of sleep quality and the historical sleep assessment results.

[0126] Sub-step 2033: fusing the third text feature corresponding to the speech description information and the fourth text feature corresponding to the text description information based on the information weight to obtain the second text feature.

[0127] In the embodiment of the present application, the third text feature corresponding to the speech description information and the fourth text feature corresponding to the text description information may be fused in a weighted average manner to obtain the second text feature.

[0128] By extracting the first semantic feature based on the speech description information and the second semantic feature based on the text description information; when the similarity between the first semantic feature and the second semantic feature is less than or equal to the first threshold, determining the information weights corresponding to the speech description information and the text description information respectively; fusing the third text feature corresponding to the speech description information and the fourth text feature corresponding to the text description information based on the information weights to obtain the second text feature. When there is a disagreement between the speech description information and the text description information, the weights corresponding to each description information can be determined, and the second text feature can be obtained by weighted fusion, which helps to improve the accuracy of the second text feature.

[0129] Step 204: fuse the facial expression feature sequence, the first text feature, and the second text feature to obtain a target fused feature.

[0130] In an embodiment of the present application, the facial expression feature sequence, the first text feature, and the second text feature can be fused to obtain a target fusion feature. Specifically, the feature vectors corresponding to the facial expression feature sequence, the first text feature, and the second text feature can be concatenated to obtain a feature vector of the target fusion feature. The above features can also be fused in other ways, such as by a feature fusion neural network model, according to preset fusion weights corresponding to various features, etc., which are not specifically limited in the embodiment of the present application.

[0131] Reference Figure 4 , Figure 4 A feature fusion schematic diagram provided in an embodiment of the present application is shown. Figure 4As shown, after the user logs in, it is determined whether the user is logging in for the first time. If it is the first time to log in, the user is asked whether to turn on the sleep quality assessment function. After the user turns on the function and authorizes the acquisition of sleep data, the sleep data is collected; the facial features of the target image in the sleep data are captured, and the facial expression feature sequence is identified based on the facial features; the target audio in the sleep data can be voice recognized through the voice recognition API to obtain the first text feature; the sleep quality description information can be identified through natural language processing technology to obtain the second text feature; then the facial expression feature sequence, the first text feature and the second text feature are feature fused to obtain the target fusion feature; the target fusion feature is input into the target neural network model to obtain the sleep assessment result output by the target neural network model. If the user is not logging in for the first time, the physiological indicator data and / or sleep quality description information collected by the wearable device can be used to preliminarily determine whether the user's sleep quality has problems. If the judgment result is that there may be problems with the sleep quality, the user is notified and asked whether the user turns on the sleep assessment function. The sleep data is collected after the user authorizes.

[0132] Optionally, step 204 may include:

[0133] Step A1, determining the sleep index of the target user based on the facial expression feature sequence; wherein the sleep index includes at least one of sleep duration, awakening times, awakening duration and sleep stage duration.

[0134] In an embodiment of the present application, since the facial expression feature sequence can reflect the facial expression of the target user at various times during sleep, the facial expression feature sequence can be analyzed to determine the sleep index of the target user, wherein the sleep index may include but is not limited to at least one of sleep duration, number of awakenings, awakening duration, and duration of sleep stages, and the sleep stage may include but is not limited to non-rapid eye movement sleep stage (NREM) and rapid eye movement sleep stage (REM). The non-rapid eye movement sleep stage can be further divided into falling asleep stage, light sleep stage, deep sleep stage, etc.

[0135] Specifically, the target user's sleep state at the corresponding time can be determined by facial expression features. For example, if the target user's facial expression features change continuously for more than a preset number of times within a preset time, it can be determined that the target user is in the stage of falling asleep. If the number of continuous changes is less than or equal to the preset number of times and greater than 0 times, it can be determined that the target user is in the stage of light sleep. If the number of continuous changes is equal to 0 times, it can be determined that the target user is in the stage of deep sleep. If the target user's eye opening angle is greater than a preset angle, it can be determined that the target user is in the stage of waking up. The sleep stage determination strategy of the above example can be used to count sleep indicators such as the duration of each sleep stage, total sleep time, number of awakenings, and awakening time.

[0136] Step A3: extracting sleep index features based on the sleep index.

[0137] In the embodiment of the present application, the sleep index can be feature extracted to obtain the sleep index feature. Specifically, each sleep index can be vectorized to obtain the feature vector of each sleep index, and then the feature vectors of each sleep index can be spliced ​​to obtain the sleep index feature.

[0138] Step A2, fusing the sleep index feature, the facial expression feature sequence, the first text feature and the second text feature to obtain the target fusion feature.

[0139] In an embodiment of the present application, the sleep index feature, the facial expression feature sequence, the first text feature, and the second text feature can be fused to obtain a target fusion feature. Specifically, the feature vectors corresponding to the sleep index feature, the facial expression feature sequence, the first text feature, and the second text feature can be concatenated to obtain a feature vector of the target fusion feature. The above features can also be fused in other ways, such as fusion through a feature fusion neural network model, fusion according to preset fusion weights corresponding to various features, etc., which are not specifically limited in the embodiment of the present application.

[0140] Based on the facial expression feature sequence, the sleep index of the target user is determined; wherein the sleep index includes at least one of the sleep duration, the number of awakenings, the awakening duration, and the duration of the sleep stage; based on the sleep index, a sleep index feature is extracted; and the sleep index feature, the facial expression feature sequence, the first text feature, and the second text feature are fused to obtain the target fusion feature. The sleep index feature can be introduced into the target fusion feature, and the sleep index feature is generated based on a more intuitive facial expression feature sequence and has a higher accuracy, so that the target fusion feature can more accurately reflect the sleep condition of the target user, which helps to improve the accuracy of determining the user's sleep quality based on the target fusion feature.

[0141] Optionally, the facial expression feature sequence may be further processed to extract expression statistical indicators, and the information contained in the front and back facial expression features in the facial expression feature sequence may be fully utilized. Step 204 may include:

[0142] Step B1, generating expression statistical indicators of the target user based on the facial expression feature sequence.

[0143] In the embodiment of the present application, the expression statistics of the target user can be generated based on the facial expression feature sequence. The expression statistics may include but are not limited to the number of expression changes, the frequency of expression changes, the duration of various expressions, etc., which are not specifically limited in the embodiment of the present application.

[0144] Specifically, the number of inconsistent adjacent facial expression feature pairs in the facial expression feature sequence can be counted to obtain the number of expression changes; the frequency of expression changes can be determined based on the ratio of the total statistical duration of the facial expression feature sequence and the number of expression changes; the duration of each type of user's facial expression can be determined based on the acquisition interval between target images corresponding to adjacent facial expression features in the facial expression feature sequence and the product of the number of appearances of each type of facial expression feature in the facial expression feature sequence.

[0145] Step B2, fusing the expression statistical index, the first text feature and the second text feature to obtain the target fusion feature.

[0146] In an embodiment of the present application, the expression statistical index, the first text feature and the second text feature can be fused to obtain a target fused feature. Specifically, the expression statistical index can be first feature vectorized, and the feature vector of the expression statistical index can be spliced ​​with the vector of the first text feature and the second text feature to obtain the target fused feature.

[0147] It should be noted that in the schemes in which facial expression feature sequences participate in feature fusion, expression statistical indicators can be generated based on the facial expression feature sequences, and then the facial expression feature sequences are replaced by the expression statistical indicators to participate in feature fusion to obtain corresponding target fusion features. Through the expression statistical indicators, the key features in the facial expression feature sequences can be extracted, and the use of longer facial expression feature sequences to participate in feature fusion can be avoided, which can effectively reduce the feature length of the target fusion features and improve the efficiency of subsequent sleep quality judgment based on the target fusion features.

[0148] Step 205: input the target fusion feature into the target neural network model to obtain the first sleep assessment result output by the target neural network model.

[0149] In an embodiment of the present application, the target fusion feature can be input into the target neural network model to obtain a first sleep assessment result output by the target neural network model.

[0150] It should be noted that the target neural network model may have different model structures and training methods due to different input data. For example, if the input data of the target neural network model is a facial expression feature sequence and a first text feature, the target neural network model may include a feature fusion layer and an output layer. If the input data of the target neural network model is a target fusion feature, the target neural network model may include an output layer. The above-mentioned process of extracting features and fusing features to obtain target fusion features can be implemented by a feature extraction module and a feature fusion module outside the target neural network model. A feature extraction layer and a feature fusion layer can also be designed in the target neural network model. The target neural network model directly receives sample image sequences, sample audio data, and sleep quality description information, and performs feature extraction and feature fusion within the model. The embodiments of the present application do not make specific limitations.

[0151] The second text feature is obtained by extracting text features from the sleep quality description information of the target user; the facial expression feature sequence, the first text feature and the second text feature are fused to obtain the target fusion feature; the target fusion feature is input into the target neural network model to obtain the first sleep assessment result output by the target neural network model. The target fusion feature can be obtained according to the subjective sleep feeling of the target user, so that the target fusion feature contains feature information related to the subjective sleep feeling of the user, so that the sleep quality analysis based on the target fusion feature can achieve higher accuracy.

[0152] Step 206: Acquire the physiological indicator data of the target user.

[0153] In the embodiment of the present application, the physiological indicators of the target user can also be collected through the wearable device of the target user, and the physiological indicator data corresponding to the target image sequence and the target audio data collection period of the target user can be obtained, and the first sleep assessment result can be corrected by these physiological indicator data to improve the accuracy of the sleep quality assessment. Among them, the above-mentioned physiological indicators may include but are not limited to heart rate, respiratory rate, blood oxygen saturation, blood pressure, etc. Wearable devices may include but are not limited to smart bracelets, vital sign monitors, smart glasses, etc., which are not specifically limited in the embodiment of the present application.

[0154] Step 207: Generate a second sleep assessment result based on the physiological indicator data.

[0155] In an embodiment of the present application, a second sleep assessment result can be generated based on the physiological indicator data. Specifically, in one embodiment, physiological statistical data can be generated based on the physiological indicator data, and then the physiological statistical data can be input into the physiological indicator sleep assessment model to obtain a second sleep assessment result output by the physiological indicator sleep assessment model. Among them, the physiological statistical data may include but are not limited to: average heart rate, maximum heart rate, minimum heart rate, heart rate change rate, average respiratory rate, maximum respiratory rate, minimum respiratory rate, respiratory rate change rate, average blood oxygen saturation, highest blood pressure, etc. The physiological indicator sleep assessment model may include but is not limited to a mathematical model, a conditional judgment model, a decision tree model, a neural network model, etc., which is not specifically limited in the embodiment of the present application.

[0156] In another embodiment, the physiological indicator data can be directly input into the physiological indicator sleep assessment model, and the physiological indicator sleep assessment model extracts features from the physiological indicator data to obtain physiological indicator features, and outputs a second sleep assessment result according to the physiological indicator features. The physiological indicator sleep assessment model can include a neural network model.

[0157] Step 208: correct the first sleep assessment result based on the second sleep assessment result to obtain a target sleep assessment result.

[0158] In an embodiment of the present application, the second sleep assessment result can be used to correct the first sleep assessment result to obtain a target sleep assessment result. Specifically, the second sleep assessment result and the first sleep assessment result can be averaged to obtain a target sleep assessment result. The second sleep assessment result and the first sleep assessment result can also be weighted averaged to obtain a target sleep assessment result, wherein the weight of the weighted average can be a preset value, for example, the weight of the second sleep assessment result can be set to 0.2, and the weight of the first sleep assessment result can be set to 1; the weight of the weighted average can also be determined according to the number of indicator types contained in the physiological indicator data. If the number of indicator types contained in the physiological indicator data is more, the weight of the second sleep assessment result can be set to be higher relative to the weight of the first sleep assessment result. If the number of indicator types contained in the physiological indicator data is less, the weight of the second sleep assessment result can be set to be lower relative to the weight of the first sleep assessment result.

[0159] It should be noted that in the embodiment of the present application, feature extraction can also be performed on physiological indicator data to obtain physiological indicator features, and the physiological indicator features can be used to participate in the above-mentioned feature fusion process to obtain the above-mentioned target fusion features, so that the target neural network model can output a more accurate first sleep assessment result based on the target fusion features.

[0160] By acquiring the physiological indicator data of the target user; generating a second sleep assessment result based on the physiological indicator data; and correcting the first sleep assessment result based on the second sleep assessment result, a target sleep assessment result is obtained. The second sleep assessment result determined based on the traditional physiological indicator characteristics can be used to correct the first sleep assessment result to obtain the target sleep assessment index, which helps to improve the accuracy of the target sleep assessment index.

[0161] Optionally, the sleep assessment result may be processed through the following steps C1 to C2:

[0162] Step C1, generating sleep improvement suggestion information based on the first sleep assessment result.

[0163] In an embodiment of the present application, sleep improvement suggestion information corresponding to the target user can also be generated based on the above sleep assessment results (such as the first sleep assessment result or the target sleep assessment result). Specifically, in one embodiment, a correspondence between different sleep assessment results and preset suggestion information can be established. For example, the sleep assessment result can include a sleep quality score or a sleep quality level, and preset suggestion information corresponding to different sleep quality scores or sleep quality levels can be preset. After obtaining the sleep assessment result of the target user, the corresponding preset suggestion information can be queried to obtain sleep improvement suggestion information.

[0164] In another embodiment, the improvement suggestion model can be pre-trained, and the improvement suggestion model can be trained based on sample sleep quality scores and corresponding sample sleep improvement suggestion information set by expert experience. The sleep assessment results of the target user can be input into the improvement suggestion model to obtain the sleep improvement suggestion information output by the improvement suggestion model.

[0165] It should be noted that the specific content of the above-mentioned sleep improvement suggestion information may include but is not limited to bedtime suggestions, sleep duration suggestions, etc., and the expression form may include but is not limited to text information, voice information, graphic information, etc., which is not specifically limited in the embodiments of the present application.

[0166] Step C2: sending the sleep improvement suggestion information to the target user and / or an associated user corresponding to the target user.

[0167] In an embodiment of the present application, after obtaining the sleep improvement suggestion information, the sleep improvement suggestion information can be sent to the target user and / or the associated user corresponding to the target user. The associated user may include but is not limited to the guardian user of the target user, the authorized user of the target user, etc., which is not specifically limited in the embodiment of the present application. In addition, the first sleep assessment result of the target user or the target sleep assessment result may also be sent to the target user and / or the associated user.

[0168] By generating sleep improvement suggestion information based on the first sleep assessment result and sending the sleep improvement suggestion information to the target user and / or the associated users corresponding to the target user, sleep improvement suggestions can be provided based on more accurate sleep assessment results, so that the sleep improvement suggestions also have high accuracy and can be synchronized to the target user and / or the associated users, thereby facilitating the target user to adjust his or her sleeping habits according to the sleep improvement suggestions.

[0169] In summary, another sleep quality assessment method provided by the embodiment of the present application includes: obtaining a target image sequence and target audio data of a target user; extracting facial expression features from the target image sequence to obtain a facial expression feature sequence, extracting text features from the target audio data to obtain a first text feature; inputting the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep assessment result output by the target neural network model. Facial expression features and text features can be extracted based on the image sequence and audio data of the user during sleep. Since facial expression features can reflect the user's body expression and text features can reflect the user's language expression, the facial expression features and text features can more accurately reflect the user's stability during sleep. Therefore, by analyzing the facial expression features and text features through the neural network model, a sleep quality assessment result with higher accuracy can be output.

[0170] like Figure 5 As shown, Figure 5 A flowchart of the steps of another sleep quality assessment method provided in an embodiment of the present application, the method may include:

[0171] Step 301: Acquire a target image sequence and target audio data of a target user.

[0172] This step can refer to step 101 and will not be described in detail in this embodiment of the present application.

[0173] Step 302: extract facial expression features from the target image sequence to obtain a facial expression feature sequence, and extract text features from the target audio data to obtain a first text feature.

[0174] This step can be referred to as step 102, and will not be described in detail in this embodiment of the present application.

[0175] Step 303: The facial expression features corresponding to the target image in the facial expression feature sequence and the segment text features corresponding to the target image are fused to obtain the target segment features corresponding to the target image.

[0176] In an embodiment of the present application, the first text feature may include a segment text feature obtained by extracting features from a target audio segment corresponding to a target image, the target audio segment is obtained from the target audio data based on the acquisition time of the target image, and the number of target audio segments is the same as the number of target images. Exemplarily, the shooting time of the target image sequence may be from 23:00 to 7:00 the next day, a total of 8 hours, and the target image shooting interval in the target image sequence may be 1 minute; accordingly, the target audio data may include continuous audio recorded from 23:00 to 7:00 the next day, and the target audio data may be segmented into 1-minute segments based on the acquisition time of the target image to obtain a target audio segment corresponding to each target image, thereby obtaining target audio segments equal to the number of target images, and by extracting text features from the target audio segment, the segment text feature corresponding to the target audio segment may be obtained, that is, the segment text features of all target audio segments may constitute the first text feature corresponding to the target audio data.

[0177] The facial expression features corresponding to the target image in the facial expression feature sequence and the segment text features of the target audio segment corresponding to the target image can be fused to obtain the target segment features corresponding to the target image. The above steps are repeated until the target segment features corresponding to all the target images in the target image sequence are obtained.

[0178] Optionally, step 303 may include:

[0179] Sub-step 3031, obtaining physiological indicator data of the target user.

[0180] In the embodiment of the present application, the physiological indicators of the target user can also be collected through the wearable device of the target user, and the physiological indicator data corresponding to the target image sequence and the target audio data collection period of the target user can be obtained. Among them, the above physiological indicators may include but are not limited to heart rate, respiratory rate, blood oxygen saturation, blood pressure, etc. Wearable devices may include but are not limited to smart bracelets, vital sign monitors, smart glasses, etc., which are not specifically limited in the embodiment of the present application.

[0181] Sub-step 3032: based on the acquisition time of the target image, extracting physiological indicator segments corresponding to the target image from the physiological indicator data, wherein the number of the physiological indicator segments is the same as the number of the target images.

[0182] In an embodiment of the present application, the physiological indicator segments corresponding to the target image can be intercepted from the physiological indicator data based on the acquisition time of the target image, and the number of physiological indicator segments is the same as the number of target images. Exemplarily, the shooting time of the target image sequence can be from 23:00 to 7:00 the next day, a total of 8 hours, and the target image shooting interval in the target image sequence can be 1 minute; accordingly, the physiological indicator data can be collected from 23:00 to 7:00 the next day, and the collection time interval can also be 1 minute; based on the acquisition time of the target image, the physiological indicator data can be divided into 1-minute segments to obtain the physiological indicator segments corresponding to each target image, so that the physiological indicator segments equal to the number of target images can be obtained.

[0183] Sub-step 3033, fusing the facial expression features corresponding to the target image in the facial expression feature sequence, the segment text features corresponding to the target image, and the physiological indicator segment corresponding to the target image to obtain the target segment features.

[0184] In an embodiment of the present application, feature extraction can be performed on the physiological indicator segment to obtain the physiological indicator segment feature, and the physiological indicator segment feature is used to participate in the above-mentioned feature fusion process to obtain the above-mentioned target fusion feature, that is, the facial expression features corresponding to the target image in the facial expression feature sequence, the segment text features corresponding to the target image, and the physiological indicator segment features corresponding to the target image are fused to obtain the target segment feature, so that the target neural network model can output a more accurate segment sleep assessment result based on the target segment feature.

[0185] By acquiring the physiological indicator data of the target user; based on the acquisition time of the target image, the physiological indicator segments corresponding to the target image are intercepted from the physiological indicator data, and the number of physiological indicator segments is the same as the number of target images; the facial expression features corresponding to the target image in the facial expression feature sequence, the segment text features corresponding to the target image, and the physiological indicator segments corresponding to the target image are integrated to obtain the target segment features, which can improve the feature dimension of the target segment features, introduce physiological indicator features into the target segment features, and improve the accuracy of the fragmented sleep assessment results obtained based on the target segment feature analysis, thereby improving the accuracy of the sleep quality assessment for the target user.

[0186] Step 304: input the target segment feature into the target neural network model to obtain the segment sleep assessment result corresponding to the target image output by the target neural network model.

[0187] In an embodiment of the present application, the target neural network model can receive the target segment features as input and output the segment sleep assessment results. The target segment features corresponding to each target image can be input into the target neural network model to obtain the segment sleep assessment results corresponding to each target image. Among them, the segment sleep assessment results are similar to the first sleep assessment results in the aforementioned embodiment, and can include a sleep quality score value or a sleep quality rating.

[0188] It should be noted that the above-mentioned target fragment features can also be obtained by fusing other features extracted based on other types of data. The feature types involved in the feature fusion to obtain the target fragment features can be the same as the feature types involved in the target fusion features in the aforementioned embodiments, and the embodiments of this application will not be repeated here.

[0189] Step 305 : Average the segment sleep assessment results corresponding to the target images to obtain a first sleep assessment result of the target user.

[0190] After obtaining the segment sleep evaluation results corresponding to each target image, each segment sleep evaluation result may be averaged to obtain the first sleep evaluation result of the target user.

[0191] By fusing the facial expression features corresponding to the target image in the facial expression feature sequence and the segment text features corresponding to the target image, the target segment features corresponding to the target image are obtained; the target segment features are input into the target neural network model to obtain the segment sleep assessment results corresponding to the target image output by the target neural network model; the segment sleep assessment results corresponding to each target image are averaged to obtain the first sleep assessment result of the target user. The target segment features of multiple sleep segments of the target user during sleep can be generated, and the target neural network model outputs the corresponding segment sleep assessment results according to each sleep segment. The target user's sleep process can be refined into multiple segments and evaluated separately. Finally, the assessment results of all sleep segments are summarized to obtain the first sleep assessment result of the target user, which helps to improve the accuracy of the first sleep assessment result.

[0192] like Figure 6 As shown, Figure 6 A logic block diagram of a sleep quality assessment device provided in an embodiment of the present application, the sleep quality assessment device 600 may include:

[0193] An acquisition module 601 is used to acquire a target image sequence and target audio data of a target user;

[0194] An extraction module 602 is used to extract facial expression features from the target image sequence to obtain a facial expression feature sequence, and to extract text features from the target audio data to obtain a first text feature;

[0195] The first sleep assessment result module 603 is used to input the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep assessment result output by the target neural network model.

[0196] Optionally, the first sleep assessment result module includes:

[0197] A text feature submodule, used to obtain a second text feature obtained by performing text feature extraction on the sleep quality description information of the target user;

[0198] A fusion feature submodule, used for fusing the facial expression feature sequence, the first text feature and the second text feature to obtain a target fusion feature;

[0199] The fusion feature evaluation submodule is used to input the target fusion feature into the target neural network model to obtain the first sleep evaluation result output by the target neural network model.

[0200] Optionally, the fusion feature submodule includes:

[0201] A sleep index unit, used to determine the sleep index of the target user based on the facial expression feature sequence; wherein the sleep index includes at least one of the sleep duration, the number of awakenings, the awakening duration, and the duration of the sleep stage;

[0202] A sleep index feature unit, used for extracting sleep index features based on the sleep index;

[0203] The first fusion unit is used to fuse the sleep index feature, the facial expression feature sequence, the first text feature and the second text feature to obtain the target fusion feature.

[0204] Optionally, the fusion feature submodule includes:

[0205] An expression statistical index unit, used for generating an expression statistical index of the target user based on the facial expression feature sequence;

[0206] The second fusion unit is used to fuse the expression statistical index, the first text feature and the second text feature to obtain the target fusion feature.

[0207] Optionally, the sleep quality description information includes voice description information and text description information, and the text feature submodule includes:

[0208] A semantic feature unit, configured to extract a first semantic feature based on the speech description information and a second semantic feature based on the text description information;

[0209] an information weight unit, configured to determine information weights corresponding to the speech description information and the text description information respectively when the similarity between the first semantic feature and the second semantic feature is less than or equal to a first threshold;

[0210] The text feature unit is used to fuse the third text feature corresponding to the speech description information and the fourth text feature corresponding to the text description information based on the information weight to obtain the second text feature.

[0211] Optionally, the first text feature includes a segment text feature obtained by extracting features of a target audio segment corresponding to the target image, the target audio segment is intercepted from the target audio data based on the acquisition time of the target image, and the number of the target audio segments is the same as the number of the target images. The first sleep assessment result module includes:

[0212] A target segment feature submodule, used for fusing the facial expression features corresponding to the target image in the facial expression feature sequence and the segment text features corresponding to the target image to obtain a target segment feature corresponding to the target image;

[0213] A fragment sleep assessment result submodule, used for inputting the target fragment feature into the target neural network model to obtain a fragment sleep assessment result corresponding to the target image output by the target neural network model;

[0214] The first sleep assessment result determination submodule is used to average the segment sleep assessment results corresponding to each of the target images to obtain a first sleep assessment result of the target user.

[0215] Optionally, the target segment feature submodule includes:

[0216] A physiological indicator data unit, used to obtain the physiological indicator data of the target user;

[0217] A physiological indicator segment unit, used for intercepting physiological indicator segments corresponding to the target image from the physiological indicator data based on the acquisition time of the target image, wherein the number of the physiological indicator segments is the same as the number of the target images;

[0218] The target segment feature fusion unit is used to fuse the facial expression features corresponding to the target image, the segment text features corresponding to the target image, and the physiological indicator segment corresponding to the target image in the facial expression feature sequence to obtain the target segment feature.

[0219] Optionally, the device further comprises:

[0220] A physiological indicator data module, used to obtain the physiological indicator data of the target user;

[0221] A second sleep assessment result module, configured to generate a second sleep assessment result based on the physiological indicator data;

[0222] The target sleep assessment result module is used to correct the first sleep assessment result based on the second sleep assessment result to obtain a target sleep assessment result.

[0223] Optionally, the device further comprises:

[0224] A sleep improvement suggestion information module, configured to generate sleep improvement suggestion information based on the first sleep assessment result;

[0225] An information sending module is used to send the sleep improvement suggestion information to the target user and / or an associated user corresponding to the target user.

[0226] In summary, a sleep quality assessment device provided by an embodiment of the present application includes an acquisition module for acquiring a target image sequence and target audio data of a target user; an extraction module for performing facial expression feature extraction on the target image sequence to obtain a facial expression feature sequence, and performing text feature extraction on the target audio data to obtain a first text feature; and a first sleep assessment result module for inputting the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep assessment result output by the target neural network model. Facial expression features and text features can be extracted from the image sequence and audio data of the user during sleep. Since facial expression features can reflect the user's body expression and text features can reflect the user's language expression, the user's stability during sleep can be more accurately reflected through facial expression features and text features. Therefore, by analyzing the facial expression features and text features through the neural network model, a sleep quality assessment result with high accuracy can be output.

[0227] The sleep quality assessment device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a GPU BOX, a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0228] The sleep quality assessment device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, a Linux operating system, a Windows operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0229] The sleep quality assessment device provided in the embodiment of the present application can achieve Figures 1 to 5 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0230] Alternatively, if Figure 7 As shown, an embodiment of the present application also provides an electronic device M00, including a processor M01 and a memory M02, wherein the memory M02 stores a program or instruction that can be executed on the processor M01, and when the program or instruction is executed by the processor M01, the various steps of the above-mentioned sleep quality assessment method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0231] In an embodiment of the present application, the memory M02 can be used to store software programs and various data. The memory M02 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory M02 may include a volatile memory or a non-volatile memory, or the memory x09 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory M02 in the embodiment of the present application includes but is not limited to these and any other suitable types of memories.

[0232] The processor M01 may include one or more processing units; optionally, the processor M01 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor M01.

[0233] An embodiment of the present application also provides an electronic device, including the sleep quality assessment device and the refrigerant circulation device as described above, to implement the various processes of the above-mentioned sleep quality assessment method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0234] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned sleep quality assessment method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0235] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0236] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned sleep quality assessment method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0237] It should be understood that the chip involved in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0238] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned sleep quality assessment method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0239] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0240] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0241] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for evaluating sleep quality, characterized in that: The method comprises: Acquire a target image sequence and target audio data of a target user; Performing facial expression feature extraction on the target image sequence to obtain a facial expression feature sequence, and performing text feature extraction on the target audio data to obtain a first text feature; The facial expression feature sequence and the first text feature are input into a target neural network model to obtain a first sleep assessment result output by the target neural network model.

2. The method according to claim 1, characterized in that The step of inputting the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep assessment result output by the target neural network model includes: Acquire a second text feature obtained by performing text feature extraction on the sleep quality description information of the target user; fusing the facial expression feature sequence, the first text feature, and the second text feature to obtain a target fused feature; The target fusion feature is input into the target neural network model to obtain the first sleep assessment result output by the target neural network model.

3. The method according to claim 2, characterized in that The fusing the facial expression feature sequence, the first text feature and the second text feature to obtain a target fusion feature comprises: Determine the sleep index of the target user based on the facial expression feature sequence; wherein the sleep index includes at least one of the sleep duration, the number of awakenings, the awakening duration, and the duration of the sleep stage; Extracting sleep index features based on the sleep index; The sleep index feature, the facial expression feature sequence, the first text feature, and the second text feature are fused to obtain the target fusion feature.

4. The method according to claim 2, characterized in that: The fusing the facial expression feature sequence, the first text feature and the second text feature to obtain a target fusion feature comprises: Based on the facial expression feature sequence, generating expression statistical indicators of the target user; The expression statistical index, the first text feature and the second text feature are fused to obtain the target fusion feature.

5. The method according to claim 2, characterized in that: The sleep quality description information includes voice description information and text description information, and the obtaining of a second text feature obtained by performing text feature extraction on the sleep quality description information of the target user includes: Extracting a first semantic feature based on the speech description information, and extracting a second semantic feature based on the text description information; When the similarity between the first semantic feature and the second semantic feature is less than or equal to a first threshold, determining information weights corresponding to the voice description information and the text description information respectively; The third text feature corresponding to the speech description information and the fourth text feature corresponding to the text description information are fused based on the information weight to obtain the second text feature.

6. The method according to claim 1, characterized in that The first text feature includes a segment text feature obtained by extracting features from a target audio segment corresponding to the target image, the target audio segment is obtained by intercepting the target audio data based on the acquisition time of the target image, and the number of the target audio segments is the same as the number of the target images. The facial expression feature sequence and the first text feature are input into a target neural network model to obtain a first sleep assessment result output by the target neural network model, including: Fusion of the facial expression features corresponding to the target image in the facial expression feature sequence and the segment text features corresponding to the target image to obtain a target segment feature corresponding to the target image; Inputting the target segment feature into the target neural network model to obtain a segment sleep assessment result corresponding to the target image output by the target neural network model; The segment sleep assessment results corresponding to the target images are averaged to obtain a first sleep assessment result of the target user.

7. The method according to claim 6, characterized in that The step of fusing the facial expression features corresponding to the target image in the facial expression feature sequence and the segment text features corresponding to the target image to obtain the target segment features corresponding to the target image includes: Acquiring physiological indicator data of the target user; Based on the acquisition time of the target image, intercepting physiological indicator segments corresponding to the target image from the physiological indicator data, the number of the physiological indicator segments being the same as the number of the target images; The facial expression features corresponding to the target image in the facial expression feature sequence, the segment text features corresponding to the target image, and the physiological indicator segment corresponding to the target image are integrated to obtain the target segment features.

8. The method according to claim 1, characterized in that The method further comprises: Acquiring physiological indicator data of the target user; generating a second sleep assessment result based on the physiological indicator data; The first sleep evaluation result is corrected based on the second sleep evaluation result to obtain a target sleep evaluation result.

9. The method according to claim 1, characterized in that: The method further comprises: generating sleep improvement suggestion information based on the first sleep assessment result; The sleep improvement suggestion information is sent to the target user and / or an associated user corresponding to the target user.

10. A sleep quality assessment device, characterized in that: The device comprises: An acquisition module, used to acquire a target image sequence and target audio data of a target user; An extraction module is used to extract facial expression features from the target image sequence to obtain a facial expression feature sequence, and to extract text features from the target audio data to obtain a first text feature; The first sleep assessment result module is used to input the facial expression feature sequence and the first text feature into a target neural network model to obtain a first sleep assessment result output by the target neural network model.

11. An electronic device, characterized in that: The sleep quality assessment device according to claim 10 is used to implement the sleep quality assessment method according to any one of claims 1 to 9.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the sleep quality assessment method according to any one of claims 1 to 9 is implemented.

Images