Intelligent sleep monitoring system and method based on multimodal data fusion

Abstract

The present application relates to the field of sleep monitoring, and specifically discloses an intelligent sleep monitoring system and method based on multimodal data fusion, which first obtains a user's sleep body movement state image collected by a camera, a user's snoring signal collected by a sound sensor, and a user's vital sign data collected by a wearable device, and then uses deep learning technology to perform feature extraction and correlation analysis on the three, and finally uses a classifier to determine whether to issue a sleep breathing abnormality monitoring warning signal, so as to timely discover sleep breathing abnormalities and better meet the user's health needs.

Description

Technical Field

[0001] The present application relates to the field of sleep monitoring, and more specifically, to an intelligent sleep monitoring system and method based on multimodal data fusion. Background Art

[0002] Obstructive sleep apnea-hypopnea syndrome is a sleep breathing disorder characterized by partial or complete upper airway obstruction, leading to apnea and hypopnea, which in turn causes intermittent hypoxia and decreased sleep quality during sleep. Clinically, obstructive sleep apnea-hypopnea syndrome is often manifested as frequent snoring. Although its cause is not fully understood, it has been proven to be related to multiple factors. This disease is more common in middle-aged and elderly people, and may lead to serious consequences such as cardiovascular disease, metabolic disorders, and decreased neurocognitive function, posing a threat to the health of users.

[0003] Currently, polysomnography is an effective tool for diagnosing obstructive sleep apnea-hypopnea syndrome, which can comprehensively evaluate apnea events during sleep. However, the detector has some limitations. First, it is bulky and can usually only be tested in a hospital setting, which requires users to be tested in the hospital. Due to the large number of users of polysomnography, it is difficult for hospitals to provide enough beds to meet the demand. In addition, the system requires the use of a large number of gel electrodes and bulky equipment, which is directly connected to the human body through multiple wires, which interferes with natural sleeping patterns. It is not only uncomfortable and inconvenient, but also has low user acceptance. For example, when a user needs to go to the toilet, all connected sensors must be removed one by one, and then reconnected and started after returning, which is a rather cumbersome process.

[0004] Therefore, an intelligent sleep monitoring system and method based on multimodal data fusion is desired. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed. The embodiment of this application provides an intelligent sleep monitoring system and method based on multimodal data fusion, which first obtains the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device, and then uses deep learning technology to perform feature extraction and correlation analysis on the three, and finally uses a classifier to determine whether to issue a sleep breathing abnormality monitoring warning signal, so as to timely detect sleep breathing abnormalities and better meet the user's health needs.

[0006] According to one aspect of the present application, there is provided an intelligent sleep monitoring system based on multimodal data fusion, which includes:

[0007] A user sleep monitoring data acquisition module is used to acquire the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device;

[0008] A user sleep monitoring data extraction module is used to extract a user sleep body movement attention feature vector and a user sleep signal association feature vector from the user sleep body movement state image collected by the camera, the user snoring signal collected by the sound sensor, and the user vital sign data collected by the wearable device;

[0009] The sleep breathing abnormality judgment and warning module is used to judge whether to issue a sleep breathing abnormality monitoring and warning signal based on the user's sleep body movement attention feature vector and the user's sleep signal association feature vector.

[0010] According to another aspect of the present application, there is provided an intelligent sleep monitoring method based on multimodal data fusion, which includes:

[0011] Acquire the user's sleeping body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device;

[0012] Extracting a user's sleep body movement attention feature vector and a user's sleep signal association feature vector from the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device;

[0013] Based on the user's sleep body movement attention feature vector and the user's sleep signal association feature vector, it is determined whether to issue a sleep abnormal breathing monitoring warning signal.

[0014] Compared with the prior art, the present application provides an intelligent sleep monitoring system and method based on multimodal data fusion, which first obtains the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device, and then uses deep learning technology to perform feature extraction and correlation analysis on the three, and finally uses a classifier to determine whether to issue a sleep breathing abnormality monitoring warning signal, so as to timely detect sleep breathing abnormalities and better meet the user's health needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 4 is a block diagram of an intelligent sleep monitoring system based on multimodal data fusion according to an embodiment of the present application.

[0017] Figure 2 4 is a block diagram of a user sleep monitoring data extraction module in an intelligent sleep monitoring system based on multimodal data fusion according to an embodiment of the present application.

[0018] Figure 3 4 is a block diagram of a sleep body movement state feature extraction unit in an intelligent sleep monitoring system based on multimodal data fusion according to an embodiment of the present application.

[0019] Figure 4 The present invention is a block diagram of a sleep breathing abnormality judgment and warning module in an intelligent sleep monitoring system based on multimodal data fusion according to an embodiment of the present application.

[0020] Figure 5 Flow chart of an intelligent sleep monitoring method based on multimodal data fusion according to an embodiment of the present application.

[0021] Figure 6 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0023] Figure 1 FIG. 1 is a block diagram of an intelligent sleep monitoring system based on multimodal data fusion according to an embodiment of the present application. Figure 1 As shown, according to an embodiment of the present application, an intelligent sleep monitoring system 100 based on multimodal data fusion includes: a user sleep monitoring data acquisition module 110, which is used to obtain a user's sleep body movement state image collected by a camera, a user's snoring signal collected by a sound sensor, and a user's vital sign data collected by a wearable device; a user sleep monitoring data extraction module 120, which is used to extract a user's sleep body movement attention feature vector and a user's sleep signal association feature vector from the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device; a sleep breathing abnormality judgment and warning module 130, which is used to judge whether to issue a sleep breathing abnormality monitoring warning signal based on the user's sleep body movement attention feature vector and the user's sleep signal association feature vector.

[0024] In the above-mentioned intelligent sleep monitoring system 100 based on multimodal data fusion, the user sleep monitoring data acquisition module 110 is used to obtain the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device. It should be understood that obstructive sleep apnea hypopnea syndrome is a sleep breathing disease characterized by partial or complete obstruction of the airway, resulting in apnea and hypopnea, which in turn causes sleep interruption and oxygen deficiency, affecting sleep quality. In medical practice, this syndrome is often manifested as frequent snoring. Although its cause is not yet fully understood, it has been confirmed to be related to multiple factors. The disease is more common in the middle-aged and elderly population, and may lead to serious consequences such as cardiovascular disease, metabolic disorders, and decreased neurocognitive function, posing a threat to the patient's health. At present, polysomnography is an effective diagnostic tool for evaluating apnea events during sleep. However, the device has some limitations. First, it is bulky and can usually only be used in hospital settings, resulting in users needing to go to the hospital for testing. Due to the high demand for polysomnography, it is difficult for hospitals to provide enough beds. In addition, the system requires the use of a large number of gel electrodes and bulky equipment, which are directly connected to the human body through multiple wires, which interferes with natural sleep. It is not only uncomfortable and inconvenient, but also has low user acceptance. For example, if a user needs to go to the toilet, he needs to remove all connected sensors one by one, and then reconnect and start after returning, which is a rather cumbersome process. Therefore, in the technical solution of the present application, by acquiring the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device, and combining it with deep learning technology, it is possible to monitor and warn of abnormal sleep breathing. This method can detect abnormal sleep breathing in a timely manner and better meet the health needs of users.

[0025] Specifically, the sleep body movement state images collected by the camera can provide information about the user's body activity and posture changes during sleep. By analyzing these images, the system can detect the user's body movement frequency, duration and irregularity, thereby evaluating the user's sleep depth and quality. The user's snoring signal collected by the sound sensor is another important monitoring indicator. Snoring is often associated with abnormal breathing and sleep quality problems during sleep. The system can analyze the frequency, intensity and pattern of snoring to help determine whether the user has sleep breathing problems, such as sleep apnea syndrome. The user's physical sign data collected by wearable devices include physiological parameters such as heart rate, respiratory rate, and motion data. These data can provide more objective and detailed information on the user's health status, help monitor sleep quality, assess cardiovascular health, and detect physiological changes during sleep. Combining these data sources, the intelligent sleep monitoring system can comprehensively analyze the user's sleep situation, identify sleep problems and abnormalities, and provide users with personalized sleep advice and intervention measures.

[0026] In the above-mentioned intelligent sleep monitoring system 100 based on multimodal data fusion, the user sleep monitoring data extraction module 120 is used to extract the user's sleep body movement attention feature vector and the user's sleep signal association feature vector from the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device. It should be understood that by comprehensively extracting the user's sleep body movement attention feature vector and the user's sleep signal association feature vector, the system can more comprehensively evaluate the user's sleep condition, identify potential sleep problems, and provide personalized sleep management suggestions. This feature extraction method of multimodal data fusion can make full use of the advantages of different data sources, improve the accuracy and reliability of the sleep monitoring system, provide users with more effective sleep health management services, and help improve the user's sleep quality and quality of life.

[0027] Figure 2 FIG. 1 is a block diagram of a user sleep monitoring data extraction module in an intelligent sleep monitoring system based on multimodal data fusion according to an embodiment of the present application. Figure 2 As shown, in a specific embodiment of the present application, the user sleep monitoring data extraction module 120 includes: a sleep body movement state feature extraction unit 121, used to extract features from the user sleep body movement state image collected by the camera to obtain the user sleep body movement attention feature vector; a snoring signal feature extraction unit 122, used to extract features from the user snoring signal collected by the sound sensor to obtain a user snoring signal time domain enhancement feature vector; a vital sign data feature extraction unit 123, used to extract features from the user vital sign data collected by the wearable device to obtain a user vital sign semantic feature vector; a user sleep signal association unit 124, used to associate the user snoring signal time domain enhancement feature vector with the user vital sign semantic feature vector to obtain the user sleep signal association feature vector.

[0028] It should be understood that the feature extraction process involves extracting visual features related to body movement from the sleep body movement state image. These features may include information such as the location, shape, size, and motion trajectory of the body movement. By calculating these features, the system can identify and track the user's body movement during sleep, thereby analyzing the frequency, duration, and regularity of the body movement. In addition, feature extraction can also include temporal feature analysis of body movement, that is, modeling and extracting the changing pattern of body movement in the time series. This temporal feature analysis can help the system identify the different body movement stages and activity cycles of the user during sleep, and further reveal the user's sleep body movement patterns and regularities. By comprehensively utilizing these different types of feature extraction methods, the system can extract the user's sleep body movement attention feature vector from the user's sleep body movement state image captured by the camera, providing a more accurate and comprehensive information basis for sleep monitoring and analysis, helping users understand their sleep conditions and take corresponding improvement measures.

[0029] Furthermore, feature extraction is performed on the user's snoring signal collected by the sound sensor, and key information related to the user's snoring characteristics can be extracted through analysis and processing of the sound signal, so as to evaluate the user's sleep quality and possible sleep problems. This feature extraction method utilizes signal processing and acoustic analysis technology, which can mine useful information from the sound signal and provide more comprehensive data support for the sleep monitoring system. Among them, the feature extraction process involves time domain analysis of the snoring signal, that is, processing and analyzing the waveform of the signal in the time domain. This includes extracting the time domain features of the sound signal, such as the energy of the sound signal, the shape of the time domain waveform, the amplitude change, etc. These time domain features can reflect important information such as the frequency, intensity, and duration of the snoring signal. The comprehensively extracted time domain enhanced feature vector of the user's snoring signal can help the system analyze the user's snoring pattern, frequency, intensity, and other information, and then identify possible sleep breathing problems or abnormal sleep quality.

[0030] Furthermore, feature extraction of user vital sign data collected by wearable devices can extract features with semantic information from the user's physiological signals, so as to more comprehensively understand the user's health status and sleep quality. By extracting features from user vital sign data collected by wearable devices, the system can obtain user vital sign semantic feature vectors, which contain rich physiological and semantic information and can comprehensively evaluate the user's health status and sleep quality. This method of comprehensive analysis and mining of user physiological data provides deeper and more accurate information support for the sleep monitoring system, helping users to better manage and improve sleep quality.

[0031] In particular, associating the time-domain enhanced feature vector of the user's snoring signal with the semantic feature vector of the user's vital signs can combine the information of the sound signal and the physiological signal to comprehensively evaluate the user's sleep state and sleep quality from multiple angles. This association analysis method can provide more comprehensive and accurate sleep monitoring results, helping users to discover sleep problems in a timely manner and take corresponding measures. Among them, association analysis can help the system discover the correlation features of the user's sleep signals, that is, the correlation features between the sound signal and the physiological signal. These correlation features can include the correlation between the spectral features of the sound signal and the heart rate variability, the correlation between the amplitude change of the sound signal and the respiratory rate, etc. By mining these correlation features, the system can more accurately evaluate the user's sleep state and provide more targeted sleep monitoring results.

[0032] Figure 3 FIG. 1 is a block diagram of a sleep body motion state feature extraction unit in an intelligent sleep monitoring system based on multimodal data fusion according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, the sleep body movement state feature extraction unit 121 includes: a sleep body movement state target detection subunit 1211, which is used to pass the user's sleep body movement state image collected by the camera through a sleep body movement state target detection model to obtain the user's sleep body movement interest region; a body movement interest region feature encoding subunit 1212, which is used to feature encode the user's sleep body movement interest region to obtain the user's sleep body movement attention feature vector.

[0033] It should be understood that through the sleep body movement state target detection model, the system can analyze and process the user's sleep body movement state image collected by the camera, and identify the sleep body movement target area in the image. This target detection model can automatically detect and locate sleep body movement based on deep learning algorithms, such as convolutional neural networks, so as to accurately mark the user's body movement area during sleep. After obtaining the user's sleep body movement interest area, the system can further analyze and extract features from these areas to obtain relevant information about the user's sleep body movement, such as frequency, duration, intensity, etc. By comprehensively analyzing this information, the system can evaluate the user's sleep quality, sleep depth, and possible sleep problems, and provide users with personalized sleep suggestions and improvement plans. By combining the user's sleep body movement state image collected by the camera with the sleep body movement state target detection model, the system can realize automatic monitoring and analysis of the user's sleep body movement, and provide more comprehensive and accurate sleep information. This sleep monitoring method based on computer vision technology can not only reduce the user's monitoring burden, but also provide strong support for sleep research and sleep disorder diagnosis, helping users improve sleep quality and maintain physical and mental health. Specifically, the sleep body movement state target detection model is used to process the user sleep body movement state image collected by the camera with the following target detection formula to obtain the user sleep body movement region of interest;

[0034] Among them, the target detection formula is:

[0035] ROI=(cls(ψ det ,B),Regr(ψ det ,B))

[0036] Among them, ψ det is the user's sleep body motion state image collected by the camera, B is the anchor frame, ROI is the user's sleep body motion region of interest, cls(ψ det ,B) indicates classification, Regr(ψ det ,B) indicates regression.

[0037] Furthermore, by encoding the features of the user's sleep body movement interest area, the system can convert the body movement information in the image into a numerical feature representation. These feature codes can include features in shape, color, texture, etc., reflecting the various visual attributes of the body movement area. By encoding these features, the system can convert complex image information into feature vectors that are easy to process and analyze.

[0038] In a specific embodiment of the present application, the body motion region of interest feature encoding subunit 1212 includes: passing the user sleep body motion interest region through a user sleep body motion feature encoder based on a spatial attention module to obtain a user sleep body motion attention feature map; pooling the user sleep body motion attention feature map to obtain the user sleep body motion attention feature vector.

[0039] It should be understandable that the user sleep body movement feature encoder based on the spatial attention module can learn the feature weights of different positions in the body movement area of ​​interest, so that the model can pay more attention to important body movement areas. This attention mechanism can help the system dynamically adjust the importance of features at different positions during the feature encoding process, highlight the key information in the body movement area, and improve the accuracy and expression ability of feature representation. After obtaining the user's sleep body movement attention feature map through the feature encoder based on the spatial attention module, the system can display important body movement information in the form of an image, and intuitively present the user's body movement behavior characteristics. This attention feature map can help users and medical staff more intuitively understand the user's body movement patterns and rules, discover abnormal body movement behaviors, and take corresponding measures in time. Specifically, the convolution encoding part of the user sleep body movement feature encoder based on the spatial attention module is used to perform deep convolution encoding on the user sleep body movement interest area to obtain an initial convolution feature map; the initial convolution feature map is input into the spatial attention part of the user sleep body movement feature encoder based on the spatial attention module to obtain a spatial attention map; the spatial attention map is passed through a Softmax activation function to obtain a spatial attention feature map; and the positional point multiplication of the spatial attention feature map and the initial convolution feature map is calculated to obtain the user sleep body movement attention feature map.

[0040] Furthermore, by pooling the user's sleep body movement attention feature map, the two-dimensional feature map information can be converted into a one-dimensional feature vector representation. This pooling operation can help the system better understand the user's body movement behavior, extract key features and perform sleep monitoring. Among them, by performing a pooling operation on the user's sleep body movement attention feature map, the feature information in the image can be reduced in dimensionality, and the two-dimensional feature map can be converted into a one-dimensional feature vector. This operation can help the system reduce data dimensions, improve computational efficiency, and retain key feature information, making subsequent analysis more concise and efficient.

[0041] In a specific embodiment of the present application, the snoring signal feature extraction unit 122 includes: calculating a user snoring signal time domain enhancement map of the user snoring signal collected by the sound sensor; passing the user snoring signal time domain enhancement map through a snoring signal time domain enhancement feature encoder based on a convolutional neural network to obtain the user snoring signal time domain enhancement feature vector.

[0042] It should be understood that by performing time domain enhancement processing on the user's snoring signal, the noise interference in the signal can be effectively reduced, the effective part of the signal can be enhanced, and the signal-to-noise ratio of the signal can be improved. This enhancement can help the system capture the key features of the snoring signal more clearly, such as the frequency, intensity, duration and other information of the snoring, which helps the system to more accurately judge the user's snoring situation. After obtaining the time domain enhancement graph of the user's snoring signal, the system can better display the waveform characteristics of the snoring signal, so that analysts can intuitively observe and understand the changing patterns of the snoring signal. The generation of this time domain enhancement graph can help medical staff more quickly detect abnormal conditions in the snoring signal, such as frequent snoring events or long duration, so as to take corresponding intervention measures in time.

[0043] Furthermore, the time domain enhancement map of the snoring signal is processed through a convolutional neural network encoder, which can convert the image information into a high-dimensional feature vector representation, retaining the important feature information in the snoring signal. These feature vectors can include the spectral features, time domain features, etc. of the snoring signal, providing the system with a richer and more comprehensive representation of the snoring signal, which helps the system to more accurately judge the user's snoring condition. Specifically, each layer of the snoring signal time domain enhancement feature encoder based on the convolutional neural network performs convolution processing, mean pooling processing based on the local feature matrix, and nonlinear activation processing on the input data in the forward transfer of the layer, so that the last layer of the snoring signal time domain enhancement feature encoder based on the convolutional neural network outputs the user's snoring signal time domain enhancement feature vector, wherein the input of the snoring signal time domain enhancement feature encoder based on the convolutional neural network is the user's snoring signal time domain enhancement map.

[0044] In a specific embodiment of the present application, the vital sign data feature extraction unit 123 includes: segmenting the user vital sign data collected by the wearable device to obtain a user vital sign text word sequence; passing the user vital sign text word sequence through a user vital sign text context semantic encoder to obtain the user vital sign semantic feature vector.

[0045] It should be understood that by segmenting the user's vital sign data, the continuous physiological signal data can be divided into discrete time windows or data segments, each data segment corresponding to a text word sequence. This word segmentation operation can help the system better distinguish the physiological signal characteristics in different time periods, so that the system can more finely analyze the user's physiological state changes, such as heart rate changes, breathing frequency, etc. Segmenting the user's vital sign data collected by wearable devices to obtain the user's vital sign text word sequence can help the system better understand and analyze the user's physiological state changes, and provide more diversified and comprehensive data features for the sleep monitoring system.

[0046] Furthermore, by passing the user's vital sign text word sequence through the user's vital sign text context semantic encoder, the user's physiological data can be semantically modeled and feature extracted using natural language processing technology, thereby gaining a deeper understanding of the user's physiological state and health information. Through the processing of the user's vital sign text context semantic encoder, the user's physiological data can be converted into semantic feature vectors, which contain abstract semantic information of the user's vital sign data and reflect the semantic similarity and correlation between different vital sign data. This feature vector can help the system better understand the changes in the user's physiological state, identify potential abnormalities, and provide a more comprehensive and in-depth data analysis capability for the sleep monitoring system. Specifically, the embedding layer of the user's vital sign text context semantic encoder is used to map each word in the user's vital sign text word sequence to a word embedding vector to obtain a sequence of vital sign word embedding vectors; the converter-based Bert model of the user's vital sign text context semantic encoder is used to perform global context semantic encoding on the sequence of vital sign word embedding vectors to obtain multiple feature vectors; and the multiple feature vectors are cascaded to obtain the user's vital sign semantic feature vector.

[0047] In the above-mentioned intelligent sleep monitoring system 100 based on multimodal data fusion, the sleep breathing abnormality judgment and warning module 130 is used to determine whether to issue a sleep breathing abnormality monitoring and warning signal based on the user's sleep body movement attention feature vector and the user's sleep signal associated feature vector. It should be understood that based on the user's sleep signal associated feature vector, the system can comprehensively consider the user's sleep signal data and other related feature vector information to more comprehensively analyze the user's sleep state. This comprehensive analysis can help the system identify potential signs of sleep breathing abnormalities, such as apnea, hypoxemia, etc., and provide more clues for monitoring abnormal conditions. By analyzing the user's sleep signal associated feature vector, the system can establish a monitoring model and rules for sleep breathing abnormalities, and set corresponding thresholds and rule judgment criteria. When it is detected that the user's sleep signal characteristics are significantly associated with the breathing abnormality-related feature vector, the system can trigger a warning signal to notify the user or relevant medical staff for further inspection and processing.

[0048] Figure 4 FIG. 1 is a block diagram of a sleep breathing abnormality judgment and warning module in an intelligent sleep monitoring system based on multimodal data fusion according to an embodiment of the present application. Figure 4As shown, in a specific embodiment of the present application, the sleep breathing abnormality judgment and warning module 130 includes: a sleep breathing feature fusion unit 131, which is used to fuse the user's sleep body movement attention feature vector and the user's sleep signal association feature vector to obtain a sleep breathing abnormality classification feature vector; a sleep breathing feature optimization unit 132, which is used to perform viewpoint metric prediction enhancement based on the attention-oriented weight on the sleep breathing abnormality classification feature vector to obtain an optimized sleep breathing abnormality classification feature vector; a sleep breathing abnormality warning unit 133, which is used to pass the optimized sleep breathing abnormality classification feature vector through a classifier to obtain a classification result, and the classification result is used to determine whether to issue a sleep breathing abnormality monitoring warning signal.

[0049] It should be understood that fusing the user's sleep body movement attention feature vector and the user's sleep signal association feature vector can comprehensively consider the user's body movement during sleep and the characteristics of the sleep signal, thereby more comprehensively describing the user's sleep state. Sleep body movement is a common phenomenon during sleep, and sleep signals contain more physiological data information. By fusing these two types of information, the system can more comprehensively understand the user's sleep situation and improve the accuracy of sleep breathing abnormalities.

[0050] In particular, in the technical solution of the present application, considering the different types of data collected from cameras, sound sensors and wearable devices, there may be complexity in fusing these multi-source data, resulting in the features of certain data sources being over-emphasized or ignored in the final feature vector. The target detection model, spatial attention module, convolutional neural network and text context semantic encoder may not be able to fully capture the features of all important sleep body movements, snoring signals and vital signs data. The user sleep body movement feature encoder and the snoring signal time domain enhancement feature encoder may not be able to fully express the subtle changes in the signal, especially when the signal is highly complex or noisy. Pooling the user sleep body movement attention feature map may lose some local information that is critical to classification. This will lead to the problem that the sleep breathing abnormality classification feature vector will have a class harmony offset relative to the predetermined class label of the classifier when the classifier is used for classification judgment, affecting the classification iteration effect. In order to solve this problem, in the technical solution of the present application, the sleep breathing abnormality classification feature vector is enhanced by viewpoint metric prediction based on attention-oriented weights to obtain an optimized sleep breathing abnormality classification feature vector.

[0051] Among them, the sleep abnormal breathing classification feature vector is enhanced by the viewpoint metric prediction based on the attention-oriented weight to obtain an optimized sleep abnormal breathing classification feature vector, including: performing probabilistic activation based on the Sigmoid function on the sleep abnormal breathing classification feature vector to obtain a probabilistic sleep abnormal breathing classification feature vector; inputting the sleep abnormal breathing classification feature vector into a classifier to obtain an initial class label probability value; calculating the difference between the feature value of each position in the probabilistic sleep abnormal breathing classification feature vector and the initial class label probability value to obtain a difference feature vector, and passing the difference feature vector through a mask function to obtain an initial weight vector for viewpoint metric prediction of sleep abnormal breathing classification; calculating the initial weight vector for viewpoint metric prediction of sleep abnormal breathing classification. The autocorrelation matrix of the weight vector is obtained, and the autocorrelation matrix of the sleep abnormal breathing classification viewpoint metric prediction initial weight vector is divided by the square root of the length of the sleep abnormal breathing classification viewpoint metric prediction initial weight vector, and then multiplied by the sleep abnormal breathing classification viewpoint metric prediction initial weight vector through a softmax function to obtain a viewpoint metric prediction attention-oriented weight vector; the viewpoint metric prediction attention-oriented weight vector is multiplied by the sleep abnormal breathing classification feature vector according to position points to obtain a first weighted sleep abnormal breathing classification feature vector; a predetermined weight hyperparameter is multiplied by the sleep abnormal breathing classification feature vector, and then the predetermined weight hyperparameter is added to the first weighted sleep abnormal breathing classification feature vector according to position to obtain an optimized sleep abnormal breathing classification feature vector.

[0052] The step of performing viewpoint metric prediction enhancement based on attention-oriented weight on the sleep apnea classification feature vector to obtain an optimized sleep apnea classification feature vector includes:

[0053] V cp =sigmoid(V co )

[0054]

[0055]

[0056] Among them, V co represents the sleep apnea classification feature vector, sigmoid represents the activation function, V cp represents the probabilistic sleep apnea classification feature vector, P represents the initial class label probability value, represents positional subtraction, mask represents the mask function, x represents the variable of the input mask function, θ represents the predetermined threshold, V e represents the initial weight vector for sleep apnea classification viewpoint metric prediction, represents matrix multiplication, T represents the transpose of a vector, L represents the length of the initial weight vector for the sleep apnea classification viewpoint metric prediction, Softmax represents a normalized exponential function, V ea represents the viewpoint metric prediction attention-oriented weight vector, α represents the predetermined weight hyperparameter, ⊙ represents the position point multiplication, Indicates addition by position, V' co Represents the optimized sleep apnea classification feature vector

[0057] That is, in the technical solution of the present application, the sleep breathing abnormality classification feature vector is enhanced by viewpoint metric prediction based on attention-guided weights. First, the sleep breathing abnormality classification feature vector is subjected to probabilistic activation based on the Sigmoid function to obtain a probabilistic sleep breathing abnormality classification feature vector, and the sleep breathing abnormality classification feature vector is input into the classifier to obtain an initial class label probability value, and the viewpoint metric prediction contribution of each position in the canonical probabilistic sleep breathing abnormality classification feature vector to the predetermined class probability label is initialized through inverse error analysis and masking.

[0058] Then, the viewpoint space of the inverse optimization is expanded by using the autocorrelation attention representation of the initial weight vector predicted by the viewpoint metric of the sleep abnormal breathing classification, and the sub-dimensional consistency is used as the attention-guided weight to apply the inverse consistency attention mechanism of the sub-dimensional distribution to the sleep abnormal breathing classification feature vector. In this way, the viewpoint metric prediction based on post-machine learning is used to perform a learning predictive matching between the forward feature manifold and the backward class probability distribution, so that the optimized sleep abnormal breathing classification feature vector has a relatively better distribution consistency with respect to the class probability distribution of the classifier, so as to improve the accuracy of its classification judgment by the classifier.

[0059] Furthermore, the classification results obtained by the classifier can be used as a basis for determining whether to issue a sleep breathing abnormality monitoring warning signal. When the classifier determines that the user has potential sleep breathing abnormalities, the system can automatically trigger a warning signal to notify the user or relevant medical staff for further inspection and treatment. This automated monitoring and warning system can help improve the efficiency and accuracy of sleep breathing abnormality detection and provide more comprehensive protection for the user's health.

[0060] In summary, the embodiment of the present application first obtains the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device, and then uses deep learning technology to perform feature extraction and correlation analysis on the three. Finally, a classifier is used to determine whether to issue a sleep breathing abnormality monitoring warning signal, so as to timely detect sleep breathing abnormalities and better meet the user's health needs.

[0061] As described above, the intelligent sleep monitoring system 100 based on multimodal data fusion according to the embodiment of the present application can be implemented in various terminal devices. In one example, the intelligent sleep monitoring system 100 based on multimodal data fusion can be integrated into the terminal device as a software module and / or a hardware module. For example, the intelligent sleep monitoring system 100 based on multimodal data fusion can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the intelligent sleep monitoring system 100 based on multimodal data fusion can also be one of the many hardware modules of the terminal device.

[0062] Alternatively, in another example, the intelligent sleep monitoring system 100 based on multimodal data fusion and the terminal device may also be separate devices, and the intelligent sleep monitoring system 100 based on multimodal data fusion may be connected to the terminal device via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0063] Figure 5 FIG. 1 is a flow chart of an intelligent sleep monitoring method based on multimodal data fusion according to an embodiment of the present application. Figure 5 As shown, according to the intelligent sleep monitoring method based on multimodal data fusion according to the embodiment of the present application, it includes: S110, acquiring the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device; S120, extracting the user's sleep body movement attention feature vector and the user's sleep signal association feature vector from the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device; S130, based on the user's sleep body movement attention feature vector and the user's sleep signal association feature vector, determining whether to issue a sleep breathing abnormality monitoring warning signal.

[0064] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned intelligent sleep monitoring method based on multimodal data fusion have been referred to above. Figures 1 to 4 The invention has been introduced in detail in the description of the intelligent sleep monitoring system based on multimodal data fusion, and therefore, its repeated description will be omitted.

[0065] Below, reference Figure 6 To describe an electronic device according to an embodiment of the present application.

[0066] like Figure 6As shown, the electronic device 10 includes an input device 11, an input interface 12, a central processing unit 13, a memory 14, an output interface 15, an output device 16, and a bus 17. The input interface 12, the central processing unit 13, the memory 14, and the output interface 15 are connected to each other through the bus 17, and the input device 11 and the output device 16 are connected to the bus 17 through the input interface 12 and the output interface 15, respectively, and then connected to other components of the electronic device 10.

[0067] Specifically, the input device 11 receives input information from the outside and transmits the input information to the central processing unit 13 through the input interface 12; the central processing unit 13 processes the input information based on the computer executable instructions stored in the memory 14 to generate output information, temporarily or permanently stores the output information in the memory 14, and then transmits the output information to the output device 16 through the output interface 15; the output device 16 outputs the output information to the outside of the electronic device 10 for user use.

[0068] In one embodiment, Figure 6 The electronic device 10 shown can be implemented as a network device, which may include: a memory configured to store programs; a processor configured to run the programs stored in the memory to execute any one of the intelligent sleep monitoring methods based on multimodal data fusion described in the above embodiments.

[0069] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.

[0070] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and which can be accessed by a computer.

[0071] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.

Claims

1. An intelligent sleep monitoring system based on multimodal data fusion, characterized in that: include: A user sleep monitoring data acquisition module is used to acquire the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device; A user sleep monitoring data extraction module is used to extract a user sleep body movement attention feature vector and a user sleep signal association feature vector from the user sleep body movement state image collected by the camera, the user snoring signal collected by the sound sensor, and the user vital sign data collected by the wearable device; A sleep breathing abnormality judgment and warning module, used to judge whether to issue a sleep breathing abnormality monitoring and warning signal based on the user's sleep body movement attention feature vector and the user's sleep signal association feature vector; Wherein, the user sleep monitoring data extraction module includes: A sleep body movement state feature extraction unit, used for extracting features from the user's sleep body movement state image captured by the camera to obtain a sleep body movement attention feature vector of the user; A snoring signal feature extraction unit, used for extracting features from the user's snoring signal collected by the sound sensor to obtain a time-domain enhanced feature vector of the user's snoring signal; A vital sign data feature extraction unit, used for extracting features from the user vital sign data collected by the wearable device to obtain a user vital sign semantic feature vector; The user sleep signal associating unit is used to associate the time domain enhancement feature vector of the user snoring signal with the user vital sign semantic feature vector to obtain the user sleep signal associated feature vector.

2. The intelligent sleep monitoring system based on multimodal data fusion according to claim 1, characterized in that: The sleep body movement state feature extraction unit comprises: A sleep body movement state target detection subunit, configured to obtain a user sleep body movement region of interest by applying the user sleep body movement state image acquired by the camera to a sleep body movement state target detection model; The body movement region of interest feature encoding subunit is used to perform feature encoding on the user's sleep body movement region of interest to obtain the user's sleep body movement attention feature vector.

3. The intelligent sleep monitoring system based on multimodal data fusion according to claim 2, characterized in that: The body motion region of interest feature encoding subunit comprises: The user's sleep body movement interest region is passed through a user's sleep body movement feature encoder based on a spatial attention module to obtain a user's sleep body movement attention feature map; The user's sleep body movement attention feature map is pooled to obtain the user's sleep body movement attention feature vector.

4. The intelligent sleep monitoring system based on multimodal data fusion according to claim 3, characterized in that: The snoring signal feature extraction unit comprises: Calculate a user snoring signal time domain enhancement graph of the user snoring signal collected by the sound sensor; The time domain enhancement image of the user's snoring signal is passed through a snoring signal time domain enhancement feature encoder based on a convolutional neural network to obtain a time domain enhancement feature vector of the user's snoring signal.

5. The intelligent sleep monitoring system based on multimodal data fusion according to claim 4, characterized in that: The vital sign data feature extraction unit comprises: Segmenting the user's vital sign data collected by the wearable device to obtain a user's vital sign text word sequence; The user vital sign text word sequence is passed through a user vital sign text context semantic encoder to obtain the user vital sign semantic feature vector.

6. The intelligent sleep monitoring system based on multimodal data fusion according to claim 5, characterized in that: The sleep breathing abnormality judgment and warning module includes: A sleep breathing feature fusion unit, used for fusing the user's sleep body movement attention feature vector and the user's sleep signal association feature vector to obtain a sleep breathing abnormality classification feature vector; a sleep breathing feature optimization unit, configured to perform viewpoint metric prediction enhancement based on attention-oriented weights on the sleep breathing abnormality classification feature vector to obtain an optimized sleep breathing abnormality classification feature vector; The sleep breathing abnormality warning unit is used to pass the optimized sleep breathing abnormality classification feature vector through a classifier to obtain a classification result, and the classification result is used to determine whether to issue a sleep breathing abnormality monitoring warning signal.

7. The intelligent sleep monitoring system based on multimodal data fusion according to claim 6, characterized in that: The sleep breathing characteristic optimization unit comprises: Performing probabilistic activation based on a Sigmoid function on the sleep breathing abnormality classification feature vector to obtain a probabilistic sleep breathing abnormality classification feature vector; Inputting the sleep apnea classification feature vector into a classifier to obtain an initial class label probability value; Calculating the difference between the feature value of each position in the probabilistic sleep breathing abnormality classification feature vector and the initial class label probability value to obtain a difference feature vector, and passing the difference feature vector through a mask function to obtain an initial weight vector for sleep breathing abnormality classification viewpoint metric prediction; Calculating the autocorrelation matrix of the sleep abnormal breathing classification viewpoint metric prediction initial weight vector, and dividing the autocorrelation matrix of the sleep abnormal breathing classification viewpoint metric prediction initial weight vector by the square root of the length of the sleep abnormal breathing classification viewpoint metric prediction initial weight vector, and then multiplying it by the sleep abnormal breathing classification viewpoint metric prediction initial weight vector through a softmax function to obtain a viewpoint metric prediction attention-oriented weight vector; Multiplying the viewpoint metric prediction attention orientation weight vector by the sleep breathing abnormality classification feature vector by position points to obtain a first weighted sleep breathing abnormality classification feature vector; The predetermined weight hyperparameter is multiplied by the sleep abnormal breathing classification feature vector, and then the resultant is added to the first weighted sleep abnormal breathing classification feature vector by position to obtain an optimized sleep abnormal breathing classification feature vector.

8. An intelligent sleep monitoring method based on multimodal data fusion, characterized in that: include: Acquire the user's sleeping body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device; Extracting a user's sleep body movement attention feature vector and a user's sleep signal association feature vector from the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device; Based on the user's sleep body movement attention feature vector and the user's sleep signal association feature vector, determining whether to issue a sleep breathing abnormality monitoring warning signal; The method of extracting a user's sleep body movement attention feature vector and a user's sleep signal association feature vector from the user's sleep body movement state image collected by the camera, the user's snoring signal collected by the sound sensor, and the user's vital sign data collected by the wearable device includes: Performing feature extraction on the user's sleep body movement state image captured by the camera to obtain the user's sleep body movement attention feature vector; Performing feature extraction on the user's snoring signal collected by the sound sensor to obtain a time-domain enhanced feature vector of the user's snoring signal; Performing feature extraction on the user vital sign data collected by the wearable device to obtain a user vital sign semantic feature vector; The time-domain enhanced feature vector of the user's snoring signal is associated with the user's physical sign semantic feature vector to obtain the user's sleep signal associated feature vector.

Images