A machine learning-based multi-modal sleep disorder detection method and system

Abstract

The present application relates to the field of health assessment, and particularly relates to a multi-modal sleep disorder detection method and system based on machine learning. The present application innovatively introduces a quantitative calculation index of "dark eye circle" degree, combines multi-modal information fusion analysis of face video and audio, realizes rapid, accurate and objective detection of sleep disorders, and provides effective technical support for preliminary screening and auxiliary diagnosis of sleep disorders. The detection efficiency is much higher than that of traditional PSG examination (which requires all-night monitoring). At the same time, the method of the present application supports batch processing, and can be deployed in community health service centers, enterprise health examination stations and other scenes, so as to realize efficient sleep disorder screening of large-scale population.

Description

Technical Field

[0001] This invention relates to the field of health assessment, and in particular to a multimodal sleep disorder detection method and system based on machine learning. Background Technology

[0002] Sleep disorders are a common health problem, mainly manifested as difficulty falling asleep, poor sleep quality, and early awakening. Long-term sleep disorders not only affect an individual's daily life and work efficiency but can also lead to a range of physical and mental health problems, such as weakened immunity, memory loss, and mood disorders. Therefore, accurate and timely detection of sleep disorders is of great significance for safeguarding people's physical and mental health.

[0003] Currently, the main methods for detecting sleep disorders include the following:

[0004] (1) Polysomnography (PSG): This is currently the "gold standard" method for diagnosing sleep disorders. By monitoring multiple physiological indicators such as electroencephalogram (EEG), electrooculogram (EOG), electromyogram (EMG), electrocardiogram (ECG), respiratory airflow, and blood oxygen saturation, it comprehensively assesses sleep structure and the type of sleep disorder. However, PSG examinations need to be performed in a professional sleep laboratory, the equipment is expensive and the operation is complicated, and patients need to stay overnight in the hospital, which is inconvenient for patients and makes it difficult to achieve large-scale screening.

[0005] (2) Wearable device monitoring: In recent years, wearable devices such as smart bracelets and smartwatches have become increasingly popular. These devices can monitor users' sleep duration and sleep stages through accelerometers, heart rate sensors, etc. However, the accuracy of these devices is limited, mainly relying on single indicators such as body movement and heart rate, and cannot comprehensively reflect the characteristics of sleep disorders.

[0006] (3) Questionnaire survey method: Sleep quality is assessed by completing standardized questionnaires such as the Pittsburgh Sleep Quality Index (PSQI). This method is simple and easy to implement, but it has the drawbacks of strong subjectivity and susceptibility to the memory bias and subjective concealment of respondents, making it difficult to guarantee the objectivity and accuracy of the assessment results.

[0007] (4) Video and audio-based monitoring methods: In recent years, some studies have attempted to use computer vision and speech analysis technologies to assess sleep status by analyzing facial video and audio signals. For example, fatigue level and sleep quality can be determined by analyzing facial expressions, eye states, and speech features. However, most existing methods only focus on single-dimensional features and fail to fully integrate multimodal information for comprehensive analysis. Using only video or only audio fails to fully integrate the complementary advantages of multimodal information, resulting in limited accuracy and reliability of the assessment results.

[0008] In summary, existing sleep disorder detection technologies are either too specialized and complex (such as PSG) or too simplified (such as wearable devices), lacking a solution that can guarantee both accuracy and ease of daily use and large-scale screening. Therefore, there is a need for a more easily deployed, accurate, and comprehensive machine learning-based multimodal sleep disorder detection method and system. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings in portability and accuracy of existing technologies, and to provide a multimodal sleep disorder detection method and system based on machine learning.

[0010] To achieve the above-mentioned objectives, the present invention provides the following technical solution: A machine learning-based multimodal sleep disorder detection method includes the following steps: S1: Acquire the RGB video and audio data of the person to be tested; S2: Extract video features of the person to be detected based on the RGB video; and extract audio features of the person to be detected based on the audio data; the video features include dark circle features, facial expression features, and emotion features; S3: The video features and audio features are spliced ​​and fused using principal component analysis to generate a fusion vector; S4: Filter the fusion vector according to the set conditions, perform standardization processing, and output the standardized fusion vector; S5: Input the standardized fusion vector into a pre-trained machine learning model for recognition, and generate sleep disorder detection probability results.

[0011] As a preferred embodiment of the present invention, the extraction of the dark circle feature includes the following steps: S211: Detect face regions from video frames of the RGB video using a deep learning face detection algorithm and locate facial key points; the facial key points include the inner and outer corners of both eyes and the midpoint of the lower eyelid; S212: Extract the lower eye region based on the facial key points; wherein, the lower eye region is: taking the midpoint of the lower eyelid of each eye as the center, extending to the inner and outer corners to obtain two fold lines, and shifting the two fold lines downward by a set distance to output the resulting polygonal region as the lower eye region; S213: Extract a facial reference region based on the facial key points; the facial reference region is: a rectangular region of a set size extracted from the center of the cheek in the face region, and output as the facial reference region; S214: Extract the grayscale features and Lab color space features of the area under the human eyes and the facial reference area, and calculate the average feature value and dark circle feature index of the two areas; wherein, the dark circle feature index includes: Dark circle grayscale index = average grayscale value of the area under the eyes / average grayscale value of the reference area of ​​the face; Dark circle brightness index = average L-channel value of the area under the eyes / average L-channel value of the reference area of ​​the face; The characteristic index of vascular dark circles is calculated as: average b-channel value of the facial reference area - average b-channel value of the under-eye area. S215: Calculate the dark circle features of a set series of frames in the RGB video based on the dark circle feature indicators; the dark circle features include the mean, standard deviation and trend of each dark circle feature indicator.

[0012] As a preferred embodiment of the present invention, the extraction of dark circle features further includes S216 extraction of texture features in the under-eye area; comprising the following steps: S2161: Divide the area under the human eye into several sub-regions evenly; S2162: For each pixel in each sub-region, using its corresponding grayscale value as the center, calculate the uniform local binary mode value of the current pixel based on the circular neighborhood defined by the set parameters. The calculation formula is as follows:

[0013]

[0014]

[0015] in, R represents the uniform local binary mode value of the circular neighborhood of the current pixel, where R is the set radius of the circular neighborhood and P is the set number of neighborhood points. Let p be the gray value of the p-th neighboring point. This is the grayscale value of the current pixel. s(x) is the uniformity measure of the current circular neighborhood, and s(x) is the sign function; S2163: Calculate the histogram of the uniform local binary pattern encoding in each sub-region to obtain a multi-dimensional feature vector; S2164: Calculate the texture roughness based on the multidimensional feature vector, wherein the texture roughness includes: Information entropy: ; energy: ; Contrast Ratio: ; Where N is the number of dimensions of the multidimensional feature vector. Let be the frequency of occurrence of the j-th LBP mode in the sub-region; S2165: Calculate the texture roughness of the set consecutive frames in the RGB video, output the corresponding mean, standard deviation and rate of change as multi-dimensional texture features, and concatenate the multi-dimensional texture features with the dark circle features in S215 to output a new dark circle feature.

[0016] As a preferred embodiment of the present invention, the facial expression features are extracted using the OpenFace toolkit, including the following steps: S221: Facial motion features: Extract the intensity values ​​and existence states of several set facial motions in the RGB video and output them as facial motion features; wherein, the set facial motions include any one or more of the following: inner eyebrow raising, outer eyebrow raising, eyebrow drooping, upper eyelid raising, cheek raising, eyelid tightening, nose wrinkling, upper lip raising, corner of mouth raising, pursing lips, corner of mouth turning down, chin raising, lips stretching, lips tightening, lips separating, chin drooping, and blinking; S222: Gaze direction feature: Extract eye feature points from the RGB video, generate a gaze vector in three-dimensional space, and output the gaze direction feature; S223: Head posture features: Detect the rotation angle of the head of the person to be detected in the RGB video in three-dimensional space; the rotation angle includes pitch angle, yaw angle and roll angle; S224: Facial geometric features: Detect the coordinates of facial key points in the RGB video; and calculate facial geometric features based on the coordinates of the facial key points; the facial geometric features include the aspect ratio of the eyes, the degree of mouth opening, and facial symmetry; S225: Perform temporal analysis on the facial motion features, gaze direction features, head posture features, and facial geometric features respectively, calculate the statistical features, frequency domain features, and dynamic features of each feature, and output the facial expression features of the person to be detected.

[0017] As a preferred embodiment of the present invention, the statistical characteristics include any one or more of the mean, standard deviation, maximum value, minimum value, and range; The frequency domain features are obtained by extracting frequency components through Fourier transform; The dynamic characteristics include the rate of change and / or the amplitude of fluctuation.

[0018] As a preferred embodiment of the present invention, the emotional features are extracted using the PosterV2 deep learning model; The emotional features include emotion category prediction and deep emotion features; The emotion category prediction is the probability distribution of the basic emotion category output by the PosterV2 deep learning model; The deep sentiment feature is the globally averaged pooled feature output by the last layer Transformer encoder of the PosterV2 deep learning model.

[0019] As a preferred embodiment of the present invention, the audio features are extracted using the OpenSMILE analysis library; the audio features include fundamental frequency features, energy features, speech rate features, and timbre features: The fundamental frequency characteristics include the fundamental frequency mean, standard deviation, and range of variation; the energy characteristics include short-time energy and energy change rate; the speech rate characteristics include syllable rate and pause frequency; and the timbre characteristics include harmonic noise ratio and formant characteristics.

[0020] As a preferred embodiment of the present invention, step S4 includes the following steps: S41: Variance filtering: Remove features in the fused vector whose variance is close to zero; S42: Statistical significance test: The Mann-Whitney U test was used to evaluate the significance of the differences between the sleep disorder group and the normal group, and the false discovery rate was corrected to retain only the significant features with p values ​​less than the set threshold. S43: Correlation analysis: Calculate the Pearson correlation coefficient between each feature, obtain feature pairs with a correlation coefficient greater than a set threshold, and delete features in the feature pairs that have a lower correlation with the sleep disorder label. S44: Feature Standardization: The remaining features are processed using the Z-score standardization method to output a standardized fusion vector.

[0021] As a preferred embodiment of the present invention, the pre-trained machine learning model in step S5 adopts a gradient boosting decision tree model; the machine learning model includes the following optimizations: Hyperparameter optimization: Bayesian optimization was performed using the Optuna framework; Model accuracy evaluation: GroupKFold 5-fold cross-validation was used; Class imbalance handling: SMOTE oversampling or class weight adjustment strategy is adopted.

[0022] A machine learning-based multimodal sleep disorder detection system includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform any of the aforementioned machine learning-based multimodal sleep disorder detection methods.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a multimodal sleep disorder detection method based on video facial and audio signals. By innovatively introducing a quantitative calculation index for the degree of "dark circles under the eyes" and combining it with multimodal information fusion analysis of facial video and audio, a rapid, accurate, and objective detection of sleep disorders is achieved, providing effective technical support for the initial screening and auxiliary diagnosis of sleep disorders. Furthermore, the detection efficiency is far higher than that of traditional PSG examinations (which require overnight monitoring). Simultaneously, the method of this invention supports batch processing and can be deployed in scenarios such as community health service centers and corporate health check stations, enabling efficient sleep disorder screening for large-scale populations. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a multimodal sleep disorder detection method based on machine learning as described in Embodiment 1 of the present invention. Figure 2 This is a flowchart illustrating a multimodal sleep disorder detection method based on machine learning as described in Embodiment 2 of the present invention. Figure 3 This is a schematic diagram of the structure of a machine learning-based multimodal sleep disorder detection system as described in Embodiment 4 of the present invention. Detailed Implementation

[0025] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0026] Example 1 like Figure 1 The aforementioned multimodal sleep disorder detection method based on machine learning includes the following steps: S1: Acquire the RGB video and audio data of the person to be tested.

[0027] S2: Extract video features of the person to be detected based on the RGB video; and extract audio features of the person to be detected based on the audio data; the video features include dark circle features, facial expression features, and emotion features.

[0028] S3: The video features and audio features are spliced ​​and fused using principal component analysis to generate a fused vector.

[0029] S4: Filter the fusion vector according to the set conditions, perform standardization processing, and output the standardized fusion vector.

[0030] S5: Input the standardized fusion vector into a pre-trained machine learning model for recognition, and generate sleep disorder detection probability results.

[0031] Current video analysis methods lack quantitative analysis of "dark circles under the eyes," a typical indicator of sleep disorders. Dark circles are a typical external manifestation of insufficient sleep or poor sleep quality, but current technologies only remain at the qualitative observation level and have failed to establish objective quantitative indicators, resulting in a lack of important objective evidence for the assessment of sleep disorders.

[0032] Example 2 like Figure 2 As shown, this embodiment is a specific implementation of the multimodal sleep disorder detection method based on machine learning described in Embodiment 1, including the following steps: S1: Acquire the RGB video and audio data of the person to be tested.

[0033] This invention requires only a regular camera and microphone to collect data, eliminating the need for any wearable devices and achieving truly non-contact sleep disorder detection. Users can undergo testing in a natural state, avoiding the discomfort associated with wearing devices and eliminating the possibility of users altering their sleep behavior due to wearing monitoring devices, resulting in more accurate and reliable test results.

[0034] Furthermore, the audio data also includes the following preprocessing steps: based on the audio data, a noise reduction preprocessing method using spectral gating based on noisereduce is performed on the audio data in batches to remove the interference of device operating noise; and a dual-threshold endpoint detection algorithm is used to remove non-human voice parts in the recording to reduce interference with subsequent research.

[0035] S2: Extract video features of the person to be detected based on the RGB video; and extract audio features of the person to be detected based on the audio data; the video features include dark circle features, facial expression features, and emotion features.

[0036] Characteristics of dark circles: S211: Detect face regions from video frames of the RGB video using a deep learning face detection algorithm and locate facial key points; the facial key points mainly include key positions such as the inner and outer corners of the eyes and the midpoint of the lower eyelid.

[0037] Furthermore, the deep learning face detection algorithms include MediaPipe face network, RetinaFace algorithm, and MTCNN (Multi-task Cascaded Convolutional Networks), among others.

[0038] S212: Extract the under-eye region based on the facial key points; wherein, the under-eye region is: taking the midpoint of the lower eyelid of each eye as the center, extending to the inner and outer corners to obtain two fold lines, and shifting the two fold lines downward by a set distance (e.g., 15-25 pixels), and outputting the resulting polygonal region as the under-eye region, which is the main manifestation area of ​​dark circles.

[0039] S213: Extract the facial reference region based on the facial key points.

[0040] The facial reference area is selected as follows: the cheek area is chosen as the reference area, avoiding areas with obvious features such as the eyes, mouth, and nose. Specifically, the center of the cheek can be located by using facial key points, and a rectangular area of ​​a set size can be extracted as the facial reference area.

[0041] S214: Extract the grayscale features and Lab color space features (L-brightness, a-red-green; b-blue-yellow three color channels) of the area under the human eye and the facial reference area, and calculate the average feature value and dark circle index (DCI) of the two areas; wherein, the dark circle index includes: Dark circle grayscale index = average grayscale value of the area under the eyes G_under / average grayscale value of the facial reference area G_face; Dark circle brightness index = average L-channel value of the area under the eyes / average L-channel value of the reference area of ​​the face; The characteristic index of vascular dark circles is calculated as: average b-channel value of the facial reference area - average b-channel value of the area under the eyes.

[0042] S215: Calculate the dark circle features of a set consecutive frames in the RGB video based on the dark circle feature index.

[0043] This involves calculating the DCI value for multiple consecutive frames in a video sequence and extracting temporal features as dark circle features.

[0044] Furthermore, the dark circle characteristics include the mean (reflecting the overall severity of dark circles), standard deviation (reflecting the stability of dark circles), and trend (reflecting the degree of fatigue accumulation) of dark circle characteristic indicators.

[0045] Furthermore, the extraction of dark circle features in this embodiment also includes a step of extracting texture features in the under-eye area; S216: In-eye region texture feature extraction based on local binary pattern: After calculating the dark circle features in S215, to further distinguish between vascular and pigmented dark circles, this embodiment introduces Local Binary Pattern (LBP) texture analysis, specifically including the following steps: S2161: Region segmentation: Divide the area under the human eye into several sub-regions evenly.

[0046] Furthermore, for example: In this embodiment, the area under the human eye extracted in S212 is evenly divided into 3×3=9 sub-regions, and the size of each sub-region is W×H pixels (W and H are adaptively calculated according to the size of the face detection box. For example, if the width of the face box is F_w, then W=F_w / 10 and H=F_w / 15).

[0047] S2162: Uniform LBP (Uniform Local Binary Pattern) Encoding: For each pixel in each sub-region, using its corresponding grayscale value as the center, calculate the uniform local binary pattern value of the current pixel based on a circular neighborhood with set parameters. The calculation formula is as follows:

[0048]

[0049]

[0050] in, R represents the uniform local binary mode value of the circular neighborhood of the current pixel, where R is the set radius of the circular neighborhood and P is the set number of neighborhood points. Let p be the gray value of the p-th neighboring point. This is the grayscale value of the current pixel. s(x) is the uniformity measure of the current circular neighborhood, and s(x) is the sign function.

[0051] S2163: Histogram statistics: Statistically analyze the histogram of the uniform local binary pattern encoding in each sub-region to obtain a multi-dimensional feature vector.

[0052] In this embodiment, taking radius R=1 and neighborhood point number P=8 as an example, the histogram of LBP_{8,1}^{u2} is calculated for each sub-region, and the multi-dimensional feature vector is a 59-dimensional feature vector (58 uniform patterns + 1 non-uniform bucket). S2164: Texture roughness calculation: Calculate texture roughness based on the multidimensional feature vector, wherein the texture roughness includes: Information entropy: This reflects the complexity of the texture; vascular dark circles have a higher E value. energy: This reflects the uniformity of texture; pigmented dark circles have a higher Energy value. Contrast Ratio: This reflects the roughness of the texture; Where N is the number of dimensions of the multidimensional feature vector. Let be the frequency of occurrence of the j-th LBP mode in the sub-region.

[0053] S2165: Temporal aggregation: Calculate the texture roughness of a set consecutive frame in the RGB video (for example, set the frame number T=30, which corresponds to 1 second of video), output the corresponding mean, standard deviation and rate of change as multi-dimensional texture features, and concatenate the multi-dimensional texture features with the dark circle feature in S215 to output a new dark circle feature.

[0054] In this embodiment, taking 3×3=9 sub-regions as an example, the multi-dimensional texture feature is a 27-dimensional texture feature (9 sub-regions × 3 statistics), which is spliced ​​with the DCI feature in S215 to form a new dark circle feature.

[0055] This invention innovatively proposes a quantitative calculation method for dark circles under the eyes based on grayscale ratio (DCI), transforming traditional subjective observation into an objective quantitative indicator. This method is simple to calculate, highly real-time, and can quickly calculate the degree of dark circles while video is being captured, providing important objective evidence for sleep disorder detection. Experiments show that the DCI index is significantly negatively correlated with sleep quality scores (r=-0.68, p<0.001), effectively distinguishing between people with insufficient sleep and those with normal sleep.

[0056] Facial expression features: This invention uses the OpenFace toolkit for facial expression feature extraction. OpenFace is an open-source facial behavior analysis tool capable of extracting rich facial features in real time. Specifically, the extracted features include: S221: Facial motion features: Extract the intensity values ​​and existence status of several set facial motions in the RGB video and output them as facial motion features.

[0057] Facial Action Units (AUs): In this embodiment, OpenFace extracts the intensity values ​​and presence states of 17 facial actions based on the Facial Action Coding System (FACS). The defined facial actions include any one or more of the following: AU01 (inner eyebrow raised), AU02 (outer eyebrow raised), AU04 (eyebrow drooping), AU05 (upper eyelid raised), AU06 (cheek raised), AU07 (eyelid tightening), AU09 (nose wrinkling), AU10 (upper lip raised), AU12 (corner of mouth raised), AU14 (lip pursed), AU15 (corner of mouth turned down), AU17 (chin raised), AU20 (lips stretched), AU23 (lips tightened), AU25 (lips parted), AU26 (chin drooping), and AU45 (blinking). These AU features can finely depict changes in facial expressions, reflecting states such as fatigue and drowsiness.

[0058] S222: Gaze direction feature: Extract eye feature points from the RGB video, generate a gaze vector in three-dimensional space, and output the gaze direction feature.

[0059] Gaze Direction: OpenFace estimates the gaze direction of both eyes using eye feature points, outputting a gaze vector (x, y, z components) in three-dimensional space. By analyzing the stability and frequency of change of gaze direction, sleep disorder-related manifestations such as inattention and wandering gaze can be detected. Extracted temporal features include: gaze direction mean, standard deviation, amplitude of change, and fixation concentration.

[0060] S223: Head posture features: Detect the rotation angle of the head of the person to be detected in the RGB video in three-dimensional space; the rotation angle includes pitch angle, yaw angle and roll angle.

[0061] Head posture: OpenFace estimates the rotation angles of the head in three-dimensional space, including pitch, yaw, and roll. Patients with sleep disorders often exhibit fatigue characteristics such as head drooping and frequent head nodding. Extracted head posture features include: mean and standard deviation of each angle, frequency of head drooping, and amplitude of head swaying.

[0062] S224: Facial geometric features: Detect the coordinates of facial key points in the RGB video; and calculate facial geometric features based on the coordinates of the facial key points.

[0063] Facial Landmarks: OpenFace detects the coordinates of 68 facial landmarks, including eyebrow contours, eye contours, nose contours, mouth contours, and jaw contours. Based on these landmarks, facial geometric features can be calculated, including eye aspect ratio (EAR, used to detect closed eyes), mouth opening degree, and facial symmetry.

[0064] S225: Perform temporal analysis on the facial movement features, gaze direction features, head posture features, and facial geometric features respectively, calculate the statistical features, frequency domain features, and dynamic features of each feature, and output the facial expression features of the person to be detected, thereby comprehensively depicting the dynamic change pattern of facial expressions.

[0065] Furthermore, the statistical characteristics include any one or more of the following: mean, standard deviation, maximum value, minimum value, and range; The frequency domain features are obtained by extracting frequency components through Fourier transform; The dynamic characteristics include the rate of change and / or the amplitude of fluctuation.

[0066] Emotional characteristics: This invention employs the PosterV2 deep learning model for emotion feature extraction. PosterV2 is a facial expression recognition model based on Vision Transformer, capable of extracting high-level emotional features from facial images. These emotional features include emotion category prediction and deep emotional features; the specific implementation is as follows: (1) Model architecture: PosterV2 adopts a pyramid structure (Pyramid Vision Transformer), which effectively captures multi-scale facial expression information through hierarchical feature extraction and progressive spatial reduction. The model contains multiple Transformer encoder layers, and each layer models the relationship between different regions of the face through a self-attention mechanism.

[0067] (2) Emotion Category Prediction: The PosterV2 model outputs the probability distribution of 7 basic emotion categories, including: Angry, Disgust, Fear, Happy, Sad, Surprise, and Neutral. The emotion category prediction is the probability distribution of the basic emotion categories output by the PosterV2 deep learning model; patients with sleep disorders often exhibit characteristics such as an increased proportion of sad and neutral emotions and a decreased proportion of happy emotions.

[0068] (3) Deep Feature Extraction: In addition to the emotion category output, this invention also extracts the feature vectors of the intermediate layers of the PosterV2 model as deep emotion features. Specifically, the global average pooling features output from the last Transformer encoder layer are extracted, with a dimension of 512. These deep emotion features contain rich high-level semantic information of facial expressions and are of great value for sleep disorder detection.

[0069] (4) Temporal sentiment analysis: Temporal analysis is performed on the sentiment prediction results of multiple consecutive frames in the video sequence to extract features such as sentiment stability, sentiment transition frequency, and duration of dominant sentiment. For example, frequent mood fluctuations may reflect mood instability caused by poor sleep quality.

[0070] Furthermore, the audio features are extracted using the OpenSMILE analysis library; the audio features include the following speech features related to sleep disorders: fundamental frequency features, energy features, speech rate features, and timbre features: The fundamental frequency characteristics include the fundamental frequency mean, standard deviation, and range of variation, which are used to reflect the fatigue level of the voice; the energy characteristics include short-time energy and energy change rate, which are used to reflect the vitality of the voice; the speech rate characteristics include syllable rate and pause frequency, which are used to reflect the agility of thought; and the timbre characteristics include harmonic noise ratio and formant characteristics, which are used to reflect the fatigue state of the vocal cords.

[0071] S3: The video features and audio features are spliced ​​and fused using principal component analysis to generate a fused vector.

[0072] This invention fuses features from dark circles under the eyes, OpenFace facial expression features, PosterV2 emotion features, and audio features to construct a comprehensive feature set for sleep disorder detection. (1) Summary of video features: • Characteristics of dark circles: time-series features such as DCI mean, standard deviation, and trend of change; • OpenFace features: mean and frequency of 17 AU intensities, gaze direction statistics, head pose statistics, and facial geometric features; • PosterV2 features: mean probability of 7 emotion categories, 512-dimensional deep emotion feature vector, and emotion stability index.

[0073] (2) Audio features: Extract the following speech features related to sleep disorders: fundamental frequency features (fundamental frequency mean, standard deviation, range of variation); energy features (short-time energy, rate of energy change); speech rate features (syllable rate, pause frequency); timbre features (harmonic noise ratio, formant features).

[0074] (3) Feature fusion strategy: An early fusion strategy is adopted to splice all the above features at the feature level to form a multimodal fused feature vector. For the problem of inconsistent feature dimensions, principal component analysis (PCA) is used to reduce dimensionality, retaining more than 95% of the variance information, ensuring that the fused features contain multimodal information and avoid the curse of dimensionality.

[0075] This invention, by fusing features from dark circles under the eyes, facial video, and audio, enables the system to capture the physiological manifestations of sleep disorders from multiple dimensions, overcoming the limitations of single-modal information. The multimodal fusion model achieves an AUC of 0.76 on the sleep disorder detection task, significantly outperforming the single-modal model (video modality AUC = 0.59, audio modality AUC = 0.58), demonstrating the effectiveness of the multimodal fusion strategy.

[0076] S4: Filter the fusion vector according to the set conditions, perform standardization processing, and output the standardized fusion vector.

[0077] S41: Variance Filtering: Removes features with variance close to zero from the fused vector. These features change almost no in the dataset and contribute very little to the model's predictive ability.

[0078] S42: Statistical significance test: The Mann-Whitney U test is then used to assess the significance of the differences between the sleep disorder group and the normal group. The results are corrected by the false discovery rate (FDR), and only significant features with p-values ​​(representing the probability of obtaining such extreme data when the null hypothesis is true) less than the set threshold are retained.

[0079] S43: Correlation analysis: Calculate the Pearson correlation coefficient between each feature, obtain feature pairs with a correlation coefficient greater than a set threshold (preferably 0.9 in this embodiment), and delete features in the feature pairs that have a lower correlation with the sleep disorder label, thereby reducing redundant features.

[0080] S44: Feature Standardization: The remaining features are processed using the Z-score standardization method, outputting a standardized fusion vector. This ensures that each feature has zero mean and unit variance, eliminating the impact of dimensional differences on model training.

[0081] S5: Input the standardized fusion vector into a pre-trained machine learning model for recognition, and generate sleep disorder detection probability results.

[0082] Furthermore, the pre-trained machine learning model employs a gradient boosting decision tree model, such as CatBoost, LightGBM, or XGBoost, and the optimal model can be selected based on actual needs. Specifically, the machine learning model includes the following optimizations: Hyperparameter optimization: Bayesian optimization is performed using the Optuna framework; in this embodiment, the optimal combination of hyperparameters is intelligently searched through 200 trials. The hyperparameters include key parameters such as learning rate, tree depth, number of iterations, and L2 regularization coefficient.

[0083] Model accuracy evaluation: Group KFold 5-fold cross-validation was used to ensure that samples from the same subject did not cross folds, guaranteeing the accuracy and reliability of the model evaluation. Simultaneously, the AUC difference between the training and validation sets was monitored to prevent overfitting.

[0084] Class Imbalance Handling: SMOTE oversampling or class weight adjustment strategies are employed. To address the potential class imbalance between normal and abnormal samples in sleep disorder detection, SMOTE oversampling or class weight adjustment strategies are used to improve the model's ability to identify the minority class.

[0085] This invention employs a gradient boosting decision tree model, which offers excellent interpretability. Through feature importance analysis, the contribution of each feature to sleep disorder detection can be clearly defined, providing decision-making references for clinicians. Simultaneously, the system outputs the sleep disorder risk, facilitating physicians in developing appropriate intervention plans based on this risk assessment.

[0086] Furthermore, the multimodal sleep disorder detection method based on machine learning described in this invention can be implemented as a software system. The entire system uses main.py as the entry point and connects modules such as config.py, data_loader.py, feature_engineering.py, model_trainer.py, model_evaluator.py, and utils.py. Each module has a clear division of labor and works together to complete the entire process from data processing to model training, evaluation, and prediction. Specifically: config.py is responsible for global configuration management; data_loader.py is responsible for data loading and preprocessing; feature_engineering.py is responsible for feature extraction and filtering, including sub-modules such as OpenFace feature extraction, PosterV2 feature extraction, and dark circle calculation; model_trainer.py is responsible for model training and optimization; model_evaluator.py is responsible for model evaluation and prediction; and utils.py provides utility functions such as logging, model saving and loading, and checkpoint management. main.py, as the main program entry point, provides a flexible command-line interface that supports user-defined training tasks, model types, and feature patterns.

[0087] The system of this invention adopts a modular design, with clear responsibilities for each module, facilitating maintenance and upgrades. A comprehensive logging system and checkpoint management function ensure traceability of system operation status and recoverability of the training process. The model serialization function supports model saving and loading, facilitating model version management and deployment updates. Furthermore, it can complete the initial screening of sleep disorders for a single user within 5 minutes, with detection efficiency far exceeding that of traditional PSG checks (which require overnight monitoring). Simultaneously, the system supports batch processing and can be deployed in scenarios such as community health service centers and corporate health check stations, achieving efficient sleep disorder screening for large populations.

[0088] Example 3 This embodiment is a specific implementation for verifying the feasibility and effectiveness of the multimodal sleep disorder detection method based on machine learning described in Embodiment 2. The relevant experimental parameters and results in this embodiment are shown below: 1. Data source and annotation method: This embodiment uses a self-built Mental Disorder 4D (MD4D) dataset, with a total of 488 participants, including 278 in the sleep disorder group and 210 in the control group. All participants underwent mental health screening using the MINI Clinical Structured Interview tool and obtained gold standard labels. Each participant recorded a 30-second video segment, including RGB images and synchronized depth images, with a resolution of 640×640, a frame rate of approximately 17fps, and a total of 900 frames. Informed consent was obtained for all samples before collection, and the process underwent ethical review.

[0089] 2. Data preprocessing: The preprocessing procedure proposed in Example 2 includes: (1) After region cropping, bilinear interpolation is used to unify the pixels to 256×256; (2) Linear interpolation and Gaussian smoothing are used to fill in the frame sequence and reduce noise.

[0090] 3. Training parameters for machine learning models: • Optimizer: Adam; • Initial learning rate: 1e-4, dynamically adjusted using a cosine annealing strategy; Batch size: 8; • Epoch: 200 rounds, employing an early stopping strategy to prevent overfitting; • Loss function: Cross-entropy loss is used for classification, and MSE loss is used for regression.

[0091] 4. Comparative Experiment and Results: Five-fold cross-validation was performed on the MD4D dataset, and the results are shown in the table below: Table of Cross-Validation Results Precision Recall AUC Face video RGB monomodal prediction 0.62 0.60 0.70 Audio single-modal prediction 0.58 0.60 0.58 This invention provides multimodal fusion prediction 0.69 0.67 0.76 Based on the above experimental results, it can be seen that the multimodal sleep disorder detection of the present invention can greatly improve the accuracy and recall of sleep detection. At the same time, according to the AUC (Area Under Curve) results, the performance of the machine learning model of the present invention is also greatly improved.

[0092] Example 4 like Figure 3As shown, a machine learning-based multimodal sleep disorder detection system includes at least one processor, a memory communicatively connected to the at least one processor, and at least one input / output interface communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enables the at least one processor to perform the machine learning-based multimodal sleep disorder detection method described in the foregoing embodiments. The input / output interface may include a display, keyboard, mouse, and USB interface for inputting and outputting data.

[0093] Furthermore, this machine learning-based multimodal sleep disorder detection system can be a desktop computer, mobile phone, tablet computer, wearable machine learning-based multimodal sleep disorder detection system, or any other machine learning-based multimodal sleep disorder detection system capable of deep information recognition.

[0094] Furthermore, the processor may include one or more processing cores. The processor connects various parts of the machine learning-based multimodal sleep disorder detection system using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by calling data stored in memory. Optionally, the processor may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also be implemented separately as a communication chip, without being integrated into the processor.

[0095] The memory may include random access memory (RAM) or read-only memory (ROM). The memory can be used to store instructions, programs, code, code sets, or instruction sets, such as instructions or code sets used to implement the machine learning-based multimodal sleep disorder detection method provided in this application embodiment. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc. The data storage area may also store data created during the use of the machine learning-based multimodal sleep disorder detection system (such as a mapping table of modulation sequences and depths, image data, spectrogram data, etc.).

[0096] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory, magnetic disks, or optical disks.

[0097] When the integrated units of the present invention are implemented as software functional units and sold or used as independent products, they can also be stored in a computer-readable storage medium. The computer-readable storage medium stores program code, which can be called by a processor to execute the methods described in the above method embodiments. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes electronic memories such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code that executes any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.

[0098] This invention is used in a reasonable manner.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal sleep disorder detection method based on machine learning, characterized in that, Includes the following steps: S1: Acquire the RGB video and audio data of the person to be tested; S2: Extract video features of the person to be detected based on the RGB video; and extract audio features of the person to be detected based on the audio data; the video features include dark circle features, facial expression features, and emotion features; S3: The video features and audio features are spliced ​​and fused using principal component analysis to generate a fusion vector; S4: Filter the fusion vector according to the set conditions, perform standardization processing, and output the standardized fusion vector; S5: Input the standardized fusion vector into a pre-trained machine learning model for recognition, and generate sleep disorder detection probability results.

2. The multimodal sleep disorder detection method based on machine learning according to claim 1, characterized in that, The extraction of the dark circle feature includes the following steps: S211: Detect face regions from video frames of the RGB video using a deep learning face detection algorithm and locate facial key points; the facial key points include the inner and outer corners of both eyes and the midpoint of the lower eyelid; S212: Extract the lower eye region based on the facial key points; wherein, the lower eye region is: taking the midpoint of the lower eyelid of each eye as the center, extending to the inner and outer corners to obtain two fold lines, and shifting the two fold lines downward by a set distance to output the resulting polygonal region as the lower eye region; S213: Extract a facial reference region based on the facial key points; the facial reference region is: a rectangular region of a set size extracted from the center of the cheek in the face region, and output as the facial reference region; S214: Extract the grayscale features and Lab color space features of the area under the human eyes and the facial reference area, and calculate the average feature value and dark circle feature index of the two areas; wherein, the dark circle feature index includes: Dark circle grayscale index = average grayscale value of the area under the eyes / average grayscale value of the reference area of ​​the face; Dark circle brightness index = average L-channel value of the area under the eyes / average L-channel value of the reference area of ​​the face; The characteristic index of vascular dark circles is calculated as: average b-channel value of the facial reference area - average b-channel value of the under-eye area. S215: Calculate the dark circle features of a set series of frames in the RGB video based on the dark circle feature indicators; the dark circle features include the mean, standard deviation and trend of each dark circle feature indicator.

3. The multimodal sleep disorder detection method based on machine learning according to claim 2, characterized in that, The extraction of dark circle features also includes S216 under-eye region texture feature extraction; including the following steps: S2161: Divide the area under the human eye into several sub-regions evenly; S2162: For each pixel in each sub-region, using its corresponding grayscale value as the center, calculate the uniform local binary mode value of the current pixel based on the circular neighborhood defined by the set parameters. The calculation formula is as follows: in, R represents the uniform local binary mode value of the circular neighborhood of the current pixel, where R is the set radius of the circular neighborhood and P is the set number of neighborhood points. Let p be the gray value of the p-th neighboring point. This is the grayscale value of the current pixel. s(x) is the uniformity measure of the current circular neighborhood, and s(x) is the sign function; S2163: Calculate the histogram of the uniform local binary pattern encoding in each sub-region to obtain a multi-dimensional feature vector; S2164: Calculate the texture roughness based on the multidimensional feature vector, wherein the texture roughness includes: Information entropy: ; energy: ; Contrast Ratio: ; Where N is the number of dimensions of the multidimensional feature vector. Let be the frequency of occurrence of the j-th LBP mode in the sub-region; S2165: Calculate the texture roughness of the set consecutive frames in the RGB video, output the corresponding mean, standard deviation and rate of change as multi-dimensional texture features, and concatenate the multi-dimensional texture features with the dark circle features in S215 to output a new dark circle feature.

4. The multimodal sleep disorder detection method based on machine learning according to claim 1, characterized in that, The facial expression features were extracted using the OpenFace toolkit, including the following steps: S221: Facial motion features: Extract the intensity values ​​and existence states of several set facial motions in the RGB video and output them as facial motion features; wherein, the set facial motions include any one or more of the following: inner eyebrow raising, outer eyebrow raising, eyebrow drooping, upper eyelid raising, cheek raising, eyelid tightening, nose wrinkling, upper lip raising, corner of mouth raising, pursing lips, corner of mouth turning down, chin raising, lips stretching, lips tightening, lips separating, chin drooping, and blinking; S222: Gaze direction feature: Extract eye feature points from the RGB video, generate a gaze vector in three-dimensional space, and output the gaze direction feature; S223: Head posture features: Detect the rotation angle of the head of the person to be detected in the RGB video in three-dimensional space; the rotation angle includes pitch angle, yaw angle and roll angle; S224: Facial geometric features: Detect the coordinates of facial key points in the RGB video; and calculate facial geometric features based on the coordinates of the facial key points; the facial geometric features include the aspect ratio of the eyes, the degree of mouth opening, and facial symmetry; S225: Perform temporal analysis on the facial motion features, gaze direction features, head posture features, and facial geometric features respectively, calculate the statistical features, frequency domain features, and dynamic features of each feature, and output the facial expression features of the person to be detected.

5. The multimodal sleep disorder detection method based on machine learning according to claim 4, characterized in that, The statistical characteristics include any one or more of the following: mean, standard deviation, maximum value, minimum value, and range; The frequency domain features are obtained by extracting frequency components through Fourier transform; The dynamic characteristics include the rate of change and / or the amplitude of fluctuation.

6. The multimodal sleep disorder detection method based on machine learning according to claim 1, characterized in that, The emotional features were extracted using the PosterV2 deep learning model; The emotional features include emotion category prediction and deep emotion features; The emotion category prediction is the probability distribution of the basic emotion category output by the PosterV2 deep learning model; The deep sentiment feature is the globally averaged pooled feature output by the last layer Transformer encoder of the PosterV2 deep learning model.

7. The multimodal sleep disorder detection method based on machine learning according to claim 1, characterized in that, The audio features were extracted using the OpenSMILE analysis library; these features include fundamental frequency features, energy features, speech rate features, and timbre features. The fundamental frequency characteristics include the fundamental frequency mean, standard deviation, and range of variation; the energy characteristics include short-time energy and energy change rate; the speech rate characteristics include syllable rate and pause frequency; and the timbre characteristics include harmonic noise ratio and formant characteristics.

8. The multimodal sleep disorder detection method based on machine learning according to claim 1, characterized in that, S4 includes the following steps: S41: Variance filtering: Remove features in the fused vector whose variance is close to zero; S42: Statistical significance test: The Mann-Whitney U test was used to evaluate the significance of the differences between the sleep disorder group and the normal group, and the false discovery rate was corrected to retain only the significant features with p values ​​less than the set threshold. S43: Correlation analysis: Calculate the Pearson correlation coefficient between each feature, obtain feature pairs with a correlation coefficient greater than a set threshold, and delete features in the feature pairs that have a lower correlation with the sleep disorder label. S44: Feature Standardization: The remaining features are processed using the Z-score standardization method to output a standardized fusion vector.

9. The multimodal sleep disorder detection method based on machine learning according to claim 1, characterized in that, The pre-trained machine learning model in S5 employs a gradient boosting decision tree model; the machine learning model includes the following optimizations: Hyperparameter optimization: Bayesian optimization was performed using the Optuna framework; Model accuracy evaluation: GroupKFold 5-fold cross-validation was used; Class imbalance handling: SMOTE oversampling or class weight adjustment strategy is adopted.

10. A multimodal sleep disorder detection system based on machine learning, characterized in that, The device includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform a machine learning-based multimodal sleep disorder detection method according to any one of claims 1 to 9.

Images