Multi-modal fusion obstructive sleep apnea identification method based on DCCA

Abstract

The invention provides a multi-modal fusion obstructive sleep apnea (OSA) identification method based on deep canonical correlation analysis (DCCA). For multi-modal characteristics, a whole process from data preprocessing, feature extraction and screening to feature fusion and classification is designed. Firstly, AHI indexes, voice data and craniofacial image data of a subject are collected and preprocessed, and DeepSpectrum voice features and geometric morphology image features based on MediaPipe are extracted. Then, feature screening is carried out by adopting a competitive group optimizer (CSO) and a ReliefF algorithm respectively; for a class imbalance problem, an oversampling (AMDO) method based on an adaptive mahalanobis distance is introduced to balance data. Non-linear mapping and deep fusion are carried out on high-dimensional features of the two modals through DCCA, low-dimensional high-correlation features are generated, and information complementarity between the modals is improved. And finally, inputting the fused features into a classifier and evaluating the performance by using five-fold cross validation. According to the invention, effective fusion between modals is realized by using DCCA, and a new method is provided for early screening and auxiliary diagnosis of OSA.

Description

Technical Field

[0001] This invention belongs to the field of diagnostic screening technology for obstructive sleep apnea syndrome, specifically involving a multimodal fusion method for identifying obstructive sleep apnea based on deep canonical correlation analysis. Background Technology

[0002] Obstructive sleep apnea (OSA) is a common respiratory disorder characterized by upper airway collapse (especially in the pharynx) during sleep, leading to interruption or restriction of airflow, resulting in decreased oxygen saturation and disrupted sleep structure. OSA significantly impacts patients' quality of life, often accompanied by symptoms such as daytime sleepiness, depression, cognitive impairment, and mood changes. Approximately one billion adults worldwide are affected, and its incidence continues to increase due to population aging and rising obesity rates. Untreated OSA not only negatively impacts patients and their families but also imposes a significant socioeconomic burden, puts pressure on healthcare systems, and poses public safety risks.

[0003] The diagnosis of OSA primarily relies on the apnea-hypopnea index (AHI), which assesses disease severity by measuring the frequency of apnea and hypopnea events during sleep per hour. However, AHI measurement requires nocturnal polysomnography (PSG). Although PSG is widely recognized as the gold standard for OSA diagnosis, its complexity, time-consuming nature, and high cost limit its application in large-scale screening and early diagnosis. Clinical questionnaires, as an auxiliary tool, provide a simple, quick, and low-cost initial screening method for OSA by collecting information such as patients' physical characteristics, medical history, and symptoms. However, the predictive performance of questionnaires is greatly affected by patient subjectivity, resulting in limited accuracy and reliability, which is insufficient to meet the needs of precise diagnosis. Therefore, researchers are attempting to screen OSA patients in a simpler, non-invasive, and cost-effective manner, gradually shifting their attention to automated recognition methods based on speech signals and craniofacial images.

[0004] Studies have shown that OSA patients exhibit characteristic changes in their upper airway structure, with soft tissue collapse obstructing the airway during sleep, thus affecting the acoustic properties of their speech. Simultaneously, imaging studies have revealed a close correlation between certain craniofacial features (such as skeletal and soft tissue components) and the pathogenesis of OSA. These modalities offer the advantages of being non-invasive and readily available. However, OSA identification methods based on a single modality have significant limitations. Speech signals are susceptible to environmental noise, individual differences, or signal distortion, while craniofacial images are easily affected by lighting, changes in facial posture, or fluctuations in facial expression. These factors can lead to unstable feature extraction, thereby affecting the accuracy and robustness of the diagnosis.

[0005] Therefore, fusing information from both speech signals and craniofacial images can not only compensate for the shortcomings of single-modal methods but also fully leverage the complementarity of each modality, improving information representation and system robustness. Currently, multimodal fusion mainly employs two approaches: feature-level fusion and decision-level fusion. Decision-level fusion assigns weights to the independent decision results of each modality or performs formula calculations to obtain the final decision; however, this method ignores the interactions between modalities and fails to fully utilize their correlation and complementarity. In contrast, feature-level fusion directly integrates the features of multimodal data, such as through simple concatenation or weighted concatenation. While intuitive, this method is prone to the curse of dimensionality and ignores differences in scale, distribution, and structure between modalities, making it difficult to effectively capture the synergistic effects of information between modalities.

[0006] In summary, in the research on multimodal fusion OSA recognition, how to effectively capture the correlation between modalities, utilize the differential information of modalities, and remove redundant features remains a key problem that urgently needs to be solved. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal fusion obstructive sleep apnea (OSA) identification method based on DCCA.

[0008] This invention first preprocesses, extracts, and filters features from the collected speech signals and craniofacial image data of the subjects, and performs oversampling processing on imbalanced data. Then, deep fusion of speech signals and craniofacial image features is achieved through DCCA (Discretionary DC-CA), and finally, the fused features are input into a classifier to accurately identify and classify the severity of OSA (Obstructive Stress Awareness).

[0009] The specific steps of this invention are as follows:

[0010] Step 1: Collect the subject's AHI index, speech data, and craniofacial image data. The AHI index is used as the basis for subsequent classification tasks; speech data includes vowels, isolated syllables, and sentences; craniofacial image data includes facial images of the subject taken from three different angles (front, 90° right, and 90° left).

[0011] Step 2: Preprocess the acquired speech and craniofacial image data. For speech data, silence segments are removed and noise is reduced to ensure only meaningful speech information is retained. For image data, brightness enhancement and sharpening are performed first, followed by facial region detection, image cropping, and size standardization to ensure data consistency and high quality.

[0012] Step 3: DeepSpectrum feature extraction is performed on the preprocessed speech data. Specifically, the Mel spectrum of the audio is extracted first, and then the image is input into the pre-trained network of DenseNet 121 to extract 1024-dimensional features. For the preprocessed craniofacial image data, MediaPipe is used to detect facial feature points and calculate geometric morphological features such as distance, angle and area, finally obtaining 724-dimensional features.

[0013] Step 4: Select an appropriate feature selection method based on the nature of different features. For speech features, a competitive swarm optimizer (CSO) is used for feature selection; while for craniofacial image data, the ReliefF algorithm is used for feature selection to improve classification performance.

[0014] Step 5: For imbalanced data, use the Adaptive Mahalanobis Distance-based Over-sampling (AMDO) method to perform modal oversampling, balancing the number of samples from different classes to improve the robustness and accuracy of the classifier.

[0015] Step 6: Input the processed high-dimensional feature matrices of the two modalities into two deep neural networks (DNNs). Perform nonlinear mapping on the features of each modality to generate low-dimensional feature representations. Based on this, use a loss function based on canonical correlation analysis (CCA) to calculate the covariance matrix of the projection matrices of the two modalities, and maximize the correlation between the two modalities through eigenvalue decomposition. Finally, output the low-dimensional feature representations of the two modalities after network processing. These feature representations have high correlation, effectively fusing speech and image information and providing more discriminative features for subsequent classification tasks.

[0016] Step 7: Input the low-dimensional features processed by DCCA into the classifier, use the five-fold cross-validation method for model training and evaluation, and finally output the model's evaluation metrics, including accuracy, precision, F1 score, and AUC score, in order to comprehensively evaluate the model's performance.

[0017] Based on the above, the innovation and advantages of this invention are as follows:

[0018] (1) A feature fusion method based on deep canonical correlation analysis (DCCA) is adopted to effectively capture the correlation between speech signals and craniofacial images. At the same time, it utilizes the differential information of the modes to overcome the shortcomings of traditional feature splicing or decision layer fusion methods. In the process of deep feature fusion, by maximizing the correlation between modes, the discriminative ability of the fused features is further enhanced, providing more robust and accurate support for OSA classification.

[0019] (2) This invention provides a complete OSA automatic recognition process, covering the entire process from data acquisition, preprocessing, feature extraction and filtering, oversampling, modality fusion to final classification, ensuring full utilization and efficient processing of multimodal data. It systematically solves the problems that may occur in the feature extraction stage of speech and image modalities, such as environmental interference and individual differences, laying a solid foundation for subsequent classification.

[0020] (3) The Adaptive Mahalanobis Distance Oversampling (AMDO) method is introduced to address the class imbalance problem by performing modal processing, effectively balancing the sample distribution, and improving the classifier's sensitivity and prediction ability for minority class samples.

[0021] (4) Based on the characteristics of different modal features, two feature selection methods, Competitive Group Optimizer (CSO) and ReliefF, are used to optimize speech and craniofacial image features respectively, thereby reducing redundant features and improving the computational efficiency and accuracy of the classification model.

[0022] (5) The inputs are voice signals and craniofacial images, both of which are non-invasive data. The acquisition method is simple and low-cost, making it suitable for large-scale screening and application promotion. It provides a practical alternative or supplement to the current diagnostic tools (such as PSG and questionnaires). Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the overall process of the DCCA-based multimodal fusion obstructive sleep apnea identification method of the present invention.

[0024] Figure 2 This is a schematic diagram of the modules of the multimodal fusion obstructive sleep apnea identification method based on DCCA of the present invention. Detailed Implementation

[0025] This invention discloses a multimodal fusion obstructive sleep apnea (OSA) identification method based on deep canonical correlation analysis (DCCA), which can achieve deep fusion of speech signals and craniofacial image data and accurate classification of OSA severity. The technical solution of this invention is described in detail below with reference to the accompanying drawings, but is not limited to the described content.

[0026] Combination Figure 2As shown, the modules of this invention can be divided into: a data acquisition module, a data preprocessing module, a feature extraction module, a feature filtering module, a data oversampling module, a modality fusion module, and a classification module. Combined with... Figure 1 As shown, the specific steps are as follows:

[0027] Step 1: Collect the subject's AHI index, voice data, and craniofacial image data.

[0028] First, the AHI index of the subjects was obtained through overnight polysomnography (PSG) equipment, which was used for the subsequent binary classification task. Specifically, two AHI grading standards were used for classification: one was no OSA / mild (AHI < 15 times / hour) and moderate to severe (15 times / hour ≤ AHI); the other was no OSA / mild to moderate (AHI < 30 times / hour) and severe (30 times / hour ≤ AHI).

[0029] The participants' speech data was recorded using a high-fidelity digital recording device (Roland R-44, 48kHz sampling rate), with the microphone pointed directly at the participants' mouths. The audio files were saved in WAV format. The speech data included 5 vowels ( / a / , / e / , / i / , / o / , / u / ), 24 isolated syllables ( / ji1 / , / ji2 / , / ji3 / , / ji4 / , / ke2 / , / ka1 / , / ku3 / , / ku4 / , / hu1 / , / hu2 / , / hu3 / , / hu4 / , / ma1 / , / ma2 / , / ma3 / , / ma4 / , / mo1 / , / mo2 / , / mo3 / , / mo4 / , / heng1 / , / hang2 / , / hong3 / , / meng4 / ), and a one-minute sentence.

[0030] Craniofacial images were captured using a Hikvision E14a camera. Subjects sat upright in the experimental environment, and the camera captured images from the front, right 90° (facing the right side of the face), and left 90° (facing the left side of the face). The frontal image ensured coverage of the area from the top of the head to the ears and upper neck; the side image included the nose, top of the head, back of the head, and the area where the chin meets the upper neck.

[0031] Step 2: Perform data preprocessing on the collected voice data and craniofacial image data.

[0032] For the audio data, Adobe Audition software was used to remove silent segments, improving data quality by automatically identifying and removing silent segments in the audio. In addition, noise reduction processing was employed to remove background noise from the recordings, ensuring the clarity of the audio signal.

[0033] Image data is preprocessed using the MediaPipe toolkit. First, brightness enhancement and sharpening are performed to improve image clarity and contrast. Then, facial regions in the image are automatically identified, and facial key points are marked. Based on the marked facial key points, the facial regions are cropped, and the images are uniformly adjusted to 480×480 pixels to ensure data consistency and high quality.

[0034] Step 3: Extract features from the preprocessed speech and craniofacial image data.

[0035] For the preprocessed speech data, a Short-Time Fourier Transform (STFT) is first used to convert it into a time-frequency representation. Then, a Mel filter bank is used to convert the linear frequency scale of the audio signal to the Mel frequency scale, and the conversion formula is as follows:

[0036]

[0037] The Mel frequency scale is more sensitive to low frequencies and sparser to high frequencies. Next, the energy spectrum of each Mel band is calculated, and a logarithmic transformation is used to simulate the nonlinear response of the human ear to volume, ultimately obtaining the Mel spectrogram. Subsequently, the Mel spectrogram is input into a pre-trained DenseNet-121 network, which contains multiple convolutional and pooling layers. After removing the final classification and fully connected layers, the network is flattened to extract 1024-dimensional features.

[0038] For the preprocessed craniofacial image data, 468 facial keypoints were extracted using the MediaPipe tool. Based on these keypoints, the distances between each facial keypoint were calculated to obtain linear features, such as the distance between the eyes and the distance between the eyebrows and eyes. Facial angular features, such as the angle between the corners of the mouth and the tip of the nose, and the angle between the eyes and the tip of the nose, were calculated based on the positions of the facial keypoints. Furthermore, the areas of different facial regions, such as the areas of the eyes, nose, and mouth, were calculated using the facial keypoints. Finally, 724-dimensional features were obtained through geometric operations.

[0039] Step 4: Select an appropriate feature selection method based on the nature of different features.

[0040] For speech features, a Competitive Swarm Optimizer (CSO) is used for feature selection. CSO is a swarm intelligence-based optimization algorithm that iteratively optimizes by simulating the behavior of a swarm of particles to select the most representative and classifying features. Its basic principle can be expressed as follows:

[0041]

[0042] in, V represents the state of the i-th particle in the t-th iteration (i.e., the selection of features). i t It is the particle's velocity, r i The inertial term is randomly selected, and α and β are parameters that adjust particle updates. After CSO filtering, 263-dimensional speech features are ultimately retained.

[0043] For craniofacial image data, the ReliefF algorithm is used for feature selection. The ReliefF algorithm evaluates the importance of features based on the weights of instances. Its core idea is to select the most discriminative features by calculating the discriminative power of each feature among similar instances. The formula for ReliefF is as follows:

[0044]

[0045] Among them, w i N is the importance weight of feature i, and N is the current instance x. i The neighbor set, d i It is the discriminative power of feature i for the samples, x nearest and x secondnearest These represent the nearest neighbor and the second nearest neighbor of the current instance, respectively. After filtering using ReliefF, the final 250-dimensional craniofacial image features are retained.

[0046] Step 5: For class-imbalanced data, use the Adaptive Mahalanobis Distance-based Over-sampling (AMDO) method to perform submodal oversampling.

[0047] The AMDO method guides the generation of new samples during oversampling by calculating the relationship between Mahalanobis distance and class. This method focuses on utilizing the distributional relationships between samples to generate more representative minority class samples. The specific steps are as follows:

[0048] First, calculate the Mahalanobis distance for each minority class sample, assuming each minority class sample x min The mean of all minority class samples is μ. min The covariance matrix is ​​Σ min :

[0049]

[0050] Next, we calculate the adaptive weights for the Mahalanobis distance. Based on the similarity between samples, we assign a weight to minority class samples. This weight reflects the "density" of the samples in the feature space. Samples that are far from the mean and relatively isolated in the feature space are likely to be more representative of the minority class and are therefore given higher weights. The formula for calculating the weights is:

[0051]

[0052] Among them, w min (x) is the adaptive weight of sample x, D M (x) is the Mahalanobis distance of sample x. The larger the weight, the more likely the sample is to be selected during the oversampling process.

[0053] Then, based on the calculated adaptive weights, minority class samples are selected for oversampling. Samples with higher weights are more likely to be used to generate new samples. Assuming N new samples need to be generated, the oversampling generation method can be:

[0054] x new =αx min +(1-α)x neighbor

[0055] Where, x min It is the selected minority class sample, x neighbor It is a neighbor of the sample. Usually, samples of the same type that are close to each other are selected. α∈[0,1] is a random number that controls the difference between the generated samples.

[0056] The above method can generate new minority class samples, thereby balancing the class ratio in the dataset.

[0057] Step 6: Perform nonlinear mapping and deep fusion on the high-dimensional features of the two processed modalities using DCCA to generate low-dimensional, highly correlated features.

[0058] Let the input feature matrices of the two modes be:

[0059] Speech modal feature matrix Image modal feature matrix Where m is the number of samples, d1 is the dimension of the original speech features, and d2 is the dimension of the original image features.

[0060] Two multilayer perceptron (MLP) networks are constructed with 512, 256, and 128 hidden layer units, respectively. The input feature matrices of the two modalities are initially nonlinearly mapped through their respective MLP subnetworks to obtain low-dimensional representations, as follows:

[0061] H1 = f θ (X)∈m×r

[0062] H2=g φ (X)∈ m×r

[0063] H1 is the low-dimensional feature representation of the speech modality, H2 is the low-dimensional feature representation of the image modality, and r is the mapped dimension.

[0064] For the low-dimensional feature matrices H1 and H2, the features of each modality are processed in a decentralized manner:

[0065]

[0066] Among them, 1 m It is a column vector consisting entirely of 1s.

[0067] Calculate the covariance matrix of the two modes, including:

[0068] Cross-modal covariance matrix:

[0069]

[0070] The single-modal covariance matrix, with regularization terms r1 and r2 added:

[0071]

[0072] Where I is the identity matrix, and r1 and r2 are small regularization parameters used to enhance numerical stability. The inverse square root of the covariance matrix is ​​calculated through eigenvalue decomposition:

[0073]

[0074] Similarly,

[0075]

[0076] Construct the standardized cross-modal correlation matrix using the inverse square root matrix:

[0077]

[0078] Perform singular value decomposition (SVD) on T to obtain the singular values ​​σ. i :

[0079]

[0080] σ i It is a singular value of T, representing the correlation between two modes in the common space.

[0081] If we use all singular values, that is:

[0082]

[0083] If only k singular values ​​are used, that is:

[0084]

[0085] Simultaneously, it is used as a loss function, and during the optimization process, the correlation is maximized by taking its negative value:

[0086] Loss = -corr

[0087] Next, backpropagation is performed, and the gradient of the network is calculated using automatic differentiation based on the loss. The parameters of the two MLP networks are updated through the optimizer Adam, and this process is iterated continuously. The optimization objective is to maximize the correlation between the features of the two modalities, i.e.:

[0088]

[0089] Finally, the output is the feature representation of the fused two modalities after network processing:

[0090]

[0091] in, This indicates a feature fusion operation.

[0092] Step 7: Input the low-dimensional features processed by DCCA into multiple classic classifiers for modeling and analysis.

[0093] Low-dimensional features processed by DCCA are input into various classic classifiers for modeling and analysis, including decision trees, K-nearest neighbors (KNN), support vector machines (SVM), Bayesian classifiers, and logistic regression. During training, five-fold cross-validation is used to train and evaluate each classifier. By dividing the training and validation sets, the robustness and generalization ability of the model evaluation results are ensured. The final output evaluation metrics include accuracy, precision, F1 score, and AUC.

[0094] Under otherwise identical conditions, this invention compared the classification performance of single-modal and multi-modal fusion methods, and further evaluated the performance of simple feature concatenation and DCCA-based feature fusion methods. Support Vector Machines (SVMs) were used as classifiers in the experiments, and the results of the binary classification experiments are shown in Tables 1 and 2, respectively.

[0095] Table 1. Binary classification results for OSA without / mild (AHI < 15 times / hour) and moderate to severe (15 times / hour ≤ AHI)

[0096] accuracy Accuracy F1 value AUC value speech features 0.7879 0.8231 0.7812 0.8744 craniofacial image features 0.7970 0.8138 0.7949 0.8095 splicing features 0.8409 0.8713 0.8372 0.9639 DCCA fusion features 0.8788 0.9022 0.8767 0.9662

[0097] Table 2. Binary classification results for OSA-free / mild to moderate (AHI < 30 bpm) and severe (30 bpm ≤ AHI) cases.

[0098] accuracy Accuracy F1 value AUC value speech features 0.8182 0.8671 0.8125 0.8744 craniofacial image features 0.8106 0.8632 0.8042 0.8641 splicing features 0.8561 0.8580 0.8558 0.9435 DCCA fusion features 0.9091 0.9229 0.9082 0.9763

Claims

1. A multimodal fusion obstructive sleep apnea identification method based on DCCA, characterized by: Includes the following steps: S1. Collect the subject's AHI index, speech data, and craniofacial image data as the basis for the classification task; S2. Perform silent segment removal and noise reduction on the speech data, and perform brightness enhancement, sharpening, and facial key point calibration on the craniofacial image data to complete data preprocessing; S3. Extract high-dimensional features from speech data and craniofacial image data using specific algorithms, including speech features based on DeepSpectrum and geometric morphological image features based on MediaPipe; S4. Use the Competitive Group Optimizer (CSO) and ReliefF algorithm to perform feature filtering on audio features and image features respectively; S5. Use an oversampling method based on adaptive Mahalanobis distance (AMDO) to balance the data for minority class samples; S6. The Deep Canonical Correlation Analysis (DCCA) method is used to fuse the features of speech and image data to generate low-dimensional, highly correlated fused features; S7. Input the fused features into multiple classifiers for modeling and analysis, evaluate the classification performance through five-fold cross-validation, and finally output the evaluation index of the model.

2. The method according to claim 1, wherein step S1 is characterized in that: The AHI index was collected using a polysomnography (PSG) device and two grading standards were used (AHI<15 and AHI≥15; AHI<30 and AHI≥30); voice data was recorded using a high-fidelity recorder with a sampling rate of 48kHz, including 5 vowels, 24 isolated syllables and a one-minute long sentence; craniofacial image data covered the front and left and right 90° angles and was captured by a Hikvision E14a camera.

3. The method according to claim 1, wherein step S3 is characterized in that: Speech features are based on Mel spectrograms and 1024-dimensional features are extracted using a DenseNet-121 pre-trained network. Image features are derived from facial key points extracted by MediaPipe, and distance, angle, and area are calculated to generate 724-dimensional geometric features.

4. The method according to claim 1, wherein step S4 is characterized in that: Feature selection for speech and image features was performed using the Competitive Group Optimizer (CSO) and the ReliefF algorithm, respectively.

5. The method according to claim 1, wherein step S5 is characterized in that: Oversampling employs an adaptive Mahalanobis distance (AMDO) method.

6. The method according to claim 1, wherein step S6 is characterized in that: DCCA uses a multilayer perceptron (MLP) to achieve nonlinear mapping of features, maximizes the canonical correlation between low-dimensional features, and optimizes it through a negative correlation loss function.

7. The method according to claim 1, wherein step S7 is characterized in that: The fusion features were modeled and analyzed using Support Vector Machine (SVM), Decision Tree, K-Nearest Neighbors (KNN), Logistic Regression, and Bayesian models. Five-fold cross-validation was used to evaluate the classification performance. Evaluation metrics included accuracy, precision, F1 score, and AUC score. Among these, the fusion features based on deep canonical correlation analysis outperformed the simple feature concatenation method on the classifier.

Images