Deep learning OSA severity classification model based on vowels

Abstract

The invention discloses a deep learning OSA severity classification model based on vowels. The method comprises the following steps: 1, collecting vowel speech signals of a subject, wherein the vowel speech signals comprise five Chinese vowels: / a / , / o / , / e / , / i / and / u / ; step 2, obtaining an AHI index obtained after the subject is subjected to whole-night PSG, and grading the severity of the subject; and step 3, extracting four kinds of paranguage features from the voice signal, wherein the four kinds of paranguage features comprise eGemaps, emobase, ComParE16 and MRCG. Fusing the four sub-language features of the five vowels into a vowel feature graph, and providing a feature set with the best detection efficiency for subsequent training; and step 4, combining the obtained vowel feature map with baseline features (including age, BMI, height, weight, neck circumference and abdominal circumference), inputting the obtained vowel feature map into a constructed convolutional neural network (CNN) for training and testing, and obtaining classification results under different AHI severity degrees.

Description

Technical Field

[0001] The invention relates to the technical field of obstructive sleep apnea syndrome diagnosis and screening, and in particular to a speech-based OSA severity classification method. Background Art

[0002] Currently, the gold standard for diagnosing OSA is polysomnography (PSG), which analyzes sleep physiological data to calculate electroencephalogram, electrooculogram, respiratory flow, chest and abdominal movement, electrocardiogram and blood oxygen level, etc., to identify apnea events and calculate the apnea index (AHI), which is used to measure the severity of OSA. However, PSG is not acceptable to all patients, because many people feel uncomfortable after the sensor is installed, which affects the quality of sleep and causes the test to fail. In addition, PSG requires a polysomnography technician (RPSGT) to collect and process data, which is costly. Therefore, finding new detection methods has become a research focus.

[0003] Since the early 1980s, researchers have been working to screen for OSA by analyzing the voices of patients when they are awake. The main computer analysis techniques rely on Fourier transform features such as frequency patterns, bandwidth, and resonance peaks, but these tools are mainly for normal speech and are not accurate for analyzing pathological speech signals of OSA. OSA sound signals differ greatly from normal speech in range, frequency variation, and resonance peak trajectory, so traditional speech processing / recognition methods are not applicable.

[0004] Studies have shown that OSA is associated with abnormal anatomical shape and function of the upper airway. After long-term obstructive sleep apnea, the patient's upper airway tissue will change, and this change will be reflected in the sound. Therefore, it is feasible to predict the severity of OSA through sound detection. Recently, many researchers have begun to use voice to predict the severity of OSA and have verified its feasibility.

[0005] For example, the invention with application number CN202010981626.0 discloses a classification method for the severity of obstructive sleep apnea, which extracts features from the original obstructive sleep apnea speech file, obtains a feature matrix including HFE, and constructs a naive Bayes classification model for classification. The information contained in these features is relatively simple, which does not meet the complex and changeable characteristics of pathological speech, and has low accuracy.

[0006] The invention with application number CN202310031801.3 discloses a method for constructing a daytime speech OSA severity discrimination and recognition model, which collects the subject's speech signal, extracts multi-dimensional speech features (features include energy features, time domain features, frequency domain features and music theory features), constructs several machine learning models as base classifiers, evaluates the models, and finally integrates the models. The subject's speech signal includes vowels and designed words and sentences. It takes a certain amount of time and manpower to establish a data set, and the extraction and processing of features and the construction of the model are also relatively complicated.

[0007] In summary, we consider collecting fewer speech signals to extract more advanced and informative features, and using deep learning models to improve accuracy while reducing costs. Summary of the invention

[0008] OSA has a high incidence rate, low diagnosis rate, and high risk factors. The gold standard PSG for diagnosing OSA has high monetary and time costs. The previous speech discrimination methods have the problems of extracted features not being suitable for pathological speech, low classification accuracy, and large amount of collected data. The present invention aims to provide a method for constructing a vowel-based OSA severity discrimination model. The present invention uses only 5 vowel speech signals for each subject, extracts advanced paralinguistic features for feature fusion, and combines baseline features to identify obstructive sleep apnea syndrome and discriminate and grade severity through deep learning methods, which can provide a non-invasive, faster, and lower-cost diagnostic tool for obstructive sleep apnea syndrome.

[0009] The method for constructing a deep learning OSA severity discrimination model based on vowel speech includes the following steps:

[0010] Step 1: Collect the subject's vowel speech signals, including five Chinese vowels: / a / , / o / , / e / , / i / , / u / .

[0011] Step 2: Obtain the AHI index obtained after the subjects undergo a full-night PSG and grade the severity of the subjects.

[0012] Step 3: Extract four paralinguistic features from the speech signal, including eGemaps, emobase, ComparE16, and MRCG. The four paralinguistic features of the five vowels are merged into a vowel feature map to provide the most effective feature set for subsequent training.

[0013] Step 4: The obtained vowel feature map was combined with baseline characteristics (including age, BMI, height, weight, neck circumference and abdominal circumference), and input into the constructed convolutional neural network (CNN) for training and testing to obtain the classification results under different AHI severities, that is, three binary classification tasks were performed: a. No OSA, AHI <5 times / hour; OSA, AHI ≥5 times / hour; b. No or mild OSA, AHI <15 times / hour; moderate to severe OSA, AHI ≥15 times / hour; c. No or mild to moderate OSA, AHI <30 times / hour; severe OSA, AHI ≥30 times / hour.

[0014] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: each subject only uses 5 vowel speech signals, extracts its paralinguistic features for feature fusion, and combines with baseline features to more comprehensively and effectively obtain the speech features of OSA patients, and uses a deep learning model to improve the classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a technical flow chart of the present invention;

[0016] Figure 2 is a feature fusion flow chart of the present invention;

[0017] Figure 3 It is a deep learning network classification flow chart of the present invention.

[0018] Figure 4 This is the confusion matrix diagram of the test result of AHI threshold = 5 of the present invention

[0019] Figure 5 This is the confusion matrix diagram of the test result of AHI threshold = 15 of the present invention

[0020] Figure 6 This is the confusion matrix diagram of the test result of AHI threshold=30 of the present invention

[0021] Figure 7 This is the ROC curve of the test result of AHI threshold=5 of the present invention.

[0022] Figure 8 This is the ROC curve of the test result of AHI threshold = 15 of the present invention

[0023] Fig. 9 This is the ROC curve of the test result of AHI threshold=30 of the present invention. DETAILED DESCRIPTION

[0024] The flowchart of the method for constructing a deep learning OSA severity discrimination model based on vowel speech is as follows Figure 1 As shown, the specific implementation method is as follows:

[0025] Step 1: Collect the subject's vowel speech signals, including five Chinese vowels: / a / , / o / , / e / , / i / , / u / .

[0026] Step 2: Obtain the AHI index obtained after the subjects underwent overnight PSG and grade the severity of the subjects. The statistical table of subject information is shown in Table 1.

[0027] Table 1 Statistics of subjects' information

[0028]

[0029] Step 3: Extract four paralinguistic features from the speech signal, including eGemaps, emobase, ComparE16, and MRCG. The four paralinguistic features of the five vowels are merged into a vowel feature map to provide the most effective feature set for subsequent training.

[0030] Among them, eGemaps, emobase, and ComParE16 features are extracted by OpenSMILE. OpenSMILE is an open source sound feature extraction tool, mainly used for sound and speech processing tasks, and is widely used in feature extraction, analysis, and processing of sound signals. Among them, the emobase acoustic feature set contains mel-frequency cepstral coefficients (MFCC), voice quality, fundamental frequency (F0), F0 envelope, LSP, and intensity features and their first-order and second-order derivatives. Several statistical functions are applied to these features, with a total of 988 features. The ComParE 2016 feature set includes energy, spectrum, MFCC, and speech-related low-level descriptors (LLDs). 11d includes log harmonic noise ratio, voice quality features, Viterbi smoothing of F0, spectral harmonics, and psychoacoustic spectral clarity, and calculates statistical functions, with a total of 6,373 features. The eGeMAPS feature set originated from an attempt to reduce the above feature set to a set of basic acoustic features, which contains FO semitone, loudness, spectral flux, MFCC, jitter, flicker, F1, F2, F3, alpha ratio, Hammarberg index and slope VO features, as well as their most common statistical functions, for a total of 88 features. MRCG uses a time-frequency representation to encode the multi-resolution power distribution of the audio signal. Four cochleogram features are generated at different resolutions. The high-resolution level encodes local information, while the remaining three lower-resolution levels capture spectro-temporal information. A total of 768 features are extracted per frame, including 256 MRCG features (frame length 20ms, frame shift 10ms), 256 ΔMRCG features, and 256 ΔΔMRCG features. After applying statistical functions (mean, standard deviation, minimum, maximum, range, mode, median, skewness and kurtosis), there are a total of 6,912 features.

[0031] The flow chart of feature fusion is as follows: Figure 2 As shown in the figure, the four input paralinguistic features are standardized respectively, and then encoded and decoded using nn.Sequential. The encoded four paralinguistic features are uniformly reshaped to the same size of 55*224, and then concatenated into a 220*224 matrix. Finally, two rows are filled in the top and bottom to become a 224*224 feature image, which is then copied into a 3*224*224 three-channel image, and the sum of the reconstruction losses is calculated.

[0032] Taking the 6912-dimensional MRCG feature as an example, the 6912-dimensional feature is first converted to 2048-dimensional through two fully connected layers, and the final output is 2464-dimensional, and finally reshaped to 11*224 through the tensor.

[0033] The principle of converting 6912 features to 2048 features:

[0034] 1. Weight Matrix and Bias

[0035] In the fully connected layer, we have two main parameters:

[0036] Weight matrix W: Its shape is 2048*6912, where 2048 is the dimension of the output feature and 6912 is the dimension of the input feature.

[0037] Bias vector b: Its shape is 1*2048 and is used to adjust each feature of the output.

[0038] 2. Input features

[0039] Assume that the input feature vector is x, and its shape is 1*6912, that is, a vector containing 6912 features.

[0040] 3. Matrix multiplication

[0041] The input feature xx is matrix multiplied by the weight matrix W. The formula for matrix multiplication is:

[0042] y=W·x+b

[0043] in:

[0044] W is a 2048*6912 weight matrix.

[0045] x is the input feature vector of 1*6912.

[0046] b is a 1*2048 bias vector.

[0047] 4. Calculation process

[0048] The specific calculation process is as follows:

[0049] a. Matrix multiplication multiplies the weight matrix W by the input feature x. This is a multiplication of a 2048*6912 matrix and a 1*6912 vector, and the result is a 1*2048 vector.

[0050] b. Add bias: Add the result to the bias vector b to get the final output feature vector y, whose shape is 1*2048.

[0051] This process is done automatically in PyTorch through the nn.Linear layer.

[0052] Then, the same principle is used to convert 2048 dimensions into 2464 dimensions. Finally, through tensor reshaping, the 2464 elements are rearranged into 11*224, and the size of the five vowels concatenated is 55*224.

[0053] The other three paralinguistic features are processed in the same structure, and the last four 55*224 feature matrices are concatenated to become 220*224. Finally, two rows are filled in the top and bottom to form a 224*224 vowel feature map.

[0054] Step 4: The obtained vowel feature map was combined with baseline characteristics (including age, BMI, height, weight, neck circumference and abdominal circumference), and input into the constructed convolutional neural network (CNN) for training and testing to obtain the classification results under different AHI severities, that is, three binary classification tasks were performed: a. No OSA, AHI <5 times / hour; OSA, AHI ≥5 times / hour; b. No or mild OSA, AHI <15 times / hour; moderate to severe OSA, AHI ≥15 times / hour; c. No or mild to moderate OSA, AHI <30 times / hour; severe OSA, AHI ≥30 times / hour.

[0055] The convolutional neural network (CNN) constructed is as follows Figure 3 As shown. A series of convolutional layers, activation functions, and pooling layers are defined using nn.Sequential:

[0056] The first convolutional layer has 3 input channels, 32 output channels, a kernel size of 3x3, a stride of 1, and a padding of 1. ReLU activation function is used to increase nonlinearity. The maximum pooling layer has a pooling window size of 2x2 and a stride of 2.

[0057] The second convolutional layer has 32 input channels and 64 output channels. ReLU activation function. Max pooling layer.

[0058] The third convolutional layer has 64 input channels and 128 output channels. ReLU activation function. Adaptive average pooling layer resizes the feature map to 1x1. It outputs the average value of each feature map.

[0059] Then the six-dimensional baseline features (age, BMI, height, weight, neck circumference and abdominal circumference) were first standardized (subtracting the mean and dividing by the standard deviation) and mapped to 128 dimensions.

[0060] Finally, a fully connected layer is defined to concatenate the features from the convolutional layer (128 dimensions) and the baseline features (128 dimensions) together, for a total of 256 dimensions, and then output 2 dimensions for classification tasks.

[0061] The training set and test set are divided according to three binary classification tasks (a. No OSA, AHI < 5 times / hour; with OSA, AHI ≥ 5 times / hour; b. No or mild OSA, AHI < 15 times / hour; moderate to severe OSA, AHI ≥ 15 times / hour; c. No or mild to moderate OSA, AHI < 30 times / hour; severe OSA, AHI ≥ 30 times / hour). The dataset is divided into small batches, the data is shuffled at each epoch, and the parameters are optimized using the Adam algorithm. During training, the training loss is calculated for each epoch and the gradient is updated. An evaluation test is performed every 10 epochs, the accuracy is calculated, and the best accuracy is updated. The total loss (loss) is the weighted sum of the classification loss (cle) and the reconstruction loss (recon_loss):

[0062] loss=cle_loss+0.5*recon_loss

[0063] The method we used only used 5 vowel speech signals for each subject, extracted its paralinguistic features for feature fusion, and combined with baseline features to obtain the speech features of OSA patients more comprehensively and effectively, and used a deep learning model to improve the classification accuracy. The results of the three binary classification tests are shown in Table 2. The confusion matrices of the three results are shown in Table 2. Figure 4-6 As shown, the ROC curve is as follows Figure 7-9 shown.

[0064] Table 2 Binary classification test results

[0065]

[0066]

Claims

1. A method for constructing a deep learning OSA severity discrimination model based on vowel speech, characterized in that: The specific implementation methods are as follows: Step 1: Collect the subject's vowel speech signals, including five Chinese vowels: / a / , / o / , / e / , / i / , / u / . Step 2: Obtain the AHI index obtained after the subjects undergo a full-night PSG and grade the severity of the subjects. Step 3: Extract four paralinguistic features from the speech signal, including eGemaps, emobase, ComparE16, and MRCG. The four paralinguistic features of the five vowels are merged into a vowel feature map to provide the most effective feature set for subsequent training. Step 4: The obtained vowel feature map was combined with baseline characteristics (including age, BMI, height, weight, neck circumference and abdominal circumference), and input into the constructed convolutional neural network (CNN) for training and testing to obtain the classification results under different AHI severities, that is, three binary classification tasks were performed: a. No OSA, AHI <5 times / hour; OSA, AHI ≥5 times / hour; b. No or mild OSA, AHI <15 times / hour; moderate to severe OSA, AHI ≥15 times / hour; c. No or mild to moderate OSA, AHI <30 times / hour; severe OSA, AHI ≥30 times / hour.

2. The construction method according to claim 1, characterized in that: In step 1, only five vowel speech signals of the subject are extracted.

3. The construction method according to claim 1, characterized in that: In step 3, paralinguistic features of speech signals are extracted, including eGemaps, emobase, ComparE16, and MRCG.

4. The construction method according to claim 1, characterized in that: In step 3, the feature fusion method is used to standardize the input features respectively, and then encode and decode them. After encoding, the dimensions of all features are the same. The encoded features are reshaped into a matrix with a width of 224 and spliced ​​into a feature matrix. Finally, the top and bottom are padded to become a 224*224 feature image, and then copied into a 3*224*224 three-channel image input.

5. The construction method according to claim 1, characterized in that: Step 4 uses deep learning methods to build a convolutional neural network (CNN) for training and testing.

6. The construction method according to claim 1, characterized in that: In step 4, baseline features are introduced, including the subject's physiological information (age, BMI, height, weight, neck circumference and abdominal circumference), and the baseline features are combined with the vowel feature map as feature input.

7. The construction method according to claim 1, characterized in that: Step 4: Perform three binary classification tasks (a. no OSA, AHI < 5 times / hour; with OSA, AHI ≥ 5 times / hour; b. no or mild OSA, AHI < 15 times / hour; moderate to severe OSA, AHI ≥ 15 times / hour; c. no or mild to moderate OSA, AHI < 30 times / hour; severe OSA, AHI ≥ 30 times / hour), and divide the training set and test set into testing and training sets respectively.

Images