Ai-based hearing loss diagnosis method using voice data

Abstract

The present invention enables early, simple, and accurate diagnosis of hearing loss through a method for diagnosing hearing loss by means of an artificial intelligence (AI) algorithm using only the voice data of a subject. The present invention relates to an AI-based hearing loss diagnosis method using voice data that can accurately and effectively diagnose hearing loss regardless of the subject's age or gender by using voice data obtained by reading negative prose (a long scenario-type text having a length of about 1 minute) and voice data obtained by reading positive prose, especially voice data obtained by reading negative prose with expression of intense emotions such as frustration and agitation, and voice data obtained by reading positive prose without expressing emotions, and also reduces overfitting due to the number of data by utilizing data obtained by augmenting original data with respect to voice data that is difficult to sufficiently secure due to a collection process or security issues, while utilizing the effectively acquired voice data to accurately diagnose hearing loss.

Description

AI-based hearing loss diagnosis method using voice data

[0001] The present invention relates to a method for diagnosing hearing loss using only the voice data of a subject through an artificial intelligence (AI) algorithm, thereby enabling an early, simple, and accurate diagnosis of hearing loss. The present invention relates to an AI-based hearing loss diagnosis method using voice data, which utilizes voice data obtained by reading negative prose (a long scenario-like text of about 1 minute in length) and voice data obtained by reading positive prose, and in particular, voice data obtained by reading negative prose with all the feelings of injustice and intensity, and voice data obtained by reading positive prose without feelings, thereby enabling an accurate and effective diagnosis of hearing loss regardless of the subject's age, gender, etc. In addition, for voice data that is difficult to secure sufficiently due to issues such as the collection process or security, the present invention utilizes data obtained by augmenting the original data together with data obtained by effectively acquiring the original data, thereby reducing overfitting due to the amount of data, and enabling an accurate diagnosis of hearing loss by utilizing the effectively acquired voice data. In addition, the present invention relates to a method for diagnosing hearing loss based on voice data, which enables an early, simple, and accurate diagnosis of hearing loss by specifying voice features and a diagnostic algorithm that are more effective in the early diagnosis of hearing loss in the acquired voice data.

[0002] Humans possess five major senses: sight, hearing, smell, taste, and touch. Of these, hearing detects sound waves, or waves that escape the reach of sight, allowing us to constantly acquire information.

[0003] Hearing loss is the most common sensorineural disorder, with a prevalence rate of adult-onset hearing loss being significantly higher than that of childhood hearing loss, doubling every decade. Adult hearing loss not only represents a significant burden on the adult population worldwide, but is also associated with higher rates of hospitalization, depression, and dementia. Hearing impairment increases the risk of depression, cognitive decline, and dementia. Therefore, early diagnosis and treatment of hearing loss, or hearing loss, is crucial as it can help prevent or manage various potential health conditions. Furthermore, the prevalence of hearing loss continues to increase globally due to the aging population. Consequently, there is a growing need for simple, efficient, and accurate methods for early diagnosis of hearing loss, particularly through simple tests.

[0004] <Patent Document>

[0005] Domestic Patent No. 10-2161638 (announced on October 6, 2020) "Method, system, and computer-readable medium for determining the degree of dementia based on speech recognition technology using a machine learning model."

[0006] In the case of the prior art patent disclosed in the above <patent document>, it relates to a method for determining the degree of dementia based on speech recognition technology using a machine learning model, and more specifically, it relates to a method for determining the degree of dementia based on speech recognition technology using a machine learning model that provides content related to language ability diagnosis, recognizes the user's voice for the content, and analyzes comprehensive language abilities such as memorization, expression, inference, and interpretation, thereby providing determination information on the degree of dementia of the user. In this way, although there is AI technology for diagnosing and measuring cognitive impairment such as dementia based on voice in the existing prior art related technologies, there is no technology that can diagnose hearing loss based on voice data and ensure its accuracy. Therefore, the need for more accurate hearing loss diagnosis technology based on voice is increasing.

[0007] The present invention has been devised to solve the above problems.

[0008] The purpose of the present invention is to provide an AI-based hearing loss diagnosis method using voice data that is effective in diagnosing and treating hearing loss diseases by diagnosing hearing loss through an artificial intelligence (AI) algorithm using only the voice data of a subject, thereby enabling early, simple, and accurate diagnosis of hearing loss.

[0009] Another object of the present invention is to provide an AI-based hearing loss diagnosis method that can accurately and effectively diagnose hearing loss regardless of the age or gender of the subject by utilizing voice data that is used to diagnose hearing loss disease, such as voice data that reads negative prose (a long scenario-like text of about 1 minute in length) and voice data that reads positive prose, especially voice data that reads negative prose with all the feelings of injustice and intensity, and voice data that reads positive prose with no feelings.

[0010] Another object of the present invention is to provide an AI-based hearing loss diagnosis method using voice data that can accurately diagnose the presence or absence of hearing loss by effectively utilizing voice data acquired while reducing overfitting according to the number of data by utilizing data augmented with original data for voice data that is difficult to sufficiently secure due to issues such as the collection process or security.

[0011] Another object of the present invention is to provide an AI-based hearing loss diagnosis method using voice data that can diagnose hearing loss early, easily, and accurately more effectively by specifying voice features and a diagnostic algorithm that are more effective in early diagnosis of hearing loss from acquired voice data.

[0012] In order to achieve the above-mentioned purpose, the present invention is implemented by an embodiment having the following configuration.

[0013] According to one embodiment of the present invention, the AI-based hearing loss diagnosis method using voice data according to the present invention is characterized by including the steps of: obtaining voice data (S1); extracting voice features from the obtained voice data (S3); and receiving the extracted voice features and diagnosing hearing loss using an artificial intelligence (AI) algorithm (S5).

[0014] According to another embodiment of the present invention, the voice data acquired in the step S1 of the present invention is characterized in that it includes voice data acquired through negative long sentence reading and voice data acquired through positive long sentence reading.

[0015] According to another embodiment of the present invention, the voice data acquired in the S1 step of the present invention is characterized in that it is voice data acquired in a state loaded with emotion when reading a negative long sentence, and it is voice data acquired in a state excluding emotion when reading a positive long sentence.

[0016] According to another embodiment of the present invention, the audio data obtained in the S1 step of the present invention is characterized by including original audio data and reversed audio data obtained by enhancing the original audio data by playing it backwards using a time inversion method.

[0017] According to another embodiment of the present invention, the voice features extracted in the S3 step of the present invention are characterized in that they include at least one of eGeMAPS (extended Geneva Acoustic Parameter Set), mel-spectrogram, and wav2vec.

[0018] According to another embodiment of the present invention, the artificial intelligence (AI) algorithm used in the S5 step of the present invention is characterized by including any one of SVM (Support Vector Machine), MLP-LSTM (Multilayer Perceptron-long short-term memory), VGGish, and Swin Transformer.

[0019] According to another embodiment of the present invention, the step S5 in the present invention is characterized by receiving wav2vec speech features and diagnosing hearing loss using Swin Transformer.

[0020] According to another embodiment of the present invention, the wav2vec speech features are characterized in that they are extracted from both original speech data including speech data obtained through reading negative long sentences in an emotional state and speech data obtained through reading positive long sentences in an emotional state excluded, and reversed speech data obtained by enhancing the original speech data by playing it backwards using a time inversion method.

[0021] The present invention can obtain the following effects through the combination and use of the configuration described above and the following examples.

[0022] The present invention has an effective effect in diagnosing and treating hearing loss diseases by enabling early, simple, and accurate diagnosis of hearing loss through a method of diagnosing hearing loss using only the subject's voice data through an artificial intelligence (AI) algorithm.

[0023] The present invention utilizes voice data used for diagnosing hearing loss disease, including voice data obtained by reading negative prose (a long scenario-like text of about 1 minute in length) and voice data obtained by reading positive prose, particularly voice data obtained by reading negative prose with both aggrieved and intense emotions, and voice data obtained by reading positive prose with no emotions, thereby enabling accurate and effective diagnosis of hearing loss regardless of the subject's age or gender.

[0024] The present invention has the effect of accurately diagnosing the presence or absence of hearing loss by utilizing voice data effectively acquired while reducing overfitting according to the number of data by utilizing data augmented from original data together with voice data that is difficult to sufficiently secure due to issues such as the collection process or security.

[0025] The present invention has the effect of enabling early, simple, and accurate diagnosis of hearing loss by specifying voice features and a diagnostic algorithm that are more effective in early diagnosis of hearing loss from acquired voice data.

[0026] Figure 1 is a table and graph showing the average accuracy of 5-Fold Cross-Validation for each model before data augmentation.

[0027] Figure 2 is a table showing the average accuracy of 5-Fold Cross-Validation by model after data augmentation.

[0028] Figure 3 is a graph showing the average accuracy of 5-fold cross-validation by model after data augmentation.

[0029] Hereinafter, preferred embodiments of an AI-based hearing loss diagnosis method using voice data according to the present invention will be described in detail with reference to the attached drawings. It should be noted that identical components are represented by the same reference numerals throughout the drawings, as much as possible. Unless otherwise defined, all terms in this specification have the same general meaning as those terms understood by a person skilled in the art to which the present invention pertains, and if there is a conflict with the meaning of a term used in this specification, the definitions used in this specification shall apply. Throughout the specification, when a part is said to "include" a certain component, this does not mean that other components are excluded, but rather that other components may be further included, unless specifically stated otherwise. In addition, terms such as "... part" and "... module" described in the specification mean a unit that processes at least one function or operation, and this may be implemented by hardware, software, or a combination of hardware and software.

[0030]

[0031] Referring to FIGS. 1 and 2, an AI-based hearing loss diagnosis method using voice data according to one embodiment of the present invention may include a step of acquiring voice data (S1); a step of extracting voice features from the acquired voice data (S3); and a step of receiving the extracted voice features and diagnosing hearing loss using an artificial intelligence (AI) algorithm (S5).

[0032] The present invention relates to a method for diagnosing hearing loss by obtaining voice data from a subject for diagnosis, extracting specific voice features from the obtained voice data, inputting the extracted voice features into an artificial intelligence (AI) algorithm, and analyzing the same. The process of obtaining voice data from a subject for diagnosis includes a process in which the subject reads provided prose in an environment where external noise, etc. is blocked through a configuration of a voice input unit (microphone, etc.), thereby obtaining the subject's voice data. The process of extracting voice features and analyzing the algorithm includes extracting specific voice features from the voice data obtained as described above through a configuration of a server unit (where an extraction and analysis algorithm (program) is run), and then inputting the extracted voice features into an artificial intelligence (AI) algorithm, thereby analyzing the same, thereby diagnosing hearing loss.

[0033] The above step S1 is a process of acquiring voice data from the subject of diagnosis, and the voice data acquired in the S1 step may include voice data acquired by reading negative long sentences (a text of about 1 minute in the form of a long scenario) and voice data acquired by reading positive long sentences. In particular, when reading negative long sentences, the voice data is acquired by reading the sentences with emotion, that is, by putting all of the feelings of injustice and intensity into them, and when reading positive long sentences, the voice data is acquired by reading them as if excluding emotions. By based on the voice data acquired in this manner, it is possible to acquire voice data in a wider variety of states from the subject, and thus it is possible to accurately and effectively diagnose the presence or absence of hearing loss regardless of the subject's age, gender, etc.

[0034] In particular, the present invention is characterized in that it utilizes not only the original audio data acquired through the S1 step, but also the original audio data and the reversed audio data acquired by playing the original audio data in reverse using the time inversion method to enhance the audio data, thereby enabling the diagnosis of hearing loss to be accurately made by utilizing the audio data acquired effectively while reducing overfitting according to the number of data by utilizing the data acquired by enhancing the original audio data together with audio data that is difficult to sufficiently secure due to issues such as the audio data collection process or security.

[0035] The above S3 step is a process of extracting voice features from acquired voice data, and the voice features extracted in the S3 step may include one or more of eGeMAPS (extended Geneva Acoustic Parameter Set), mel-spectrogram, and wav2vec.

[0036] eGeMAPS (extended Geneva Acoustic Parameter Set) is a speech feature set developed as an extension of GeMAPS (Geneva Minimalistic Acoustic Parameter Set) and includes features related to emotion and gender. In addition, eGeMAPS includes 88 LLDs (Low-level descriptors) that capture various acoustic characteristics of speech signals, such as spectrum, prosody, and voice quality features. These LLDs can be used for feature extraction (speech features) using openSMILE, an open-source software. In the present invention, all 88 eGeMAPS features were utilized to specify an optimized algorithm for distinguishing the presence or absence of hearing loss.

[0037] A Mel Spectrogram is a visual representation of the frequency content of an audio signal over time. To create a Mel spectrogram from an audio file, the audio signal is first divided into short, overlapping frames, typically every 10 ms, with a window size of 25 ms. Each frame is then transformed using a short-time Fourier transform (STFT). The resulting frames are then integrated into 64 Mel-space frequency bands, and a Mel-scaling process is performed, calculating a Mel-spectrogram across 96 frequency bands for each second of the input audio. Finally, a small offset is added to transform the Mel-scaled spectral coefficients to a log scale, generating a log Mel-spectrogram.

[0038] wav2vec is a self-supervised speech recognition model developed by Facebook AI Research. It uses a convolutional neural network (CNN) trained to predict future speech frames from previous ones as an encoder, and can extract high-quality speech features by detecting useful contextual information from the input signal. wav2vec, which includes informative and abstract features of the speech signal, outputs the sequence in the hidden state based on the vector in the model's last hidden layer, extracting a 512-dimensional feature vector. Since the length of each audio is different, the overall feature dimensionality also differs (512 * total length of the input audio).

[0039] The above step S5 is a process of diagnosing hearing loss by using an artificial intelligence (AI) algorithm by inputting extracted voice features. The artificial intelligence (AI) algorithm used in the above step S5 may include any one of SVM (Support Vector Machine), MLP-LSTM (Multilayer Perceptron-long short-term memory), VGGish, and Swin Transformer.

[0040] While both machine learning and deep learning are generally considered subfields of artificial intelligence (AI), they differ in how they process information. Machine learning algorithms typically extract features from input data and use those features to train a model that can predict new data. Deep learning algorithms, on the other hand, use multi-layered neural networks to learn features directly from input data. While deep learning algorithms can be more powerful and flexible than machine learning algorithms, they require significantly more data and computational resources.

[0041] Support Vector Machines (SVMs) are a type of machine learning algorithm that can be used for classification and regression tasks. SVMs work by finding a hyperplane that best distinguishes different classes within a data space. SVMs can handle high-dimensional feature spaces and are more effective when the number of features significantly exceeds the number of samples. SVMs can be used for both linear and nonlinear classification using various kernel functions. In particular, this patent uses eGemaps features to classify hearing loss using an SVM. The SVM model was trained with 100 epochs, 16 batches, and a learning rate of 0.001, and was imported and used in sklearn.

[0042] The MLP-LSTM model is a type of neural network architecture that combines two types of layers: the Multilayer Perceptron (MLP), a type of machine learning model, and the Long Short-Term Memory (LSTM), a type of deep learning model. The MLP layer performs nonlinear transformations on input data, and the LSTM layer is designed to detect temporal dependencies in data over time. In particular, this patent utilizes eGemaps features as input to the MLP-LSTM model to classify the presence or absence of hearing loss. The MLP layer transforms the input eGemaps features into high-dimensional representations, and the LSTM layer detects temporal dependencies in the speech signal. The output of the LSTM layer is then passed through a fully connected layer to produce the final classification result. In this patent, a binary classification of hearing loss is used as the output. The MLP-LSTM model has been proven effective for speech emotion recognition because it can utilize both static and dynamic characteristics of speech signals. The MLP used in this patent performed two-class classification using three linear layers and the ReLU activation function. The epoch was 100, the batch was 16, and the learning rate was 0.0001.

[0043] VGGish is a deep learning model that uses log Mel-Spectrogram, which contains both spectral and temporal characteristics of audio, as input. The model is based on the architecture of the VGG Network, widely used in image recognition, and consists of multiple convolutional and pooling layers, as well as fully connected layers. VGGish is designed to work on short audio segments, approximately 1 second long, and has the property of classifying audio into one of 128 classes (e.g., instrument, male, female, car, etc.). The VGGish model was implemented and used with epochs of 50 or 100, batches of 50, a learning rate of 0.0001, and a sequence length of 5. The LSTM attached to VGGish used a dropout value of 0.4 and a softmax activation function.

[0044] Swin Transformer is a state-of-the-art transformer-based deep learning architecture for image recognition proposed by Microsoft Research. In this patent, pre-trained features extracted from wav2vec, rather than images, were used as input to the Swin Transformer encoder. While the Swin Transformer architecture was originally designed for image recognition, the wav2vec input dimension was (512 * the total length of the input audio) and thus usable. This patent used 1000 epochs, 8 batches, a learning rate of 5e-4, a window size of 40, and a warmup step of 20, with AdamW as the optimizer.

[0045] In particular, the present invention presents an AI-based hearing loss diagnosis method using voice data that can diagnose hearing loss early, easily, and accurately by inputting wav2vec voice features and using Swin Transformer as voice features and diagnostic algorithms that are more effective in early diagnosis of hearing loss from acquired voice data, and this is proven through the following experiments.

[0046] [Experiment] Accuracy test for diagnosing hearing loss using subjects' voice data

[0047] - Experimental subjects: The above experimental subjects are 483 people, divided into 321 women and 162 men by gender as shown in [Table 1] below, and 118 normal subjects and 365 hearing-impaired patients by the presence or absence of hearing loss. 1) Patients with a normal eardrum and a subjective hearing loss period of 3 years or longer as determined by a professional hearing test by an otolaryngologist-head and neck surgeon, 2) Patients with a hearing loss average of 40 dB or higher at 500 Hz, 1000 Hz, 2000 Hz, and 4000 Hz based on their better hearing.

[0048] Total number of normal patients: females: 89,232,321; males: 29,133,162; total number of normal patients: 11,836,5483

[0049] - Voice data collection: For each experimental subject, two tasks were performed to collect voice data: reading a negative long scenario lasting about 1 minute (chapter 7) and reading a positive long newspaper editorial (chapter 8). In chapter 7, when reading a negative long sentence, the subjects were instructed to read it with all their feelings of injustice and intensity, and in chapter 8, when reading a newspaper editorial, they were instructed to read it as if they were reading a book without any emotions. - Data augmentation: The original audio data acquired in this experiment are voice data from a total of 483 subjects, including 118 normal subjects and 365 hearing-impaired patients. Therefore, in order to reduce overfitting to the number of data and enable the model to perform well in a real environment, data augmentation was performed by reversing the original audio data to extract features (voice features) (in particular, data augmentation was performed using the time inversion method). Through this process, by combining the original audio data and the reversed audio data for training, a total of 966 audio data, 236 normal and 730 patient data, can be used for model training and validation.

[0050] - Experimental environment: This experiment was conducted in Linux and Windows environments, and one NVIDIA GeForce RTX 2070 16GB and two A100 40GB were used to train the deep learning model in a GPU environment.

[0051] - Experimental method: From all acquired voice data (dataset), three features (voice features) were extracted: eGemaps, mel-spectrogram, and wav2vec, as described above. Then, they were applied to four models (artificial intelligence algorithms): SVM, MLP-LSTM, VGGish, and Swin, as described above, and classification was performed using 5-Fold Cross-Validation with two labels: normal (0) and patient (1). The Unseen method uses 80% of the entire dataset for training and 20% for validation, so that people who have never been trained are tested. The Cross-Subject method uses 80% of one person's voice data for training and 20% for validation, so that people who have been trained once are tested, but a different portion of the voice data is used from the one used for training.

[0052] - Experimental results

[0053] 1. First, classification was performed using 5-Fold Cross-Validation without data augmentation, that is, using only the original audio data. The audio features and models (artificial intelligence algorithms) used here were MLP-LSTM and SVM using eGemaps features, Swin Transformer using wav2vec features, and VGGish using unique VGGish features after converting audio to Mel-stectrogram. As a result, as shown in 'Figure 1', the performance of Swin and VGGish 50 epochs are similar, but when the number of epochs is increased to 100, the performance of VGGish is the highest at approximately 81.24% in Unseen and 84.44% in Cross Subject. On the other hand, MLP-LSTM and SVM using eGemaps features did not achieve an accuracy of more than 75% in all results. Meanwhile, in the case of the Unseen verification method, since verification is performed on data from subjects who have never been trained, it can be confirmed that it is about 2-3% lower than the Cross Subject verification method.

[0054] 2. Since the performance of the MLP-LSTM and SVM models using eGemaps features when data augmentation was not good, we additionally conducted experiments after data augmentation using only Swin and VGGish, which had good performance (using the original speech data and the reversed speech data to extract wav2vec and VGGish features, and training and validating the Swin and VGGish models with 5-Fold Cross-Validation). As a result, as shown in 'Fig. 2' and 'Fig. 3', in the Unseen method, Swin showed the best performance with a classification accuracy of approximately 84.11%, and in the Cross subject method, VGGish showed the best performance with a classification accuracy of approximately 85.54%.

[0055] That is, through this [experiment], it was confirmed that diagnosis of hearing loss is possible by utilizing voice data (chapter 7) obtained through negative long sentence reading and voice data (chapter 8) obtained through positive long sentence reading presented in the present invention, and in particular, when the two data are combined and utilized, the classification accuracy for hearing loss increases, and it can be confirmed that the classification accuracy is even higher when both normal original voice data and data-enhanced reversed voice data are used.

[0056] In addition, in the voice features and models for diagnosing hearing loss, in the case of the Unseen verification method, the accuracy of Swin is mostly high in the low to mid 80% range, and in the case of the Cross-subject verification method, there are folds that show an accuracy of over 90% in Swin or VGGish, and when averaged, most show an accuracy of the mid 80%, and it can be confirmed that the performance of Swin and VGGish is similar. However, in the subject-independent verification method, the unseen subject task is more difficult than the cross-subject method, and considering future general use, it can be confirmed that the method of diagnosing hearing loss using the Swin Transformer by inputting wav2vec voice features is the most efficient and suitable as a model for accurately classifying the presence or absence of hearing loss.

[0057]

[0058] In the above, the applicant has described various embodiments of the present invention, but such embodiments are only examples of implementing the technical idea of ​​the present invention, and any change or modification that implements the technical idea of ​​the present invention should be interpreted as falling within the scope of the present invention.

Claims

1. Step of acquiring voice data (S1); Step (S3) of extracting voice features from acquired voice data; An AI-based hearing loss diagnosis method using voice data, characterized by including a step (S5) of diagnosing hearing loss using an artificial intelligence (AI) algorithm by inputting extracted voice features.

2. In paragraph 1, An AI-based hearing loss diagnosis method using voice data, characterized in that the voice data acquired in the above step S1 includes voice data acquired through reading negative long sentences and voice data acquired through reading positive long sentences.

3. In paragraph 2, An AI-based hearing loss diagnosis method using voice data, characterized in that the voice data acquired in the above S1 step is voice data acquired in a state loaded with emotion when reading a negative long sentence, and voice data acquired in a state excluding emotion when reading a positive long sentence.

4. In paragraph 2, An AI-based hearing loss diagnosis method using voice data, characterized in that the voice data acquired in the above step S1 includes original voice data and reversed audio data obtained by enhancing the original voice data by playing it backwards using a time inversion method.

5. In paragraph 1, An AI-based hearing loss diagnosis method using voice data, characterized in that the voice features extracted in the above S3 step include at least one of eGeMAPS (extended Geneva Acoustic Parameter Set), mel-spectrogram, and wav2vec.

6. In paragraph 5, An AI-based hearing loss diagnosis method using voice data, characterized in that the artificial intelligence (AI) algorithm used in the above step S5 includes any one of SVM (Support Vector Machine), MLP-LSTM (Multilayer Perceptron-long short-term memory), VGGish, and Swin Transformer.

7. In paragraph 6, The above S5 step is an AI-based hearing loss diagnosis method using voice data, characterized in that it receives wav2vec voice features as input and diagnoses hearing loss using Swin Transformer.

8. In paragraph 7, An AI-based hearing loss diagnosis method using voice data, characterized in that the above wav2vec voice features are extracted from both original voice data including voice data obtained by reading negative long sentences in an emotional state and voice data obtained by reading positive long sentences in an emotional state without emotion, and reversed voice data obtained by enhancing the original voice data by playing it backwards using a time inversion method.

Images