Ai-based hearing loss, cognitive impairment, and alzheimer's disease diagnosing system using voice data

Abstract

The present invention relates to an AI-based hearing loss diagnosis method using voice data. The AI-based hearing loss diagnosis method can: accurately and effectively diagnose the presence or absence of hearing loss regardless of the age or gender of a subject by utilizing, as voice data used to diagnose the presence or absence of hearing loss, voice data obtained by reading negative prose (long scenario-type text of approximately one minute in length) and voice data obtained by reading positive prose, especially, voice data obtained by reading negative prose with both the feelings of being wronged and overwhelmed and voice data obtained by reading positive prose without feelings; accurately diagnose the presence or absence of hearing loss by utilizing effectively acquired voice data while reducing overfitting due to amount of data by utilizing data obtained by augmenting the original data of the voice data that is difficult to obtain due to problems in the collection process, security, or the like; and easily and accurately diagnose the presence or absence of hearing loss more effectively in the early stage by identifying voice features and a diagnostic algorithm that is more effective for the early diagnosis of the presence or absence of hearing loss in the acquired voice data.

Description

AI-based hearing loss, cognitive impairment, and dementia screening system using voice data

[0001] The present invention relates to a system that performs everything from checking a voice data collection environment to voice data collection, prediction / diagnosis, and result confirmation through a single system, and more specifically, enables prediction or diagnosis of various brain health conditions (hearing loss, cognitive impairment, dementia, etc.) using voice data as a biomarker, and in particular, relates to an AI-based hearing loss, cognitive impairment, and dementia screening system using voice data that performs everything from checking a collection environment and user status for acquiring voice data through a single system to monitoring voice volume during the voice data acquisition process and deriving prediction and diagnosis results for hearing loss, cognitive impairment, and dementia based on the acquired voice data.

[0002] When it comes to brain health, various diseases exist, including cognitive impairment, Alzheimer's disease (dementia), and depression. Cognitive impairment refers to a decline in memory, attention, language skills, visuospatial abilities, and judgment. The degree of cognitive impairment ranges from very mild to severe. Dementia occurs when cognitive impairment is so severe that it interferes with daily life or social activities. Alzheimer's Disease (AD) is a neurodegenerative disease characterized by a long-term and gradual decline in cognitive function. It is a common form of dementia that first progresses to mild cognitive impairment (MCI) and then to AD. Early detection of Alzheimer's disease is essential for timely treatment, which can slow cognitive decline and reduce the risk of developing dementia. Depression is a condition characterized by a variety of cognitive and psychosomatic symptoms, primarily loss of motivation and feelings of sadness, which lead to a decline in daily functioning. Depressive disorder is a serious illness that causes changes in emotions, thoughts, physical state, and behavior.

[0003] Voice has various characteristics, including pitch, volume, rate, rhythm, fluency, articulation, pronunciation, and tone. These features are also the components that allow us to perceive what others are saying. These sounds are generated through various circuits in the brain, passing through various locations, and are then heard by our ears. This is why voices differ in various diseases and conditions related to brain health. These differences can be used to diagnose various disorders. However, to develop such voice biomarkers, data consistency and validity are crucial. To ensure this, secure technology and well-controlled environment for data collection are crucial.

[0004] Patent Document

[0005] Domestic Patent No. 10-2274072 (registered on July 1, 2021) "Method and device for determining the degree of dementia in a user."

[0006] The prior patent disclosed in the above <patent document> also only discloses the concept of visualizing the characteristics of the voice input by the user and using this to determine the degree of dementia.

[0007] Therefore, there is an increasing need for technologies that can collect voice data that can ensure data consistency and validity by controlling and standardizing the voice data collection environment and collection method from the voice data collection stage, technologies that can diagnose hearing loss based on voice data and ensure its accuracy, technologies that can simultaneously perform MMSE score prediction to predict the degree of cognitive impairment and AD diagnosis to diagnose Alzheimer's disease using a single algorithm, and technologies that can quickly and accurately diagnose Alzheimer's disease (AD) by easily extracting features from the participant's voice recording (voice data) are being developed.

[0008] The present invention has been devised to solve the above problems.

[0009] The purpose of the present invention is to enable prediction or diagnosis of various brain health conditions (hearing loss, cognitive impairment, dementia, etc.) using voice data as a biomarker, and in particular, to provide an AI-based hearing loss, cognitive impairment, and dementia screening system using voice data, which performs everything from checking the collection environment and user status for acquiring voice data with a single system, to monitoring voice volume during the process of acquiring voice data, and deriving prediction and diagnosis results for hearing loss, cognitive impairment, and dementia based on the acquired voice data.

[0010] Another purpose of the present invention is to provide an AI-based hearing loss, cognitive impairment, and dementia screening system using voice data, which ensures consistency and validity of data and increases the accuracy of various brain health diagnoses by checking the surrounding environment for collecting voice data, i.e., surrounding noise conditions, soundproofing conditions, etc., and checking the current physical condition of the user who needs to collect voice data.

[0011] Another object of the present invention is to provide an AI-based hearing loss, cognitive impairment, and dementia screening system using voice data, which can accurately and effectively diagnose hearing loss regardless of the age or gender of the subject by using voice data that reads negative prose (a long scenario-like text of about 1 minute in length) and voice data that reads positive prose, especially voice data that reads negative prose with all the feelings of injustice and intensity, and voice data that reads positive prose without feelings, and which can accurately and effectively diagnose hearing loss regardless of the subject's age or gender, and which can reduce overfitting according to the number of data while utilizing effectively acquired voice data by using data that is augmented with original data in voice data that is difficult to sufficiently secure due to issues such as the collection process or security, thereby accurately diagnosing hearing loss.

[0012] Another object of the present invention is to provide an AI-based hearing loss, cognitive impairment, and dementia screening system using voice data, which can simultaneously perform MMSE score prediction and Alzheimer's diagnosis using a single algorithm by inputting acoustic features extracted from voice data, and at this time, when utilizing a loss function that reflects both regression loss and classification loss in the algorithm to increase the accuracy of prediction and diagnosis, the regression loss and the classification loss can work complementarily, and the mean square error (MSE) function reflects a Bernoulli penalty according to the probability value of each section so that a decrease in the mean square error (MSE) is linked to a decrease in the cross entropy (CE), and the cross entropy (CE) function reflects the distance between the correct section and the predicted section as a weight so that the predicted MMSE score can be distributed similarly to the actual MMSE score.

[0013] Another object of the present invention is to provide an AI-based hearing loss, cognitive impairment, and dementia screening system using voice data capable of more accurately detecting Alzheimer's (AD) by utilizing disfluency features including static features, participant features, and interviewer features through voice segmentation and speaker segmentation using a voice activity detection (VAD) system, and at this time, considering, among various static features, in particular, the number of static segments between consecutive participant segments (Num_PS), the average length of static segments between consecutive participant segments (Avg_PS), and the variance of the length of static segments between consecutive participant segments (Var_PS) related to static features in the participant's speech, and considering, in relation to the interviewer feature, the number of interviewer segments in a voice recording (Num_I), and the number of speaker changes between participant segments and interviewer segments in a voice recording (Num_chn) related to the degree of interviewer intervention.

[0014] Another object of the present invention is to provide an AI-based hearing loss, cognitive impairment, and dementia screening system using voice data, which can immediately confirm the results of diagnosis / prediction using the system of the present invention, and can protect personal information by encrypting and storing the acquired voice data and automatically deleting it after a certain period of time.

[0015] In order to achieve the above-mentioned purpose, the present invention is implemented by an embodiment having the following configuration.

[0016] According to one embodiment of the present invention, an AI-based hearing loss, cognitive impairment, and dementia screening system using voice data according to the present invention is characterized by including: an input unit for collecting a user's voice data; a recording environment determination unit for checking an appropriate state of an environment for collecting the voice data; a voice volume determination unit for checking the user's voice volume during the process of collecting the voice data; a diagnosis prediction unit for diagnosing hearing loss, cognitive impairment, and dementia using an artificial intelligence (AI) algorithm based on the collected voice data; and an output unit for providing a result of the diagnosis prediction unit.

[0017] According to another embodiment of the present invention, the recording environment determination unit of the present invention is characterized by including an environment determination module that checks whether the user's surrounding environment is suitable for collecting voice data, and a user status determination module that checks whether the user's current physical condition is suitable for collecting voice data.

[0018] According to another embodiment of the present invention, the surrounding environment judgment module in the present invention includes a first judgment module that checks the surrounding environment state based on the sound received after generating a sound for judgment, a second judgment module that checks the surrounding environment state by obtaining a video or image of the surrounding environment, and a third judgment module that checks the surrounding environment noise based on information input through the input unit, and the user state judgment module includes a fourth judgment module that checks whether a disease suffered by the user is appropriate for voice data collection based on information input from the user, and a fifth judgment module that checks whether the user's voice quality is appropriate for voice data collection based on information collected through the user's natural speech.

[0019] According to another embodiment of the present invention, the diagnosis prediction unit in the present invention is characterized by including a hearing loss diagnosis unit that extracts voice features from voice data and diagnoses hearing loss using an artificial intelligence (AI) algorithm based on the extracted voice features, and a cognitive impairment and AD diagnosis unit that extracts acoustic features from voice data and predicts an MMSE (Mini-Mental Status Examination) score using an artificial intelligence (AI) algorithm based on the extracted acoustic features and diagnoses Alzheimer's Disease (AD).

[0020] According to another embodiment of the present invention, the hearing loss diagnosis unit in the present invention receives wav2vec speech features and diagnoses hearing loss using Swin Transformer, and the wav2vec speech features are characterized in that they are extracted from both original speech data including speech data obtained by reading negative long sentences in an emotional state and speech data obtained by reading positive long sentences in an emotional state excluded, and reversed speech data obtained by enhancing the original speech data by playing it backwards using a time inversion method.

[0021] According to another embodiment of the present invention, the cognitive impairment and AD diagnosis unit of the present invention includes a cognitive impairment and AD diagnosis module that performs cognitive impairment degree prediction and AD diagnosis using a complementary algorithm that simultaneously performs MMSE score prediction and Alzheimer's diagnosis, and the complementary algorithm utilizes a loss function that reflects regression loss and classification loss together to increase the accuracy of the prediction value, and the regression loss is characterized in that the mean squared error (MSE) function is applied, and the classification loss is characterized in that the cross-entropy (CE) function is applied.

[0022] According to another embodiment of the present invention, the mean square error (MSE) function in the present invention reflects a Bernoulli penalty according to the probability value of each section, so that a decrease in the mean square error (MSE) is linked to a decrease in the cross entropy (CE), and the cross entropy (CE) function reflects the distance between the correct section and the predicted section as a weight so as to reflect the degree of error between the correct section and the predicted section.

[0023] According to another embodiment of the present invention, the cognitive impairment and AD diagnosis unit further includes an AD detection module that extracts disfluency features based on the segmented voice patterns after performing voice segmentation on voice data and detects AD using an artificial intelligence (AI) algorithm based on the extracted disfluency features, wherein the disfluency features include static features, participant features, and interviewer features.

[0024] According to another embodiment of the present invention, in the present invention, the static features include a sum of the lengths of static segments in a voice recording (Len_S), a variance of the lengths of static segments in a voice recording (Var_S), an average of the lengths of static segments in a voice recording (Avg_S), a maximum length of the lengths of static segments in a voice recording (Max_S), a ratio of static segments to the entire voice length (Rat_S), a number of static segments in a voice recording (Num_S), a number of static segments between consecutive participant segments (Num_PS), an average of the lengths of static segments between consecutive participant segments (Avg_PS), and a variance of the lengths of static segments between consecutive participant segments (Var_PS); the participant features include a ratio of participant segments to the entire voice length (Rat_P) and a number of participant segments in a voice recording (Num_P); and the interviewer features include a number of interviewer segments in a voice recording (Num_I) and a number of speaker changes between a participant segment and an interviewer segment in a voice recording (Num_chn).

[0025] The present invention can obtain the following effects through the combination and use of the configuration described above and the following examples.

[0026] The present invention enables prediction or diagnosis of various brain health conditions (hearing loss, cognitive impairment, dementia, etc.) using voice data as a biomarker, and in particular, has the effect of performing everything from checking the collection environment and user status for acquiring voice data with a single system, to monitoring voice volume during the process of acquiring voice data, and deriving prediction and diagnosis results for hearing loss, cognitive impairment, and dementia based on the acquired voice data.

[0027] The present invention has the effect of ensuring consistency and validity of data and increasing the accuracy of various brain health diagnoses by checking the surrounding environment for collecting voice data, i.e., surrounding noise conditions, soundproofing conditions, etc., and checking the current physical condition of the user who needs to collect voice data.

[0028] The present invention utilizes voice data used for diagnosing the presence or absence of hearing loss, including voice data obtained by reading negative prose (a long scenario-like text of about one minute in length) and voice data obtained by reading positive prose, particularly voice data obtained by reading negative prose with both aggravated and intense emotions, and voice data obtained by reading positive prose with no emotions, thereby enabling accurate and effective diagnosis of hearing loss regardless of the age or gender of the subject, and by utilizing data obtained by augmenting the original data together with voice data that is difficult to sufficiently secure due to issues such as the collection process or security, thereby reducing overfitting according to the number of data and utilizing effectively acquired voice data, thereby enabling accurate diagnosis of hearing loss.

[0029] The present invention can simultaneously perform MMSE score prediction and Alzheimer's diagnosis using a single algorithm by inputting acoustic features extracted from voice data, and at this time, in order to increase the accuracy of prediction and diagnosis, a loss function that reflects both regression loss and classification loss in the algorithm is utilized, and the regression loss and the classification loss can work complementarily, so that the mean square error (MSE) function reflects a Bernoulli penalty according to the probability value of each section so that a decrease in the mean square error (MSE) is linked to a decrease in the cross entropy (CE), and the cross entropy (CE) function reflects the distance between the correct section and the predicted section as a weight so that the predicted MMSE score can be distributed similarly to the actual MMSE score.

[0030] The present invention can detect Alzheimer's disease (AD) more accurately by utilizing disfluency features including static features, participant features, and interviewer features through speech segmentation and speaker segmentation using a voice activity detection (VAD) system, and at this time, among various static features, in particular, by considering the number of static segments between consecutive participant segments (Num_PS), the average length of static segments between consecutive participant segments (Avg_PS), and the variance of the length of static segments between consecutive participant segments (Var_PS) related to static features in the participant's speech, and by considering the number of interviewer segments (Num_I) in a voice recording, and the number of speaker changes (Num_chn) between participant segments and interviewer segments in a voice recording related to the degree of interviewer intervention in relation to interviewer features, it has the effect of detecting Alzheimer's disease (AD) more accurately.

[0031] The present invention has the effect of protecting personal information by enabling the results of diagnosis / prediction using the system of the present invention to be immediately confirmed, and by encrypting and storing acquired voice data and automatically deleting it after a certain period of time.

[0032] Figure 1 is a configuration diagram of a system according to the present invention.

[0033] Figure 2 is a flowchart illustrating an operation process using the system of the present invention.

[0034] Figure 3 is a table and graph showing the average accuracy of 5-Fold Cross-Validation for each model before data augmentation.

[0035] Figure 4 is a table showing the average accuracy of 5-Fold Cross-Validation by model after data augmentation.

[0036] Figure 5 is a graph showing the average accuracy of 5-Fold Cross-Validation by model after data augmentation.

[0037] Figure 6 is a conceptual diagram of a complementary algorithm model according to one embodiment of the present invention.

[0038] Figure 7 is a reference diagram of a conventional MMSE score prediction model using the MSE function.

[0039] Figure 8 is a state diagram showing an example of dividing the MMSE score range into multiple equal ranges and predicting the probability of each range.

[0040] Figure 9 is a reference diagram explaining the MMSE test process using voice data in which the examinee describes the Cookie Theft photo.

[0041] Figure 10 shows the results reflecting Bernoulli penalty and distance-based weights in MMSE score prediction.

[0042] Figure 11 shows data comparing the performance of existing models that use only acoustic features in the ADReSSo dataset.

[0043] Figure 12 is an image visualizing the disfluency characteristics in speech with and without AD.

[0044] Figure 13 is a reference diagram of the non-fluent feature presented in the present invention.

[0045] Figure 14 is a graph showing the AD identification accuracy in various AD classification models.

[0046] Figure 15 is a graph showing the contribution of disfluency features.

[0047] Figure 16 is a comparison chart of AD identification accuracy in the ADReSSo test set.

[0048] Figure 17 is a comparison chart of AD identification accuracy in the MADReSS test set.

[0049] Hereinafter, preferred embodiments of an AI-based hearing loss, cognitive impairment, and dementia screening system using voice data according to the present invention will be described in detail with reference to the attached drawings. It should be noted that identical components are represented by the same reference numerals throughout the drawings, as much as possible. Unless otherwise defined, all terms in this specification have the same general meaning as those terms understood by a person skilled in the art to which the present invention pertains, and if there is a conflict with the meaning of a term used in this specification, the definitions used in this specification shall prevail. Throughout the specification, when a part is said to "include" a certain component, this does not mean that other components are excluded, but rather that other components may be included, unless specifically stated otherwise. In addition, terms such as "... part" and "... module" described in the specification mean a unit that processes at least one function or operation, and this may be implemented by hardware, software, or a combination of hardware and software.

[0050]

[0051] Referring to FIGS. 1 to 17, an AI-based hearing loss, cognitive impairment, and dementia screening system using voice data according to one embodiment of the present invention may include an input unit (10) for collecting a user's voice data; a recording environment determination unit (20) for checking an appropriate state of an environment for collecting the voice data; a voice volume determination unit (30) for checking the user's voice volume in the process of collecting the voice data; a diagnosis prediction unit (40) for diagnosing hearing loss, cognitive impairment, and dementia using an artificial intelligence (AI) algorithm based on the collected voice data; and an output unit (50) for providing a result of the diagnosis prediction unit (40).

[0052] First, the input unit (10) is a configuration for collecting the user's voice data, and generally refers to a configuration such as a microphone capable of collecting voice data spoken by the user, and may additionally include a configuration such as a camera for collecting data on facial images including the user's eye movements as needed.

[0053] The basic structure of sound equipment (microphone) used for voice recording consists of an input section, a processing section, an amplification section, and an output section. When a human voice is input, the vibration called sound is converted into an electrical signal and stored. At this time, the frequency range that can be collected varies depending on the microphone, and the maximum sound pressure range that can be collected also differs for each microphone. In order to collect good voice data as a voice biomarker, a microphone must be able to cover the frequency range and maximum sound pressure range of the entire sound range that can be produced by the human voice, so that the sound can be collected stably without being distorted, and the unique sound wave must be collected without any processing or amplification. Therefore, it is preferable that the microphone used in the input section (10) be a microphone that can collect frequencies in the entire sound range from 350 Hz to 17 kHz, that is, a microphone that can collect all sound frequencies and record all sounds without distortion or deformation up to a maximum sound pressure range that can be collected. Meanwhile, it is desirable to manage the voice data acquired through the input unit (10) in an encrypted form and store it, and automatically delete it after a certain period of time (usually 1 to 2 days after the results are obtained) to protect personal information.

[0054] The above output unit (50) is configured to provide the results of the above diagnosis prediction unit (40), and various types of configurations that provide the results as a display or output so that the user can directly check the predicted and diagnosed results based on the input voice data with their own eyes, or provide them in the form of a file, can be utilized as the output unit (50).

[0055] The above recording environment determination unit (20) is configured to check the appropriate state of the environment for collecting voice data, and for this purpose, the recording environment determination unit (20) may include, more specifically, an environment determination module (210) that checks whether the user's surrounding environment is appropriate for collecting voice data, and a user state determination module (220) that checks whether the user's current physical state is appropriate for collecting voice data.

[0056] The above-mentioned surrounding environment judgment module (210) is configured to check whether the surrounding environment of the user is suitable for collecting voice data, and for this purpose, more specifically, it may include a first judgment module (211) that generates a sound for judgment and then checks the surrounding environment condition based on the received sound, a second judgment module (212) that obtains a video or image of the surrounding environment and checks the surrounding environment condition, and a third judgment module (213) that checks the surrounding environment noise based on information input through the input unit (10).

[0057] The above first judgment module (211) is configured to check the surrounding environment condition based on the sound received after generating a sound for judgment, that is, after the system of the present invention is installed in a place where voice data is to be recorded, a sound is first generated through the system of the present invention, and then the sound returned through the input unit (10), that is, the microphone, is received, and based on this, the surrounding environment condition, in particular, whether noise other than a specific sound or noise is mixed in and to what extent, is checked.

[0058] The second judgment module (212) is configured to check the state of the surrounding environment by obtaining a video or image of the surrounding environment. Preferably, the place where voice data is to be collected is a soundproof place with two or fewer walls where 80% or more of the wall surface is made of glass and an area of ​​at least 1.2㎡. The second judgment module (212) obtains a video or image of the surrounding environment through a camera or the like, and then checks the soundproof state of the place where voice data is to be collected based on the video or image, that is, whether the place is a soundproof place where 80% or more of the wall surface is made of glass and two or fewer walls where external noise can be blocked and the floor area of ​​the place is at least 1.2㎡.

[0059] The third judgment module (213) is configured to check the ambient noise based on the information input through the input unit (10), that is, to check whether the level of ambient noise in the space where the user's voice is to be recorded using a microphone or the like used as the input unit (10) is 20 dB or less, which can minimize the effect on the quality of the acquired voice data. If the ambient noise exceeds a certain standard (20 dB or less) as a result of the check, voice data recording is not performed, and voice data is acquired only when the standard is satisfied.

[0060] The above user status judgment module (220) is configured to check whether the user's current physical condition is suitable for collecting voice data, and for this purpose, more specifically, it may include a fourth judgment module (221) that checks whether the disease the user is suffering from is suitable for collecting voice data based on information input from the user, and a fifth judgment module (222) that checks whether the user's voice quality is suitable for collecting voice data based on information collected through the user's natural speech.

[0061] The fourth judgment module (221) above is configured to check whether the disease the user is suffering from is suitable for voice data collection based on information input by the user. It requests information on the relevant disease from the user, and checks whether the disease the user is currently suffering from or has suffered from in the past may affect voice data collection based on the information input by the user, thereby making voice data collection possible. For example, if the disease the user is currently suffering from is a sore throat, pneumonia, vocal cord nodules, etc. based on information input by the user, it may be determined that voice data collection is not possible because these are diseases that may significantly affect the reliability of voice data.

[0062] The fifth judgment module (222) is configured to check whether the user's voice quality is appropriate for collecting voice data based on information collected through the user's natural speech. Even if it is confirmed that the user does not have a disease that may affect voice data collection based on information input through the fourth judgment module (221), the user's current throat condition may have a negative effect on voice data collection without the user's awareness. Therefore, the fifth judgment module (222) collects voice data through natural speech from the user (using a configuration such as the input unit (10)) and checks whether the user's voice quality determined based on this is appropriate for collecting voice data. For example, if irregular noises (due to the current throat condition) are confirmed in the voice collected through natural speech, it may be determined that voice data cannot be collected.

[0063] The above voice volume determination unit (30) is configured to check the user's voice volume during the process of collecting voice data. It is desirable that the collected voice data be recorded at an appropriate volume level of 50 to 120 dB. The actual recording, i.e., the process of collecting voice data, is monitored in real time to check whether the user's voice volume during the recording process is maintained within an appropriate range. If it is confirmed that the voice volume is outside the appropriate range, re-recording, etc. can be performed.

[0064] In this way, in the system of the present invention, the consistency and validity of voice data acquired through checking the collection environment and user status for acquiring voice data through the recording environment determination unit (20) and the voice volume determination unit (30), etc., as well as monitoring voice volume during the voice data acquisition process, can be ensured, thereby increasing the accuracy of various brain health diagnoses based thereon.

[0065] The above diagnosis prediction unit (40) is configured to diagnose hearing loss, cognitive impairment, and dementia using an artificial intelligence (AI) algorithm based on collected voice data. To this end, the diagnosis prediction unit (40) may more specifically include a hearing loss diagnosis unit (410) that extracts voice features from voice data and diagnoses hearing loss using an artificial intelligence (AI) algorithm based on the extracted voice features, and a cognitive impairment and AD diagnosis unit (420) that extracts acoustic features from voice data and predicts an MMSE (Mini-Mental Status Examination) score using an artificial intelligence (AI) algorithm based on the extracted acoustic features and diagnoses Alzheimer's Disease (AD).

[0066] The above hearing loss diagnosis unit (410) is configured to extract voice features from voice data and diagnose hearing loss using an artificial intelligence (AI) algorithm based on the extracted voice features. At this time, the voice data acquired for hearing loss diagnosis may include voice data acquired by reading negative long sentences (a long scenario-type text of about 1 minute in length) and voice data acquired by reading positive long sentences. In particular, when reading negative long sentences, the voice data is acquired by reading the sentence with emotion, that is, by putting all the feelings of injustice and intensity into it, and when reading positive long sentences, the voice data is acquired by reading the sentence as if excluding emotion. By basing the voice data acquired in this manner on it, it is possible to acquire voice data in a wider variety of states from the subject, thereby enabling the diagnosis of hearing loss to be made accurately and effectively regardless of the subject's age, gender, etc.

[0067] Rather than utilizing only the original audio data acquired in this way, by utilizing the original audio data together with the reversed audio data obtained by playing the original audio data backwards using the time inversion method, it is possible to accurately diagnose the presence or absence of hearing loss by utilizing the data acquired by effectively utilizing the acquired audio data while reducing overfitting according to the number of data, for audio data that is difficult to sufficiently secure due to issues such as the collection process or security of the audio data, by utilizing the data acquired by augmenting the original data together.

[0068] From the acquired voice data, voice features used to diagnose hearing loss are extracted. The voice features extracted at this time can be, in particular, wav2vec voice features. wav2vec is a self-supervised speech recognition model developed by Facebook AI Research. The wav2vec uses a convolution neural network (CNN) trained to predict future voice frames from previous ones as an encoder, and can obtain high-quality voice features by detecting useful context information from the input signal. wav2vec, which includes useful and abstract features of the voice signal, outputs the sequence in the hidden state based on the vector in the last hidden layer of the model to extract a 512-dimensional feature vector. At this time, since the length of all audio is different, the dimension of the entire feature also comes out differently (512 * total length of the input audio).

[0069] The artificial intelligence (AI) algorithm used to diagnose hearing loss by inputting extracted voice features may be, in particular, Swin Transformer, which is a state-of-the-art transformer-based deep learning architecture for image recognition tasks proposed by Microsoft Research. In this patent, pre-trained features extracted from wav2vec, rather than images, were used as input to the Swin Transformer encoder. Although the Swin Transformer architecture was originally designed for image recognition tasks, the wav2vec used as input was (512 * total length of input audio) in dimension, so it could be used. In this patent, 1000 epochs, 8 batches, 5e-4 learning rate, 40 window size, 20 warmup steps were used, and AdamW was used as the optimizer.

[0070] In the present invention, a method for diagnosing hearing loss based on AI using voice data is proposed, which can diagnose hearing loss early, easily, and accurately by inputting wav2vec voice features and diagnosing hearing loss using Swin Transformer, and this is proven through the following experiments.

[0071] [Experiment 1] Accuracy test for diagnosing hearing loss using subjects' voice data

[0072] - Experimental subjects: The above experimental subjects are 483 people, divided into 321 women and 162 men by gender as shown in [Table 1] below, and 118 normal subjects and 365 hearing-impaired patients by the presence or absence of hearing loss. 1) Patients with a normal eardrum and a subjective hearing loss period of 3 years or longer as determined by a professional hearing test by an otolaryngologist-head and neck surgeon, 2) Patients with a hearing loss average of 40 dB or higher at 500 Hz, 1000 Hz, 2000 Hz, and 4000 Hz based on their better hearing.

[0073] Total number of normal patients: females: 89,232,321; males: 29,133,162; total number of normal patients: 11,836,5483

[0074] - Voice data collection: For each experimental subject, two tasks were performed to collect voice data: reading a negative long scenario lasting about 1 minute (chapter 7) and reading a positive long newspaper editorial (chapter 8). In chapter 7, when reading a negative long sentence, the subjects were instructed to read it with all their feelings of injustice and intensity, and in chapter 8, when reading a newspaper editorial, they were instructed to read it as if they were reading a book without any emotions. - Data augmentation: The original audio data acquired in this experiment are voice data from a total of 483 subjects, including 118 normal subjects and 365 hearing-impaired patients. Therefore, in order to reduce overfitting to the number of data and enable the model to perform well in a real environment, data augmentation was performed by reversing the original audio data to extract features (voice features) (in particular, data augmentation was performed using the time inversion method). Through this process, by combining the original audio data and the reversed audio data for training, a total of 966 audio data, 236 normal and 730 patient data, can be used for model training and validation.

[0075] - Experimental environment: This experiment was conducted in Linux and Windows environments, and one NVIDIA GeForce RTX 2070 16GB and two A100 40GB were used to train the deep learning model in a GPU environment.

[0076] - Experimental method: Three features (voice features) were extracted from all acquired voice data (dataset): eGemaps, mel-spectrogram, and wav2vec. Then, they were applied to four models (artificial intelligence algorithms): SVM, MLP-LSTM, VGGish, and Swin. Classification was performed using 5-Fold Cross-Validation with two labels: normal (0) and patient (1). The Unseen method uses 80% of the entire dataset for training and 20% for validation, so that people who have never been trained are tested. The Cross-Subject method uses 80% of one person's voice data for training and 20% for validation, so that people who have been trained once are tested. However, a different portion of the voice data is used from the one used for training.

[0077] - Experimental results

[0078] 1. First, classification was performed using 5-Fold Cross-Validation without data augmentation, that is, using only the original audio data. The audio features and models (artificial intelligence algorithms) used here were MLP-LSTM and SVM using eGemaps features, Swin Transformer using wav2vec features, and VGGish using unique VGGish features after converting audio to Mel-stectrogram. As a result, as shown in 'Figure 3', the performance of Swin and VGGish 50 epochs are similar, but when the number of epochs is increased to 100, the performance of VGGish is the highest at approximately 81.24% in Unseen and 84.44% in Cross Subject. On the other hand, MLP-LSTM and SVM using eGemaps features did not achieve an accuracy of more than 75% in all results. Meanwhile, in the case of the Unseen verification method, since verification is performed on data from subjects who have never been trained, it can be confirmed that it is about 2-3% lower than the Cross Subject verification method.

[0079] 2. Since the performance of the MLP-LSTM and SVM models using eGemaps features when data augmentation was not good, we additionally conducted experiments after data augmentation using only Swin and VGGish, which had good performance (using the original speech data and the reversed speech data to extract wav2vec and VGGish features, and training and validating the Swin and VGGish models with 5-Fold Cross-Validation). As a result, as shown in 'Fig. 4' and 'Fig. 5', in the Unseen method, Swin showed the best performance with a classification accuracy of approximately 84.11%, and in the Cross subject method, VGGish showed the best performance with a classification accuracy of approximately 85.54%.

[0080] That is, through this [experiment], it was confirmed that diagnosis of hearing loss is possible by utilizing voice data (chapter 7) obtained through negative long sentence reading and voice data (chapter 8) obtained through positive long sentence reading presented in the present invention, and in particular, when the two data are combined and utilized, the classification accuracy for hearing loss increases, and it can be confirmed that the classification accuracy is even higher when both normal original voice data and data-enhanced reversed voice data are used.

[0081] In addition, in the voice features and models for diagnosing hearing loss, in the case of the Unseen verification method, the accuracy of Swin is mostly high in the low to mid 80% range, and in the case of the Cross-subject verification method, there are folds that show an accuracy of over 90% in Swin or VGGish, and when averaged, most show an accuracy of the mid 80%, and it can be confirmed that the performance of Swin and VGGish is similar. However, in the subject-independent verification method, the unseen subject task is more difficult than the cross-subject method, and considering future general use, it can be confirmed that the method of diagnosing hearing loss using the Swin Transformer by inputting wav2vec voice features is the most efficient and suitable as a model for accurately classifying the presence or absence of hearing loss.

[0082] The above cognitive impairment and AD diagnosis unit (420) is configured to extract acoustic features from voice data and perform cognitive impairment degree prediction and Alzheimer's (AD) diagnosis using an artificial intelligence (AI) algorithm based on the extracted acoustic features. To this end, the cognitive impairment and AD diagnosis unit (420) may more specifically include a cognitive impairment and AD diagnosis module (421) that performs cognitive impairment degree prediction and AD diagnosis using a complementary algorithm that simultaneously performs MMSE score prediction and Alzheimer's diagnosis, and an AD detection module (422) that performs voice segmentation in voice data, extracts disfluency features based on the segmented voice pattern, and detects AD using an artificial intelligence (AI) algorithm based on the extracted disfluency features.

[0083] The above cognitive impairment and AD diagnosis module (421) is configured to perform cognitive impairment degree prediction and AD diagnosis using a complementary algorithm that simultaneously performs MMSE score prediction and Alzheimer's diagnosis. The acoustic features of the voice data used for such prediction and diagnosis are extracted from the entire voice data in order to reflect the context of the voice data. In the existing model using linguistic features, the features must be extracted by creating a transcription from the voice manually or through automatic speech recognition (ASR). However, manually creating a transcription is costly and time-consuming, and the use of ASR causes problems in that the model is vulnerable to noise and the performance of the model is affected by the ASR performance. Therefore, the present invention has the feature of being able to accurately diagnose AD and predict the degree of cognitive impairment using only acoustic features (sequential information of acoustic features).

[0084] In the process of predicting the MMSE (Mini-Mental Status Examination) score and diagnosing Alzheimer's Disease (AD) by using a neural network by inputting the extracted acoustic features, the present invention is characterized by simultaneously performing MMSE score prediction and Alzheimer's diagnosis using one algorithm (complementary algorithm). That is, conventional models that predict MMSE scores or diagnose Alzheimer's disease in order to predict the degree of cognitive impairment using voice features have been separately performing MMSE score prediction and Alzheimer's disease diagnosis (AD Detection) using separate regression models and classification models, respectively. For example, the conventional MMSE score prediction model shown in FIG. 7 applies the MSE (Mean Squared Error) function to predict MMSE scores, but since the predicted value regresses around the mean, it is impossible to determine whether Alzheimer's disease is present with the predicted MMSE score, so MMSE score prediction and Alzheimer's disease diagnosis (AD Detection) still have to be performed using separate models.

[0085] In order to solve these problems, the present invention utilizes a uniquely designed complementary algorithm to simultaneously perform MMSE score prediction and Alzheimer's diagnosis. To this end, the complementary algorithm utilizes a loss function that reflects both regression loss and classification loss to increase the accuracy of the predicted value, as illustrated in FIG. 6.

[0086] Meanwhile, in order to optimize the predicted value to the actual distribution, the MMSE score prediction divides the MMSE score range into multiple equal sections, as in the example illustrated in Fig. 4, and predicts the probability of each section, and the MMSE score at this time is estimated as the expected value of the probability distribution. In the example illustrated in Fig. 8, the total score range of the MMSE (30 points) is divided into 10 equal sections of 3 points each, and the probability of each section is predicted, and then the MMSE score is estimated as the expected value of the probability distribution.

[0087] The complementary algorithm of the present invention is characterized in that the mean squared error (MSE) function is applied as the regression loss, and the cross-entropy (CE) function is applied as the classification loss. That is, in order to make the model's prediction similar to the actual correct answer distribution even with a small amount of data in predicting the MMSE score, the cross-entropy (CE) function used in classification is additionally reflected in addition to the mean squared error (MSE) function, which can be confirmed in the mathematical expression 4 below.

[0088]

[0089]

[0090]

[0091]

[0092] (In the above mathematical equations 1 to 4, is the MMSE score of the i-th patient within a batch of size N, is the probability of the jth interval of the ith MMSE score, is the probability of the jth interval of the ith predicted value)

[0093] That is, in the above mathematical expression 4, if the interval is well predicted through CE, the predicted MMSE score also becomes closer to the actual MMSE score, which has the characteristic of leading to a decrease in MSE.

[0094] However, in situations where there is little data, such as medical data, since MSE regresses the predicted value around the mean, a decrease in MSE does not necessarily lead to a decrease in CE, and thus a situation may arise where the two loss functions cannot complement each other. Therefore, the present invention additionally reflects a Bernoulli penalty in the mean squared error (MSE) function according to the probability value of each section so that a decrease in the mean squared error (MSE) is linked to a decrease in the cross entropy (CE) so that the regression loss and the classification loss can work complementarily even in situations where there is little data.

[0095] The modified MMSE calculation formula (MSE(M,g(P))) that reflects the Bernoulli penalty can be found in Equation 5 below.

[0096]

[0097] Because the variance of the Bernoulli distribution is highest when the probability is 0.5, if the predicted probability in each interval is similar to the remaining predicted probabilities, the MSE suffers a significant loss. This allows the model to predict the probability of each interval as close to 1 or 0.

[0098] Meanwhile, since the cross-entropy (CE) function is generally calculated only with the probability of the correct interval, it does not reflect the degree of error between the correct interval and the predicted interval. Therefore, this patent proposes Weighted CE, which reflects the distance between the correct interval and the predicted interval as a weight to reflect the degree of error between them. In other words, by reflecting the distance between the correct interval and the predicted interval as a weight in the cross-entropy (CE) function, the predicted MMSE score can be distributed similarly to the actual MMSE score.

[0099] The distance between the correct interval and the predicted interval is defined by two things: the absolute distance and the square distance, which can be confirmed through mathematical equations 6 and 7 below, and the proposed Weighted CE can be confirmed through mathematical equation 8 below.

[0100]

[0101]

[0102]

[0103] (At this time, and represent the absolute distance and the square distance, respectively)

[0104] Finally, the weighted MSE-CE loss function to which the Bernoulli penalty is applied in the present invention is as shown in Equation 9 below.

[0105]

[0106] In this way, the weighted MSE-CE loss function with Bernoulli penalty applied makes MSE and CE complement each other by applying Bernoulli penalty to the MMSE calculation formula, and by reflecting the distance between the correct answer interval and the predicted interval as a weight in CE, it is possible to learn to predict similarly to the actual correct answer distribution.

[0107] In the following experiments, we verify the effectiveness of the weighted MSE-CE loss function with the Bernoulli penalty proposed in the present invention, and compare its performance with existing models that use only acoustic features on the ADReSSo dataset (Koo et al.'s model that won first place in the ADReSSo Challenge 2020 (Koo and Junghyun, "Exploiting multi-modal features from pre-trained networks for Alzheimer's dementia recognition." in the Proc. of Interspeech, September 2020)).

[0108] [Experiment 2] Experimental Effect of Weighted MSE-CE Loss Function with Bernoulli Penalty

[0109] - ADReSSo dataset

[0110] The ADReSSo dataset is designed for Alzheimer's disease classification, cognitive ability assessment score inference, and cognitive decline prediction. It consists of 166 training datasets and 71 validation datasets. The audio files contain the subject's voice describing the Cookie Theft photo (see Figure 9), and may include a nurse's voice. In this patent, VGGish features extracted from the audio files and speaker-specific voice segments are used as model input.

[0111] - Experimental method

[0112] The MMSE Score Prediction model is tested by setting the number of intervals B to 10 and the penalty tuning parameter λ to 1. AD Detection is predicted by thresholding the predicted value of the model that performed MMSE Score Prediction at a score of 27. Finally, all training data are augmented with time inversion for the experiment.

[0113] - Experimental results

[0114] Referring to Fig. 10, when only MSE was used as a loss function, as in the model of Koo et al. that won first place in the ADReSSo Challenge 2020, it showed an RMSE of 6.60, which is very low performance, and when AD Detection classification was performed with the predicted scores, it showed an accuracy of 49.30%, which is very low because the model mostly predicted scores around the mean.

[0115] At this time, if CE is additionally reflected in the loss function, as in the model proposed in the present invention (Jin et al. model), the model's predicted values ​​can be trained to be similar to the distribution of actual MMSE scores, and the reduction in classification loss for the interval leads to a reduction in MSE, and it can be seen that RMSE is reduced to 5.49. Through more accurate MMSE score prediction, it can be confirmed that the classification accuracy in AD Detection also increases by 16.90%.

[0116] Here, when the Bernoulli penalty is reflected in the MMSE calculation formula, as in the model (Proposal model) ultimately proposed in this invention, the decrease in MSE leads to a decrease in CE, and the two loss functions complement each other. Therefore, we can see that the RMSE improves by 0.70 and the accuracy improves by 4.22% compared to when the Bernoulli penalty is not applied. Additionally, when the weighted CE is used, the RMSE achieves 4.55 and the accuracy reaches 77.46%, which is the best performance when using the absolute distance. However, when only the weighted CE is reflected without the Bernoulli penalty, the performance is improved only in the MMSE score prediction (since the two loss functions do not complement each other).

[0117] - Comparison with a model that uses only acoustic features from the ADReSSo dataset.

[0118] Referring to Fig. 11, 1) Among the models that performed MMSE score prediction, Luz et al. [S. Luz, F. Haider, Sdl Fuente, D. Fromm, B. MacWhinney "Detecting cognitive decline using speech only: The address challenge." in Proc, Interspeech, 2021, pp. 3780-3784] performed SVM with eGeMAPS and obtained an RMSE of 6.09. P rez-Toro et al.[PA P rez-Toro, et al., "Influence of the Interviewer on the Automatic Assessment of Alzheimer's Disease in the Context of the ADReSSo Challenge," in Proc, Interspeech, 2021, pp. 3785-3789] achieved the best RMSE of 5.35 among models that used only existing acoustic features through linear regression using Prosody Feature, X-vector, and Valence and Dominance features of VAD. However, this patent (Weighted MSE-CE Loss with Bernoulli Penalty) achieves a significantly improved RMSE of 4.55 while simultaneously performing MMSE score prediction and AD Detection using only VGGish features. 2) In addition, among models that performed AD Detection, Luz et al. obtained an accuracy of 64.79% with eGeMAPS. Gauder et al.[L. Gauder, L. Pepino, L. Ferrer, P. Riera, "Alzheimer Disease Recognition Using Speech-Based Embeddings From Pre-Trained Models," in Proc. Interspeech, 2021, pp. 3795-3799] achieved the highest accuracy of 78.9% among models that use only acoustic features by stacking 1D CNN layers using Wav2Vec 2.0. However, in this patent, a higher accuracy of 80.28% was achieved while simultaneously performing MMSE score prediction and AD detection, and a similar accuracy of 77.46% was achieved without data augmentation. 3) Finally, in this patent, the highest performance was achieved in both MMSE score prediction and AD detection among models that use only acoustic features in the ADReSSo dataset.In particular, unlike existing models that performed the two tasks as separate models, this patent achieved higher performance while simultaneously performing both tasks as a single model.

[0119] In conclusion, the model proposed in this patent reflects both regression loss and classification loss in MMSE score prediction, and reflects a penalty for the predicted probability value and a distance-based weight on the classification loss. The Bernoulli penalty makes the regression loss and the classification loss complement each other during learning, and the distance-based weight allows the model to better learn the distribution of actual MMSE scores. As a result, as confirmed in Experiment 2 above, it achieved the highest RMSE of 4.55 and accuracy of 80.28% among the results of MMSE score prediction and AD detection using only acoustic features in the ADReSSo dataset. In this way, the present invention can predict the Mini-Mental Status Examination (MMSE) score and diagnose Alzheimer's Disease (AD) by using a neural network by inputting acoustic features extracted from voice data, that is, it can simultaneously perform MMSE score prediction and Alzheimer's diagnosis using a single algorithm, and at the same time, the mean square error (MSE) function reflects the Bernoulli penalty according to the probability value of each section so that the regression loss and the classification loss can work complementarily, and the cross entropy (CE) function reflects the distance between the correct section and the predicted section as a weight so that the predicted MMSE score can be distributed similarly to the actual MMSE score, thereby enabling more accurate prediction and diagnosis.

[0120] The above AD detection module (422) is configured to detect AD by extracting disfluency features based on the segmented voice pattern after performing voice segmentation on voice data and using an artificial intelligence (AI) algorithm based on the extracted disfluency features. In particular, it focuses on disfluency in the speaker's spontaneous speech, and the voice segmentation also targets voice data in the speaker's spontaneous speech. In addition, in order to extract disfluency features from voice recordings without relying on transcription, the voice segmentation is performed using a Voice Activity Detection (VAD) system (algorithm).

[0121] The step of segmenting speech from speech data is a process of segmenting the input speech data into speech segments and static segments using a neural network (e.g., the VAD algorithm or a pre-trained audio neural network (PANN)). The goal is to divide the speech data into speech parts (speech segments) and pause parts (static segments). To this end, pre-trained audio neural networks (PANNs: large-scale pretrained audio neural networks for audio pattern recognition) can be used to capture speech patterns. The PANN receives raw recorded speech data as input and outputs audio tags. Audio segments tagged as 'speaking' are regarded as speech parts (speech segments), and other segments are regarded as pause parts (static segments).

[0122] Afterwards, the segmented voice segments are segmented into participant segments and interviewer segments using a neural network. For this speaker segmentation, the audio segments (voice segments) previously tagged as 'speaking' can be automatically separated into different speakers using a neural network (e.g., Pyannote (Pyannote.Audio: neural building blocks for speaker diarization)). For example, if the minimum number of speakers is set to 1 and the maximum number of speakers is set to 2, the speaker with a longer speech time in the audio segment can be regarded as the participant, and the other speakers can be automatically separated as the interviewer. Consequently, through this process, the voice data can be expressed as a sequence of P representing the voice segment of the participant, I representing the voice segment of the interviewer (Investigator or Interviewer), and S representing the silence segment (see Figure 12).

[0123] For reference, Figure 12 is an image visualizing the disfluency features in speech with and without AD. Referring to Figure 12, it shows that participants with AD have longer response times, more interviewer intervention, and longer and more pauses between utterances regardless of language. This part is utilized in designing the disfluency features described later.

[0124] Afterwards, disfluency features are extracted based on the segmented speech patterns. The disfluency features in this process may include static (S) features, participant (P) features, interviewer (I) features, and metadata (M) features (see Figure 13).

[0125] First, the above static features can be seen as the most important features for AD detection, because abnormal speech interruptions, longer durations of silence, and greater number of pauses in speech are symptoms that appear in Alzheimer's disease (AD) patients. The above static features basically include statistical features. That is, they can include the sum of the lengths of static segments in a voice recording (Len_S), the variance of the lengths of static segments in a voice recording (Var_S), the average of the lengths of static segments in a voice recording (Avg_S), the maximum length of static segments in a voice recording (Max_S), the ratio of static segments to the total voice length (Rat_S), and the number of static segments in a voice recording (Num_S). Among the above static features, the sum of the lengths of static segments in a voice recording (Len_S), the average of the lengths of static segments in a voice recording (Avg_S), the ratio of static segments to the total voice length (Rat_S), and the number of static segments in a voice recording (Num_S) are static features that are considered to increase the accuracy of AD diagnosis through statistical features related to static segments in the input voice data. In particular, among the above static features, the maximum length of static segments in a voice recording (Max_S) and the variance of the lengths of static segments in a voice recording (Var_S) are features that are considered to enable more accurate AD detection using static features while minimizing the influence of abnormal data that may be included in the voice data.

[0126] Meanwhile, in the above static features, only features related to static features in the participant voice data, i.e., features induced by the participant (P) among the speakers, can be additionally included. Specifically, the number of static segments between consecutive participant segments (Num_PS), the average length of static segments between consecutive participant segments (Avg_PS), and the variance of the length of static segments between consecutive participant segments (Var_PS) can be additionally included. The number of static segments between consecutive participant segments (Num_PS) can be used to identify the characteristics of the number of static segments in the participant's continuous speech process, that is, how often pauses (silence pauses) appear in the speech process. The average length of the static segments between consecutive participant segments (Avg_PS) can be used to identify the characteristics of how long the static segments, i.e., pauses (silence pauses, voiceless pauses are easier to extract from speech signals (speech data) than filled pauses such as 'uh', 'um', etc., and can be used for analysis regardless of language) continue on average in the participant's continuous speech process. The variance of the length of the static segments between consecutive participant segments (Var_PS) can be used to identify the characteristics of how evenly the length of the static segments, i.e., pauses (silence pauses), appear in the participant's continuous speech process. Therefore, by utilizing these characteristics, it is possible to identify the characteristics of how slowly the original speech is made (even in the absence of AD disease). This can improve the accuracy of AD diagnosis by filtering out participants with a speech tendency to have regular pauses between pronunciations.

[0127] The above participant features include the proportion of participant segments in the total speech length (Rat_P) and the number of participant segments in the speech recording (Num_P), and the above interviewer features include the number of interviewer segments in the speech recording (Num_I) and the number of speaker changes between participant segments and interviewer segments in the speech recording (Num_chn). The interviewer features do not include the proportion of interviewer segments (Rat_I) to avoid unnecessary duplication (i.e., Rat_S+Rat_P+Rat_I is always 1).

[0128] In particular, the above interviewer characteristics serve as an important factor in increasing the accuracy of AD diagnosis, because the interviewer's (investigator's) intervention occurs more frequently during the interview (investigation) process for AD patients, and therefore, not only the number of interviewer segments in the voice recording (Num_I), but especially the number of speaker changes between the participant segment and the interviewer segment in the voice recording (Num_chn) can be viewed as disfluency characteristics that contribute to increasing the accuracy of AD diagnosis.

[0129] The above metadata features are features that allow for the consideration of the relativity of voice data and disfluency features extracted therefrom in AD diagnosis using acquired voice data, and may include the total recording length (Len) of the voice recording, the participant's age (Age), and the participant's educational level (Edu) in years. In other words, it allows for the consideration that the older the participant, the longer it takes to speak overall and the higher the risk of AD, and that the participant's educational level affects spontaneous speech, thereby allowing for the consideration of the relativity of disfluency features for each participant, thereby enabling a more accurate AD diagnosis.

[0130] The extracted disfluent features are inputted and a neural network is used to detect Alzheimer's (AD; Alzheimer's Disease). However, since the dataset for diagnosing cognitive impairment (including AD) is generally smaller than that of other fields, overfitting to the learning data can be a problem. To prevent this overfitting, a simple machine learning classifier can be used for AD classification / diagnosis. Examples include a support vector machine (SVM) using a linear kernel and a regularization parameter 1, a multilayer perceptron (MLP) with a single hidden layer of 20 hidden units, a K-nearest neighbor (KNN) classifier with a K value of 5 (K=5), and a random forest (RF) with 20 trees and a maximum depth of 5. Among these, as will be described later, the model that can increase accuracy is the one that receives the extracted disfluent features as input and uses the random forest (RF) model to detect Alzheimer's (AD; Alzheimer's Disease).

[0131] Below, we verify the accuracy of the AD detection model using a simple classifier by utilizing the disfluency feature in speech proposed in the present invention through [Experiment 3].

[0132] - Data set used

[0133] To evaluate the performance of our model, we utilized two publicly available datasets: ADReSSo and MADReSS. These datasets were used in the AD Recognition Challenge held in 2021 and 2023, respectively. These datasets contain audio recordings of photo captions generated by individuals with AD and cognitively normal individuals. The ADReSSo dataset consists of 156 English training data and 71 English test data, while the MADReSS dataset consists of 237 English training data and 46 Greek test data. Both datasets provide only audio recordings, without any transcription. For the AD classification / diagnosis task, the audio recordings were labeled as "normal" and "suspected AD," and gender and age were balanced to minimize potential bias.

[0134] - Performance comparison of machine learning classifiers

[0135] In the present invention, the disfluency features in speech were utilized to verify the AD identification accuracy through four machine learning models, including SVM, MLP, KNN, and RF (for this, 5-fold cross-validation accuracy was used on the ADReSSo training set). As shown in Fig. 14, it can be confirmed that the proposed disfluency features showed reasonable performance in all of the various models, and in particular, the Random Forest (RF) model showed the best performance (accuracy of 78.3%, low variance). Since the RF model is an ensemble learning method that builds various decision trees during training, the following experiments were conducted using the RF model.

[0136] - Analysis of disfluency features

[0137] In order to examine the characteristics (contribution) of the disfluency features proposed in the present invention, we used the results of 5-fold cross-validation on the ADReSSo training set while sequentially removing some features from the results utilizing all disfluency features. As shown in Fig. 15, the results show that removing the number of interviewer segments (Num_I) from the voice recording has a greater impact on the performance than simply removing the features of the proportion of participant segments to the entire speech length (Rat_P) and the number of participant segments in the voice recording (Num_P). This confirms that the interviewer feature serves as a more important feature because participants with AD are more influenced and intervened by the interviewer during their speech. In addition, when the static features such as the ratio of static segments to the total speech length (Rat_S), the sum of the lengths of static segments in the voice recording (Len_S), and the maximum length of static segments in the voice recording (Max_S) were removed, the results showed that they had a great impact on the AD detection performance. This confirms that static features during speech, namely silence pauses, are important features in detecting cognitive impairments such as AD. Finally, when the metadata feature such as the participant's year of education (Edu) was removed, there was a slight decrease in performance, confirming that these metadata features provide an auxiliary function in AD diagnosis. In conclusion, among the disfluency features, static features are the most important factor in AD detection, and interviewer features, especially features related to the number of interventions, are also very important features.

[0138] - Comparison with other AD identification / diagnosis methods (models)

[0139] To verify the superiority of the model proposed in this invention, we compared it with other models that only use audio features, such as wav2vec, MFCC eGeMAPS, or ComParE. We trained the model on the ADReSSo training set and compared the test results on the test set. As shown in Figure 16, the baseline method presented the ADReSSo dataset, built a simple model using eGeMAPS and an SVM classifier, and achieved an accuracy of 64.8%, which is similar to Pappagari et al. ["Automatic detection and assessment of alzheimer's disease using speech and language technologies in low-resource scenarios," in Proc. Interspeech, 2021] and Pan et al. ["Using the outputs of different automatic speech recognition paradigms for acoustic- and BERT-based alzheimer's dementia detection through spontaneous speech," in Proc. Interspeech, 2021] achieved 74.7% accuracy on the ADReSSo test set using ASR embeddings and wav2vec 2.0. However, ASR embeddings and wav2vec features are extracted from a model trained solely on English data, making them difficult to transfer to other languages. In contrast, the disfluency feature of our proposed model (Ours) outperforms other features (76.6% accuracy), demonstrating its potential for use in other languages.

[0140] Additionally, to evaluate the performance of the model proposed in the present invention in different language scenarios, the model was trained with MADReSS English training data and the results were compared on Greek test data. As shown in Fig. 17, the performance of the model of the present invention was compared with the MADReSS baseline ["Multilingual alzheimer's dementia recognition through spontaneous speech: a signal processing grand challenge," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2023]. The results showed that the model of the present invention (Ours) using the disfluency feature achieved 80.4% accuracy on the Greek data, which was 6.5% higher than the baseline. Therefore, this result indicates that the disfluency feature is a speech feature that can be generalized across different languages ​​in the context of AD detection.

[0141] In summary, the system of the present invention enables prediction or diagnosis of various brain health conditions (hearing loss, cognitive impairment, dementia, etc.) using voice data as a biomarker, and in particular, it performs everything from checking the collection environment and user status for acquiring voice data with a single system, to monitoring voice volume during the process of acquiring voice data, and deriving prediction and diagnosis results for hearing loss, cognitive impairment, and dementia based on the acquired voice data.

[0142]

[0143] In the above, the applicant has described various embodiments of the present invention, but such embodiments are only examples of implementing the technical idea of ​​the present invention, and any change or modification that implements the technical idea of ​​the present invention should be interpreted as falling within the scope of the present invention.

Claims

1. Input unit for collecting user's voice data; A recording environment judgment unit that checks the appropriate conditions of the environment for collecting voice data; A voice volume judgment unit that checks the user's voice volume during the process of collecting voice data; A diagnostic prediction unit that diagnoses hearing loss, cognitive impairment, and dementia using an artificial intelligence (AI) algorithm based on collected voice data; and An AI-based hearing loss, cognitive impairment, and dementia screening system using voice data, characterized by including an output unit that provides the results of the above diagnostic prediction unit.

2. In paragraph 1, The above recording environment judgment unit is characterized by including an environment judgment module that checks whether the user's surrounding environment is suitable for collecting voice data, and a user condition judgment module that checks whether the user's current physical condition is suitable for collecting voice data. An AI-based hearing loss, cognitive impairment, and dementia screening system using voice data.

3. In paragraph 2, The above-mentioned ambient environment judgment module includes a first judgment module that generates a sound for judgment and then checks the ambient environment condition based on the received sound, a second judgment module that obtains a video or image of the ambient environment and checks the ambient environment condition, and a third judgment module that checks ambient environment noise based on information input through the input unit. The user status judgment module includes a fourth judgment module that checks whether the user's disease is suitable for voice data collection based on information input from the user, and a fifth judgment module that checks whether the user's voice quality is suitable for voice data collection based on information collected through the user's natural speech, wherein the AI-based hearing loss, cognitive impairment, and dementia screening system using voice data is characterized in that the user status judgment module includes a fifth judgment module that checks whether the user's voice quality is suitable for voice data collection based on information collected through the user's natural speech.

4. In paragraph 1, The above diagnosis prediction unit includes a hearing loss diagnosis unit that extracts voice features from voice data and diagnoses hearing loss using an artificial intelligence (AI) algorithm based on the extracted voice features, and a cognitive impairment and AD diagnosis unit that extracts acoustic features from voice data and predicts the MMSE (Mini-Mental Status Examination) score using an artificial intelligence (AI) algorithm based on the extracted acoustic features and diagnoses Alzheimer's Disease (AD). An AI-based hearing loss, cognitive impairment and dementia screening system using voice data.

5. In paragraph 4, The above hearing loss diagnosis unit receives wav2vec voice features and diagnoses hearing loss using Swin Transformer, and the wav2vec voice features are extracted from both original voice data including voice data obtained by reading negative long sentences in an emotional state and voice data obtained by reading positive long sentences in an emotional state without emotion, and reversed voice data obtained by enhancing the original voice data by playing it backwards using a time inversion method. An AI-based hearing loss, cognitive impairment, and dementia screening system using voice data.

6. In paragraph 4, The above cognitive impairment and AD diagnosis section includes a cognitive impairment and AD diagnosis module that performs cognitive impairment degree prediction and AD diagnosis using a complementary algorithm that simultaneously performs MMSE score prediction and Alzheimer's diagnosis. The above complementary algorithm utilizes a loss function that reflects both regression loss and classification loss to increase the accuracy of the predicted value. An AI-based hearing loss, cognitive impairment, and dementia screening system using voice data, characterized in that the regression loss applies a mean squared error (MSE) function, and the classification loss applies a cross-entropy (CE) function.

7. In paragraph 6, The above mean square error (MSE) function reflects the Bernoulli penalty according to the probability value of each interval, so that the decrease in the mean square error (MSE) is linked to the decrease in the cross entropy (CE). An AI-based hearing loss, cognitive impairment, and dementia screening system using voice data, characterized in that the cross entropy (CE) function reflects the distance between the correct answer interval and the predicted interval as a weight so as to reflect the degree of error between the correct answer interval and the predicted interval.

8. In paragraph 6, The above cognitive impairment and AD diagnosis unit additionally includes an AD detection module that extracts disfluency features based on the segmented voice patterns after performing voice segmentation on voice data, and detects AD using an artificial intelligence (AI) algorithm based on the extracted disfluency features. An AI-based hearing loss, cognitive impairment and dementia screening system using voice data, characterized in that the above disfluency features include static features, participant features and interviewer features.

9. In paragraph 8, The above static features include the sum of the lengths of static segments in a voice recording (Len_S), the variance of the lengths of static segments in a voice recording (Var_S), the average of the lengths of static segments in a voice recording (Avg_S), the maximum length of static segments in a voice recording (Max_S), the ratio of static segments to the total voice length (Rat_S), the number of static segments in a voice recording (Num_S), the number of static segments between consecutive participant segments (Num_PS), the average of the lengths of static segments between consecutive participant segments (Avg_PS), and the variance of the lengths of static segments between consecutive participant segments (Var_PS). The above participant characteristics include the ratio of participant segments to the total speech length (Rat_P), the number of participant segments in the speech recording (Num_P), An AI-based hearing loss, cognitive impairment and dementia screening system using voice data, characterized in that the above interviewer features include the number of interviewer segments (Num_I) in a voice recording and the number of speaker changes between participant segments and interviewer segments (Num_chn) in a voice recording.

Images