Depression level estimation device and method

The depression level estimation device uses acoustic features and machine learning to accurately assess the severity of depression from speech, overcoming the limitations of conventional diagnosis methods.

JP2024164554APending Publication Date: 2024-11-27HIROSHIMA CITY UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023080126
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-15
Publication Date
2024-11-27

AI Technical Summary

Technical Problem

Conventional methods can accurately diagnose depression but fail to estimate the specific level of depression from audio data.

Method used

A depression level estimation device comprising an acoustic feature extraction unit and a machine learning device that processes speech utterances to estimate the degree of depression using trained acoustic features and machine learning algorithms like SVR and LightGBM.

Benefits of technology

The device effectively estimates the degree of depression from speech, providing accurate results even for moderate levels, as demonstrated by high correlation coefficients.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024164554000001_ABST
    Figure 2024164554000001_ABST
Patent Text Reader

Abstract

To estimate a level of depression of a subject from the utterance voice of the subject.SOLUTION: A depression level estimation device 100 comprises: an acoustic feature amount extraction unit 10 which extracts an acoustic feature amount from the utterance voice of a subject 200; and a machine-learning machine 20 which has previously learned the acoustic feature amount extracted from the utterance voices of multiple speakers as input data and a depression level based on self-evaluation of the multiple speakers as output data, and which outputs an estimation value of the depression level of the subject 200 by receiving the acoustic feature amount extracted from the utterance voice of the subject 200.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a technique for estimating a depression level of a subject. [Background technology]

[0002] Depression is easier to improve if it is discovered early and treatment is started as soon as possible. For this reason, if you have symptoms of depression, it is recommended that you visit a hospital. However, there are cases where you cannot go even if you want to for reasons such as "my symptoms of depression are severe," "I don't know which hospital to go to," "I'm afraid of what others will think," or "I'm afraid the doctor will get angry with me." In addition, the diagnostic criteria for depression are determined by the patient's own judgment of whether or not the conditions apply to them, and a subjective diagnosis is made, so there are cases where patients are not satisfied with the diagnosis and refuse to be prescribed medication.

[0003] In response to such circumstances, technologies have been developed to use spoken voice to determine whether a person is depressed or not and their mental state (see, for example, Non-Patent Documents 1 and 2). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Mune Mirai, Takebayashi Yoshitake, Sekizawa Yoichi, Shimoji Takaaki: To what extent can depression be diagnosed using only "voice"? ~ Verification of the accuracy of depression screening function applying ensemble type machine learning model to voice emotion recognition technology, RIETI Discussion Paper Series 16-J-054, September 2016 [Non-Patent Document 2] Hiroshi Otsuka, 24th PST's Technology for Analyzing Pathological Conditions from Voice, Pharmacia, 2021, Vol. 57, No. 7, pp. 662-664, July 1, 2021 Summary of the Invention [Problem to be solved by the invention]

[0005] Although conventional technology has made it possible to diagnose depression with a high degree of accuracy using data obtained from voice recordings, it is not possible to estimate the specific level of depression.

[0006] In view of the above problems, an object of the present invention is to estimate the degree of depression of a subject from the subject's speech voice. [Means for solving the problem]

[0007] According to one aspect of the present invention, there is provided an apparatus for estimating a depression level of a subject from the speech of the subject, the depression level estimation apparatus comprising: an acoustic feature extraction unit that extracts acoustic features from the speech of the subject; and a machine learning device that has been trained in advance using acoustic features extracted from the speech of a plurality of speakers as input data and self-assessed depression levels of the plurality of speakers as output data, the machine learning device receiving the acoustic features extracted from the speech of the subject and outputting an estimate of the depression level of the subject.

[0008] According to another aspect of the present invention, there is provided a method for estimating a depression level of a subject from the speech of the subject, the method comprising: a step of an acoustic feature extraction unit extracting acoustic features from the speech of the subject; and a step of a machine learning device that has been trained in advance with the acoustic features extracted from the speech of a plurality of speakers as input data and the self-assessed depression levels of the plurality of speakers as output data, receiving the acoustic features extracted from the speech of the subject, and outputting an estimate of the depression level of the subject. Effect of the Invention

[0009] According to the present invention, the depression level of a subject can be estimated from the speech of the subject. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram of a depression level estimation device according to an embodiment of the present invention. [Diagram 2]This is a table listing depression status and appropriate treatments according to BDI scores. [Diagram 3] 1 is a table showing acoustic features related to MDD among the acoustic features extracted by Surfboard. [Figure 4A] This is a list of waveforms that can be measured using the eGeMAPSv02 feature set of OpenSMILE. [Figure 4B] This is a list of static features that can be calculated for waveforms measurable with the OpenSMILE eGeMAPSv02 feature set. [Diagram 5] 1 is a list of acoustic features extracted by Surfboard that are expected to improve the accuracy of depression level estimation. [Figure 6] This is a list of acoustic features extracted by the OpenSMILE eGeMAPSv02 feature set that are expected to improve the accuracy of depression level estimation. [Figure 7] This is a list of the lines that were read by the participants in the experiment. [Figure 8] This is a scatter plot of SVR machine learning results using leave-one-out cross-validation for each feature set. [Figure 9] This is a scatter plot of the LightGBM machine learning results using leave-one-out cross-validation for each feature set. [Figure 10] This is a scatter plot of SVR machine learning results using leave-one-person-out cross-validation for each feature set. [Figure 11] This is a scatter plot of the LightGBM machine learning results using leave-one-person-out cross-validation for each feature set. [Figure 12] 12 is a table summarizing the machine learning results shown in FIGS. 8 to 11. [Figure 13] This table summarizes the experimental results of LightGBM machine learning using leave-one-person-out cross-validation, excluding data from people with extremely high or low self-rated depression levels. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings as appropriate. However, more detailed explanations than necessary may be omitted. For example, detailed explanations of already well-known matters or duplicate explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the inventor provides the accompanying drawings and the following explanation so that those skilled in the art can fully understand the present invention, and does not intend to limit the subject matter described in the claims by them. In addition, the dimensions of each member depicted in the drawings, detailed shapes of details, etc. may differ from the actual ones.

[0012] (Embodiment) 1 is a block diagram of a depression degree estimation device according to one embodiment of the present invention. The depression degree estimation device 100 according to this embodiment estimates the depression degree of a subject 200 from the speech of the subject 200.

[0013] There are various indices to indicate the degree of depression, but in this embodiment, a score based on the Beck Depression Inventory (BDI) is used. The BDI was invented by Dr. Aaron T. Beck, an American who advocated cognitive behavioral therapy, and is one of the self-assessment sheets that objectively measures the degree of depression. By taking the BDI test regularly, it becomes possible to measure the tendency of one's own mood as a numerical value. The BDI allows one to look at oneself objectively, and is used as one way of judging depression.

[0014] To use the BDI, read each item carefully and choose the answer that best describes how you have been feeling over the past few days. Each item has a score ranging from 0 to 3. The total score obtained by answering all 21 items represents the subject's current level of depression. Appropriate treatment is required depending on the level of depression. Figure 2 shows a table listing the state of depression and appropriate treatment for each BDI score.

[0015] In this way, while the BDI score is normally calculated through a test in which the subject answers questions for each item, the depression level estimation device 100 can estimate the BDI score directly from the speech of the subject 200 without such a test.

[0016] Returning to FIG. 1, the depression level estimation device 100 includes an acoustic feature extraction unit 10 and a machine learning device 20. Each of these components can be configured as hardware or software. When configured as software, a processor of a general-purpose computer or server device can function as each of the above components by loading and executing a computer program or a learned parameter or a machine learning model for implementing the above components. In addition, each of the above components does not need to be concentrated in one device, and may be distributed and connected to each other by a communication line.

[0017] The acoustic feature extraction unit 10 receives the speech of the subject 200 and extracts acoustic features from the speech signal. The speech of the subject 200 may be collected by a microphone (not shown) and the speech signal may be input to the acoustic feature extraction unit 10 in real time, or the recorded speech of the subject 200 may be read out as needed and input to the acoustic feature extraction unit 10.

[0018] There are various acoustic analysis tools available for extracting acoustic features from a speech signal. Among them, the open source Surfboard and openSMILE are used for the acoustic feature extraction unit 10 in this embodiment.

[0019] Surfboard is a voice feature extraction library released by Novoic, and is an open source Python library originally developed for application in the medical field. Surfboard handles acoustic features for capturing voice changes related to various diseases, and also extracts acoustic features that have been proven to be clinically relevant to MDD (major depressive disorder). In this embodiment, acoustic features (29 types, for example) that have been proven to be clinically relevant to MDD are used. Figure 3 is a list of acoustic features related to MDD among the acoustic features extracted by Surfboard. In the table, ↑ indicates that the feature is increased compared to the voice of a healthy person, ↓ indicates that the feature is decreased compared to the voice of a healthy person, and ⇔ indicates that there is a difference in the feature but it is unclear how it changes.

[0020] On the other hand, for openSMILE, in this embodiment, acoustic features related to speech, as an example, eGeMAPSv02 feature set is used. FIG. 4A is a list of waveforms that can be measured with the eGeMAPSv02 feature set of OpenSMILE. FIG. 4B is a list of static features that can be calculated for waveforms that can be measured with the eGeMAPSv02 feature set of OpenSMILE. Of the waveforms in the table of FIG. 4A, for F0semitoneFrom27.5Hz and loudness, all 10 types of static features in the table of FIG. 4B are calculated, and for the rest, two types of static features, the arithmetic mean and the coefficient of variation, or only the arithmetic mean are calculated, for a total of 88 types of static features.

[0021] Furthermore, the acoustic features may be narrowed down to those that are expected to improve the accuracy of depression level estimation for each of Surfboard and openSMILE. FIG. 5 is a list of acoustic features that are expected to improve the accuracy of depression level estimation among the acoustic features extracted by Surfboard. As an example, there are eight types of acoustic features. FIG. 6 is a list of acoustic features that are expected to improve the accuracy of depression level estimation among the acoustic features extracted by the eGeMAPSv02 feature set of OpenSMILE. As an example, there are three types of acoustic features. The idea of ​​narrowing down the acoustic features will be described later.

[0022] 1, the machine learning device 20 is trained in advance using acoustic features extracted from the speech of multiple speakers as input data (explanatory variables) and the self-assessed depression levels of the multiple speakers as output data (target variables). That is, the machine learning device 20 trained in advance outputs an estimate of the depression level of the subject 200 by inputting acoustic features extracted from the speech of the subject 200, whose depression level is unknown.

[0023] There are various types of machine learning machines, but in this embodiment, SVR (Support Vector Regression) and LightGBM (Light Gradient Boosting Machine) are used as the machine learning machine 20.

[0024] SVR is an extension of SVM (Support Vector Machine), which is mainly used for two-class classification, to regression analysis. SVM is a learning machine proposed within the framework of statistical learning theory. SVM configures a two-class pattern classifier with linear input elements. Learning involves learning the parameters of the linear input elements based on the criterion of margin maximization, which seeks a separating hyperplane that maximizes the distance (margin) between each data point and the training sample. SVR considers a nonlinear mapping to a feature space consisting of the input, and performs linear regression in the feature space after mapping. If the output does not fall within a set range called the ε-tube, a penalty is imposed according to the slack variable ξ, which represents the distance that it is out of range, and the optimal regression coefficient is calculated to minimize this value.

[0025] LightGBM is a machine learning method using a decision tree that uses GBDT (Gradient Boosting Decision Tree), and is characterized by its "light" (fast) nature. In the case of gradient boosting, the amount of calculations increases according to the amount of data in order to find division elements and criteria that minimize error. In contrast, LightGBM is characterized by its ability to learn quickly without compromising the accuracy of each decision tree.

[0026] (Example) Next, an embodiment of the depression degree estimation device 100, particularly the pre-learning of the machine learning device 20, will be described. The training data was collected as follows. A total of 15 participants, including 11 people in their 20s (8 men and 3 women), 2 people in their 50s (1 man and 2 women), and 2 people in their 70s (1 man and 2 women), were asked to read 10 lines (extracted from the Mirai Health Handbook developed and supervised by Okeous Co., Ltd. and Hiroshima University) in the same order in succession, and the audio was recorded. Immediately after that, the participants were asked to answer the Beck Depression Questionnaire, and their BDI scores were calculated. Figure 7 is a list of the lines read by the participants. Since all of the lines were positive, if they felt negative emotions despite reading such lines aloud, it can be inferred that there may be some problem in their minds.

[0027] The recorded voice was input to the acoustic feature extraction unit 10, and 29 types of acoustic features from Surfboard, 9 types of refined acoustic features, 88 types of acoustic features from openSMILE's eGeMAPSv02 feature set, and 3 types of refined acoustic features were extracted, totaling 4 patterns of feature sets. Two types of machine learning machines 20, SVR and LightGBM, were used. Then, a total of 8 types of machine learning machines 20 were constructed by machine learning using each feature set as input data and the BDI scores self-assessed by the participants as output data. Two types of validation, leave-one-out cross validation and leave-one-person-out cross validation, were performed on the machine learning machines 20 constructed in this way.

[0028] Here, we will explain how to narrow down the acoustic features. Many acoustic features are extracted by Surfboard or openSMILE, but not all of them are correlated with depression level. Some of them are considered to have a particularly high correlation with depression level. By narrowing down to such acoustic features, the accuracy of depression level estimation is expected to improve. Therefore, the importance of each acoustic feature is calculated by LightGBM, and only acoustic features whose importance is equal to or greater than a certain value in both leave-one-out and leave-one-person-out cross-validation are narrowed down. To calculate the importance of acoustic features, gain, expressed by formula (1), is used, which indicates the degree to which a certain feature contributed to the improvement of the objective function in the entire machine learning model.

number

[0029] In addition, G j and H j I j is calculated as the set of data assigned to the j-th leaf using equations (2) and (3).

number

[0030] g i , h i can be calculated using equations (4) and (5). i is the prediction at some step number t, and l is a differentiable loss function.

number

[0031] Based on the above idea, we narrowed down the 29 types of acoustic features from Surfboard and the 88 types of acoustic features from openSMILE's eGeMAPSv02 feature set, and the results are shown in Figures 5 and 6.

[0032] FIG. 8 is a scatter plot of SVR machine learning results by leave-one-out cross-validation for each feature set. FIG. 9 is a scatter plot of LightGBM machine learning results by leave-one-out cross-validation for each feature set. FIG. 10 is a scatter plot of SVR machine learning results by leave-one-person-out cross-validation for each feature set. FIG. 11 is a scatter plot of LightGBM machine learning results by leave-one-person-out cross-validation for each feature set. FIG. 12 is a table summarizing the machine learning results shown in FIGS. 8 to 11.

[0033] In leave-one-out cross-validation, LightGBM obtained estimates with high correlation with the correct answer for all feature sets. On the other hand, in leave-one-person-out cross-validation, the correlation was highest when LightGBM was trained with the Surfboard refinement feature, but the correlation tended to be weak overall.

[0034] Looking at the LightGBM machine learning results by leave-one-person-out cross validation shown in FIG. 11, it seems that there is a strong positive correlation when excluding the data of people with self-assessed depression levels of 1, 3, 18, and 19. Therefore, leave-one-person-out cross validation was performed excluding the data of these people. FIG. 13 is a table summarizing the experimental results of the LightGBM machine learning results by leave-one-person-out cross validation excluding the data of people with extremely high or low self-assessed depression levels. When the data of people with extremely high and low self-assessed depression levels were excluded, the correlation coefficient was 0.87 and the coefficient of determination was 0.757. This shows that the depression level estimation device 100 according to this embodiment can sufficiently estimate up to a moderate depression level for unknown subjects.

[0035] <<Variations>> In the above embodiment, the BDI score is used as the depression level, but another index may be used.

[0036] In addition, in the above embodiment, the LightGBM is adopted for the machine learning device 20 in consideration of the calculation load, but other gradient boosting decision trees such as XGBoost may also be used.

[0037] As described above, the embodiment has been described as an example of the technology of the present invention. For this purpose, the attached drawings and detailed description have been provided. Therefore, among the components described in the attached drawings and detailed description, not only components essential for solving the problem but also components that are not essential for solving the problem in order to illustrate the above technology may be included. Therefore, just because those non-essential components are described in the attached drawings or detailed description, it should not be immediately recognized that those non-essential components are essential. In addition, since the above-mentioned embodiment is intended to illustrate the technology of the present invention, various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents. [Explanation of symbols]

[0038] 100 Depression Level Estimation Device 10. Acoustic feature extraction unit 20 Machine Learning Machines

Claims

1. An apparatus for estimating a depression level of a subject from a speech voice of the subject, comprising: an acoustic feature extraction unit that extracts acoustic features from the speech of the subject; a machine learning device that has been trained in advance using acoustic features extracted from speech sounds of a plurality of speakers as input data and self-assessed depression levels of the plurality of speakers as output data, the machine learning device receiving the acoustic features extracted from the speech sounds of the subject and outputting an estimate of the depression level of the subject; A depression degree estimation device equipped with the above.

2. the machine learning machine is configured with a gradient boosting decision tree, and has been trained in advance using acoustic features that are recognized to be associated with major depressive disorder among acoustic features obtained by using Surfboard for the speech sounds of the multiple speakers as input data, and self-assessed depression levels of the multiple speakers as output data; The acoustic feature extraction unit extracts acoustic features that are recognized to be associated with major depressive disorder from acoustic features obtained from the speech of the subject using Surfboard. The depression level estimation device according to claim 1 .

3. The machine learning machine has learned in advance as input data acoustic features narrowed down to the first 8 components of the Mel-frequency cepstrum coefficients divided into 13 components and a harmonic noise ratio among the acoustic features obtained using the Surfboard, and as output data self-assessed depression levels of the multiple speakers, the acoustic feature extraction unit extracts acoustic features narrowed down to the first 8 components and harmonic noise ratio of Mel-frequency cepstrum coefficients divided into 13 from the acoustic features obtained using the Surfboard; The depression level estimation device according to claim 2 .

4. the machine learning machine is configured with a gradient boosting decision tree, and has been trained in advance using speech-related acoustic features among acoustic features obtained by using openSMILE for the speech of the multiple speakers as input data and self-assessed depression levels of the multiple speakers as output data; The acoustic feature extraction unit extracts acoustic features related to speech from among acoustic features obtained from the speech of the subject using openSMILE. The depression level estimation device according to claim 1 .

5. The machine learning machine has learned in advance, as input data, acoustic features obtained by using the openSMILE, which are narrowed down to the arithmetic mean of the fundamental frequency based on 27.5 Hz, the range between the 20th and 80th percentiles of the acoustic energy ratio of the harmonic components and the noise components, and the coefficient of variation of the center frequency of the third formant, and the self-assessed depression levels of the multiple speakers are used as output data. The acoustic feature extraction unit extracts acoustic features narrowed down to the arithmetic mean of the fundamental frequency based on 27.5 Hz, the range between the 20th and 80th percentiles of the acoustic energy ratio of the harmonic components and the noise components, and the coefficient of variation of the center frequency of the third formant from the acoustic features obtained using the openSMILE. The depression level estimation device according to claim 4.

6. 6. The depression degree estimation device according to claim 1, wherein the depression degree is a score based on the Beck Depression Profile Scale.

7. A method for estimating a depression level of a subject from a speech voice of the subject, comprising: An acoustic feature extraction unit extracts acoustic features from a speech of a subject; a machine learning device that has been trained in advance using acoustic features extracted from speech sounds of a plurality of speakers as input data and self-assessed depression levels of the plurality of speakers as output data, receives the acoustic features extracted from the speech sounds of the subject, and outputs an estimate of the depression level of the subject; A method for estimating the degree of depression.