Forecasting system, forecasting method, and forecasting program

Abstract

To provide a forecasting system, a forecasting method, and a forecasting program for grasping a cognition state on the basis of utterance of a forecasting object person.SOLUTION: A support device 20 includes a control part 21 for performing machine learning for utterance data on a learning object person. The control part 21 generates a plurality of utterance blocks in which an utterance section of the utterance data on the learning object person is divided into time intervals, and calculates a spectrogram for each utterance block. Furthermore, the control part 21 calculates a feature amount of a power spectrum distribution shape for each spectrogram. The control part 21 then learns a data set that associates the feature amount with a cognition state label based on dementia diagnosis information as teacher information, and generates a forecasting model. When the utterance data on the forecasting object person is acquired, the control part forecasts the cognition state label from the feature amount of the utterance data on the forecasting object person using the forecasting model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a prediction system, a prediction method, and a prediction program for predicting the cognitive state of a subject. [Background technology]

[0002] When dementia develops, cognitive functions such as memory, judgment, comprehension, calculation, and language ability decline. For this reason, it is effective to detect mild cognitive impairment (MCI), which is a precursor state of dementia, early on. Therefore, a dementia testing system has also been considered (see, for example, Patent Document 1). The dementia testing system described in this document includes an information transmission unit that transmits information to the subject by acting on at least one of the subject's vision and hearing, and a measurement unit that measures the subject's response. Then, a plurality of different feature amounts are calculated based on the measurement results of the measurement unit, and whether or not the subject has dementia is determined based on the feature amounts. Here, whether or not the subject has dementia is determined based on at least one acoustic feature amount that is an acoustic feature amount of the voice uttered by the subject, and at least one linguistic feature amount that is a linguistic feature amount of the voice uttered by the subject. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-15139 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, it may not be possible to make an accurate judgment based on acoustic features alone, and as described in the patent literature, it may be difficult to evaluate linguistic features. [Means for solving the problem]

[0005] The prediction system for solving the above problem includes a control unit that performs machine learning of the speech data of a learning subject. The control unit generates a plurality of speech blocks by dividing the speech section of the speech data of the learning subject into time intervals, calculates a spectrogram for each of the speech blocks, calculates a feature of the distribution shape of the power spectrum for each of the spectrograms, learns a data set that associates the feature with a cognitive status label based on dementia diagnosis information as teacher information to generate a prediction model, and when the speech data of the prediction subject is acquired, predicts the cognitive status label from the feature of the speech data of the prediction subject using the prediction model. Effect of the Invention

[0006] According to the present disclosure, it is possible to efficiently grasp the cognitive state of a person to be predicted based on the speech of the person to be predicted. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram of a prediction system according to an embodiment. [Diagram 2] FIG. 2 is an explanatory diagram of a hardware configuration of the embodiment. [Diagram 3] FIG. 2 is an explanatory diagram of a processing procedure according to an embodiment. [Figure 4] 1A to 1E are explanatory diagrams of a processing procedure according to an embodiment, in which (a) is voice data, (b) is a vocalization section, (c) is a vocalization block, (d) is a spectrogram, and (e) is a power spectrum. [Diagram 5] 1A and 1B are explanatory diagrams of evaluation results of an embodiment, in which (a) is an explanatory diagram for the case of seven variables, (b) is an explanatory diagram for the case of six variables, and (c) is an explanatory diagram for the case of five variables. [Figure 6] 5A and 5B are diagrams illustrating the evaluation results of the embodiment, where (a) is an explanatory diagram for four variables and (b) is an explanatory diagram for three variables. [Figure 7] 5A and 5B are diagrams illustrating the evaluation results of the embodiment, where (a) is an explanatory diagram for the case of two variables, and (b) is an explanatory diagram for the case of one variable. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment embodying a prediction system, a prediction method, and a prediction program will be described with reference to FIGS. As shown in FIG. 1, in this embodiment, a user terminal 10 and a support device 20 connected to each other via a network are used to predict a cognitive state.

[0009] (Hardware configuration example) FIG. 2 shows an example of the hardware configuration of an information processing device H10 that functions as the user terminal 10 and the support device 20. As shown in FIG.

[0010] The information processing device H10 includes a communication device H11, an input device H12, a display device H13, a storage device H14, and a processor H 15. Note that this hardware configuration is an example, and the information processing device H10 may include other hardware.

[0011] The communication device H11 is an interface that establishes a communication path with another device and transmits and receives data, and is, for example, a network interface or a wireless interface.

[0012] The input device H12 is a device that accepts input from a user, etc., and is, for example, a mouse, a keyboard, etc. The display device H13 is, for example, a display or a touch panel that displays various information.

[0013] The storage device H14 is a storage device that stores data and various programs for executing various functions of the user terminal 10 and the support device 20. Examples of the storage device H14 include a ROM, a RAM, and a hard disk.

[0014] The processor H15 uses programs and data stored in the storage device H14 to control each process (for example, a process in the control unit 21 described later) in the user terminal 10 and the support device 20. Examples of the processor H15 include a CPU and an MPU. The processor H15 loads a program stored in a ROM or the like into a RAM and executes various processes corresponding to the various processes. For example, when an application program in the user terminal 10 or the support device 20 is started, the processor H15 operates a process that executes each process described later.

[0015] The processor H15 is not limited to a processor that performs software processing for all the processes it executes. For example, the processor H15 may include a dedicated hardware circuit (e.g., an application specific integrated circuit (ASIC)) that performs hardware processing for at least a part of the processes it executes. That is, the processor H15 may be configured as follows:

[0016] [1] One or more processors that operate according to a computer program (software). [2] One or more dedicated hardware circuits that perform at least some of the various processes; or [3] Circuits that include combinations of these The processor includes a CPU and memory, such as RAM and ROM, that stores program codes or instructions configured to cause the CPU to execute processes. Memory, i.e., computer-readable media, includes any available media that can be accessed by a general-purpose or special-purpose computer.

[0017] (Functions of each information processing device) The functions of the user terminal 10 and the support device 20 will be described with reference to FIG. The user terminal 10 is a computer terminal used by a user who uses the present system. In this embodiment, the user terminal 10 is used by the user who predicts the cognitive state of the prediction target person.

[0018] The support device 20 is a computer system that predicts the cognitive state of a prediction subject. The support device 20 includes a control unit 21, a voice information storage unit 22, a teacher information storage unit 23, and a learning result storage unit 24.

[0019] The control unit 21 performs processes (including a management stage, a learning stage, a prediction stage, etc.) described below. By executing a prediction program for this purpose, the control unit 21 functions as a management unit 211, a learning unit 212, a prediction unit 213, etc.

[0020] The management unit 211 executes a process for acquiring various types of information. The learning unit 212 executes a process of generating a prediction model for predicting the cognitive state by machine learning. As a machine learning method, a classifier capable of binary classification, for example, logistic regression or SVM (Support Vector Machine) can be used. The prediction unit 213 executes a process of predicting a cognitive state by using a prediction model.

[0021] The voice information storage unit 22 records voice information of the learning subjects (people with dementia and people without dementia). This voice information is recorded when a voice sample (utterance data) of the learning subject is acquired. The voice information records data on the voice sample and the cognitive state label for each learning subject. The voice sample is audio data obtained by recording the speech of the subject with a microphone. The cognitive status label is identification information for identifying whether the person has dementia or not based on the dementia diagnosis information. For example, it is a correct answer label at the time of learning in which dementia is set to "1" and non-dementia is set to "0".

[0022] The teacher information storage unit 23 records teacher information used for learning. This teacher information is recorded when a teacher information creation process is performed. The teacher information includes information on the speech features of the learning subject and the recognition situation label. In this embodiment, the following variables are used as the features of the speech data.

[0023] [Variable 1] Average of the centers of gravity of the power spectrum [Variable 2] Average skewness of the power spectrum (an index showing the asymmetry of the distribution) [Variable 3] Average kurtosis of the power spectrum (an index showing the sharpness of the distribution) [Variable 4] Standard deviation of the center of gravity of the power spectrum [Variable 5] Standard deviation of power spectrum skewness [Variable 6] Standard deviation of the kurtosis of the power spectrum [Variable 7] Standard deviation of spectrogram intensity

[0024] The learning result storage unit 24 records information for predicting the cognitive state. This prediction model is recorded when the prediction model is generated by executing the learning process. The prediction model is a model that predicts whether or not there is dementia for a combination of variable values ​​of [variable 1] to [variable 7] obtained from the speech data. In this embodiment, the variable values ​​of [variable 1] to [variable 7] are combined, but information combining at least one or more variables can be used. Furthermore, a threshold value for discriminating between dementia and a cognitive state other than dementia for the probability value of dementia or not from the prediction model is recorded.

[0025] (Learning process) Next, the learning process will be described with reference to FIG. First, the control unit 21 of the support device 20 executes a process of creating teacher information (step S11). Specifically, the management unit 211 of the control unit 21 acquires a voice sample for a series of utterances from the voice information storage unit 22. Next, the management unit 211 extracts a speech section of the learning subject from the voice sample.

[0026] As shown in FIG. 4( a ), the management unit 211 acquires a voice sample 500 . In this case, as shown in FIG. 4(b), the management unit 211 generates voice data 501 by extracting and combining vocalization sections of voice samples 500.

[0027] Next, as shown in Fig. 4(c), the management unit 211 divides the voice data 501 into a plurality of timings at fixed time intervals. Here, a plurality of voice blocks 502 are generated by cutting out the voice data 501 at a predetermined interval so as to overlap. Note that adjacent voice blocks 502 are made to partially overlap each other.

[0028] 4(d), the management unit 211 calculates a spectrogram 503 for each utterance block. The spectrogram 503 indicates the time dependency of the frequency of the utterance block 502.

[0029] Then, as shown in FIG. 4(e), management unit 211 calculates power spectrum 504 of each spectrogram 503 at a given time t0. Then, the management unit 211 calculates the variable values ​​of [variable 1] to [variable 7] as the feature quantities of the distribution shape.

[0030] [Variable 1] As for the average of the center of gravity of the power spectrum, the frequency of the center of gravity of the power spectrum of each utterance block 502 is calculated, and the average value thereof is calculated. [Variable 2] The average distortion of the power spectrum is calculated by calculating the distortion of the shape of the power spectrum of each utterance block 502 and calculating the average value.

[0031] [Variable 3] The average kurtosis of the power spectrum is calculated by calculating the kurtosis of the shape of the power spectrum of each speech block 502 and calculating the average value. [Variable 4] Regarding the standard deviation of the center of gravity of the power spectrum, the standard deviation of the frequency of each center of gravity of each utterance block 502 is calculated.

[0032] [Variable 5] Regarding the standard deviation of the distortion of the power spectrum, the standard deviation of the distortion of each utterance block 502 is calculated. [Variable 6] Regarding the standard deviation of the kurtosis of the power spectrum, the standard deviation of the kurtosis of each utterance block 502 is calculated.

[0033] [Variable 7] Regarding the standard deviation of the spectrogram intensity, the standard deviation of the intensity of each vocalization block 502 is calculated. Then, the management unit 211 records, in the teacher information storage unit 23, a data set in which the recognition situation labels are recorded for the variable values ​​of [variable 1] to [variable 7] for each vocalization block as teacher information.

[0034] Next, the control unit 21 of the support device 20 executes a learning process (step S12). Specifically, the learning unit 212 of the control unit 21 performs machine learning using the teacher information recorded in the teacher information storage unit 23. Here, a prediction model is generated that predicts a cognitive state label from a combination of all the variable values ​​of [variable 1] to [variable 7]. Then, the learning unit 212 calculates a threshold value for discriminating between dementia and any cognitive state other than dementia for the predicted value obtained by inputting all the variable values ​​of [variable 1] to [variable 7] into the prediction model. Then, the learning unit 212 records the generated prediction model and threshold value in the learning result storage unit 24.

[0035] (Prediction processing) Next, the prediction process will be described with reference to FIG. First, the control unit 21 of the support device 20 executes a process of acquiring predicted target person data (step S13). Specifically, the management unit 211 of the control unit 21 outputs a guide screen to the user terminal 10. This guide screen displays a question for the predicted target person. In this case, the predicted target person answers the question by speaking. Then, the management unit 211 acquires voice data of the predicted target person (predicted target person data) from the user terminal 10.

[0036] In this case, the management unit 211 extracts the speech interval of the prediction target person from the prediction target person data. Furthermore, the management unit 211 generates speech blocks divided at regular time intervals. Next, the management unit 211 calculates a spectrogram for each speech block. Then, the management unit 211 extracts the feature amount of the speech data by calculating the variable values ​​of [variable 1] to [variable 7] for each speech block.

[0037] Next, the control unit 21 of the support device 20 executes a prediction process (step S14). Specifically, the prediction unit 213 of the control unit 21 inputs the variable values ​​of [variable 1] to [variable 7] of each utterance block to the prediction model recorded in the learning result storage unit 24, thereby calculating a prediction value of whether or not there is dementia for each utterance block. Next, the prediction unit 213 calculates a statistical value (e.g., an average value) of the predicted value for each utterance block, and compares it with a threshold value recorded in the learning result storage unit 24 to predict the cognitive status. Then, the prediction unit 213 outputs the comparison result with the threshold value to the user terminal 10.

[0038] According to this embodiment, the following effects can be obtained. (1) In this embodiment, the control unit 21 of the support device 20 executes a process of creating teacher information (step S11). This makes it possible to calculate multiple types of variable values ​​for the voice of a dementia patient or a non-dementia patient.

[0039] (2) In this embodiment, the control unit 21 of the support device 20 executes a learning process (step S12). This allows a prediction model for predicting dementia or other than dementia for multiple types of variable values ​​in the voice to be generated. Furthermore, a threshold for identifying either dementia or other cognitive status from multiple predicted values ​​can be calculated.

[0040] (3) In this embodiment, the control unit 21 of the support device 20 executes a process of acquiring data of the person to be predicted (step S13). This makes it possible to calculate multiple types of variable values ​​for the voice of the person to be predicted.

[0041] (4) In this embodiment, the control unit 21 of the support device 20 executes a prediction process (step S14). This makes it possible to predict the cognitive state of the person to be predicted, whether the cognitive state is dementia or not, from the voice of the person to be predicted.

[0042] Figures 5 to 7 show the AUC (Area Under the ROC Curve) when [Variable 1] to [Variable 7] are used. The closer the AUC value is to "1", the higher the discriminatory ability. Figure 5(a) shows the evaluation result R7 when all variables [Variable 1] to [Variable 7] are used. Figure 5(b) shows the evaluation result R6 for combinations arranged in order of highest AUC when six variables from [Variable 1] to [Variable 7] are used. Although it depends on how the variables are selected, there is no significant change in AUC.

[0043] Figure 5(c) shows the evaluation result R5 for 10 combinations arranged in order of highest AUC when using five variables from [Variable 1] to [Variable 7]. There is no significant change in AUC, but combinations with high AUC include [Variable 1] and [Variable 7].

[0044] Figure 6(a) shows the evaluation result R4 for 10 combinations arranged in order of AUC when using four variables from [Variable 1] to [Variable 7]. Again, the combinations with high AUC include [Variable 1] and [Variable 7].

[0045] Figure 6(b) shows the evaluation result R3 for the 10 combinations arranged in descending order of AUC when three variables are used from [Variable 1] to [Variable 7]. When three variables are used, a slight decrease in AUC is observed.

[0046] Figure 7(a) shows the evaluation result R2 for the 10 combinations arranged in order of AUC when two variables are used from [variable 1] to [variable 7]. Figure 7(b) shows the evaluation result R1 when one variable is used from [variable 1] to [variable 7]. In both cases, a decrease in AUC is observed. In this sense, using multiple variables improves the AUC.

[0047] This embodiment can be modified as follows: This embodiment, the following embodiment, and the following modified examples can be combined with each other to the extent that there is no technical contradiction.

[0048] In the above embodiment, the user terminal 10 and the support device 20 are used, but the hardware configuration is not limited to these. The user terminal 10 and the support device 20 may be realized by a single device. In addition, the function of the support device 20 may be provided by a cloud service.

[0049] In the above embodiment, the variable values ​​of [Variable 1] to [Variable 7] are used as the feature quantities of the voice. It is not necessary to use all the elements, and some elements or other elements may be used. Any statistical value indicating the shape of the power spectrum will suffice. Also, instead of the average value, [Variable 1] to [Variable 3], and instead of the average value, [Variable 4] to [Variable 6], statistical values ​​indicating the variance of the shape of the power spectrum may be used. For example, variance, area, number of peaks, length of line, etc. are used for [Variable 1] to [Variable 7].

[0050] In the above embodiment, the control unit 21 of the support device 20 executes a process of acquiring predicted target data (step S13). Here, the management unit 211 of the control unit 21 outputs a guide screen to the user terminal 10. Instead of this, a doctor or the like may ask questions and acquire the voice of the predicted target. Also, the guide screen is not essential as long as the voice can be acquired. Also, the acquired speech content does not need to follow the guide, and may be acquired from natural conversation or the like. For example, questions from a human being such as a doctor, or a non-human being such as an avatar in a virtual space may be used. Also, conversations in daily life and telephone conversation voices may be used.

[0051] In the above embodiment, the recognition state is predicted using the features of the voice. In addition, user attributes such as gender and age may be used. For example, since the average of the center of gravity of the power spectrum of [variable 1] may depend on gender, learning and prediction processes may be performed using gender information acquired as a user attribute instead.

[0052] In addition, images of the target person such as the face, hands, and feet, and information such as natural language obtained by voice recognition of the spoken content may be used. In this case, a prediction model is generated using teacher information including the feature amount of the movement in a video of the target person and the feature amount of the text information obtained by voice recognition of the spoken content. Furthermore, for voice data, information other than the distribution shape obtained from voice, such as the duration and frequency of voiced intervals, the duration and frequency of unvoiced intervals, and the response time from a question to a response, may be used as variables.

[0053] In the above embodiment, the control unit 21 of the support device 20 executes a prediction process (step S14). Here, the prediction unit 213 calculates a statistical value (e.g., an average value) of the predicted value for each utterance block, and predicts the recognition status by comparing it with a threshold value recorded in the learning result storage unit 24. Here, the information is not limited to the statistical value as long as it allows the predicted value for each utterance block to be evaluated. For example, the recognition status may be predicted by a majority vote of the predicted values. Also, a predicted value may be obtained by performing ensemble learning using a plurality of different prediction models. In this case, for example, a plurality of prediction models with different combinations of variables 1 to 7 may be used. Alternatively, a plurality of prediction models with different machine learning methods may be used. When predicting the overall cognitive status from the predicted values ​​of each model, the cognitive status may be predicted not only by statistical values ​​but also by majority vote. [Explanation of symbols]

[0054] 10...user terminal, 20...support device, 21...control unit, 211...management unit, 212...learning unit, 213...prediction unit, 22...voice information storage unit, 23...teacher information storage unit, 24...learning result storage unit.

Claims

1. A prediction system including a control unit that performs machine learning on speech data of a learning subject, The control unit: A plurality of speech blocks are generated by dividing a speech section of the speech data of the learning subject into time intervals; Calculating a spectrogram for each of the speech blocks; Calculating a feature value of a distribution shape of a power spectrum for each of the spectrograms; A data set associating the feature amount with a cognitive status label based on dementia diagnosis information is used as training information to generate a prediction model; A prediction system characterized in that, when speech data of a person to be predicted is acquired, a cognitive state label is predicted from features of the speech data of the person to be predicted using the prediction model.

2. The prediction system according to claim 1 , wherein the control unit acquires the speech blocks at a plurality of timings in a series of utterances and calculates the spectrogram.

3. 3. The prediction system according to claim 1, wherein the feature quantity of the distribution shape includes a statistical value of the center of gravity of the power spectrum.

4. 3. The prediction system according to claim 1, wherein the feature quantity of the distribution shape includes a statistical value of the intensity of the power spectrum.

5. 3. The prediction system according to claim 1, wherein the feature quantity of the distribution shape includes a statistical value of the shape of the power spectrum.

6. 6. The prediction system according to claim 5, wherein the statistical values ​​of the shape of the power spectrum include at least one of the mean kurtosis, the standard deviation of kurtosis, the mean skewness, and the standard deviation of skewness.

7. A method for predicting a cognitive state by using a prediction system having a control unit that performs machine learning of speech data of a learning subject, comprising: The control unit: A plurality of speech blocks are generated by dividing a speech section of the speech data of the learning subject into time intervals; Calculating a spectrogram for each of the speech blocks; Calculating a feature value of a distribution shape of a power spectrum for each of the spectrograms; A data set associating the feature amount with a cognitive status label based on dementia diagnosis information is used as training information to generate a prediction model; A prediction method, characterized in that, when speech data of a person to be predicted is acquired, a cognitive state label is predicted from features of the speech data of the person to be predicted using the prediction model.

8. A program for predicting a cognitive state by using a prediction system having a control unit that performs machine learning of speech data of a learning subject, The control unit, A plurality of speech blocks are generated by dividing a speech section of the speech data of the learning subject into time intervals; Calculating a spectrogram for each of the speech blocks; Calculating a feature value of a distribution shape of a power spectrum for each of the spectrograms; A data set associating the feature amount with a cognitive status label based on dementia diagnosis information is used as training information to generate a prediction model; A prediction program characterized by functioning as a means for predicting a cognitive state label from features of the speech data of a person to be predicted using the prediction model when the speech data of the person to be predicted is acquired.

Images