Information processing device, information processing method, information processing system, and information processing program

Abstract

Provided is an information processing device that acquires speech data, which is time-series data of speech uttered by a user, and generates preprocessed speech data representing the data that, from out of the acquired speech data, is no earlier than a first time from the start point of the speech data and no later than a second time from the end of the speech data. Additionally, this information processing device generates processing result data by applying dynamic time warping to the preprocessed speech data that is generated. The information processing device calculates a score representing the degree to which the user has a predetermined disease or symptom on the basis of the generated processing result data, and estimates whether or not the user has the predetermined disease or symptom on the basis of the calculated score.

Description

[Technical Field]

[0001] The disclosed technologies relate to information processing devices, information processing methods, information processing systems, and information processing programs. [Background technology]

[0002] International Publication No. 2020 / 013296 discloses a device for estimating psychiatric or neurological disorders. This device calculates various acoustic parameters from the user's voice data and uses these acoustic parameters to estimate whether or not the user has a psychiatric or neurological disorder. [Overview of the project] [Problems that the invention aims to solve]

[0003] The device disclosed in the above-mentioned International Publication No. 2020 / 013296 estimates diseases using acoustic parameters calculated from voice data, but there is room for improvement in its accuracy.

[0004] The disclosed technology was developed in consideration of the above circumstances and provides an information processing device, information processing method, information processing system, and information processing program that can accurately estimate whether a user has a predetermined disease or symptom by applying a dynamic time stretching method to audio data, which is time-series data of voice uttered by a user. [Means for solving the problem]

[0005] To achieve the above objective, a first aspect of this disclosure is an information processing device comprising: an acquisition unit that acquires audio data which is time-series data of voice uttered by a user; a processing unit that generates pre-processed audio data from the audio data acquired by the acquisition unit, which is data from the first hour after the start point of the audio data and data from the second hour before the end point of the audio data; a generation unit that generates processing result data by applying a dynamic time warping method to the pre-processed audio data generated by the processing unit; a calculation unit that calculates a score representing the degree to which the user has a predetermined disease or symptom based on the processing result data generated by the generation unit; and an estimation unit that estimates whether or not the user has a predetermined disease or symptom based on the score calculated by the calculation unit.

[0006] A second aspect of this disclosure is an information processing method that causes a computer to perform the following steps: acquire audio data which is time-series data of voice uttered by a user; generate pre-processed audio data which represents data from the acquired audio data that is from the first hour after the start point of the audio data and from the second hour before the end point of the audio data; generate processing result data by applying Dynamic Time Warping to the generated pre-processed audio data; calculate a score representing the degree to which the user has a predetermined disease or symptom based on the generated processing result data; and estimate whether or not the user has a predetermined disease or symptom based on the calculated score.

[0007] A third aspect of this disclosure is an information processing program for causing a computer to perform the following processes: acquire audio data which is time-series data of voice uttered by a user; generate pre-processed audio data which represents data from the acquired audio data that is from the first hour after the start point of the audio data and from the second hour before the end point of the audio data; generate processing result data by applying Dynamic Time Warping to the generated pre-processed audio data; calculate a score representing the degree to which the user has a predetermined disease or symptom based on the generated processing result data; and estimate whether or not the user has a predetermined disease or symptom based on the calculated score. [Effects of the Invention]

[0008] According to the disclosed technology, by applying dynamic time stretching to audio data, which is time-series data of voice uttered by a user, it is possible to accurately estimate whether or not a user has a predetermined disease or predetermined symptoms. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the schematic configuration of the information processing system according to the first embodiment. [Figure 2] This is a diagram illustrating the outline of the first embodiment. [Figure 3] This diagram schematically shows audio data for a predetermined period. [Figure 4] This is a diagram illustrating the shifting process applied to audio data. [Figure 5] This is a diagram illustrating the sampling process for audio data. [Figure 6] This diagram schematically shows an example of how the information processing system of the first embodiment can be used. [Figure 7] This figure shows an example of a computer that makes up an information processing device. [Figure 8]It is a diagram showing an example of the processing executed by the information processing apparatus of the first embodiment. [Figure 9] It is a diagram for explaining the outline of the second embodiment. [Figure 10] It is a diagram schematically showing an example of the usage form of the information processing system of the second embodiment. [Figure 11] It is a diagram schematically showing an example of the usage form of the information processing system of the second embodiment. [Figure 12] It is a diagram of the experimental results which are examples. [Figure 13] It is a diagram of the experimental results which are examples. [Figure 14] It is a diagram of the experimental results which are examples. [Figure 15] It is a diagram of the experimental results which are examples. [Figure 16] It is a diagram of the experimental results which are examples. [Figure 17] It is a diagram of the experimental results which are examples. [Figure 18] It is a diagram of the experimental results which are examples. [Figure 19] It is a diagram of the experimental results which are examples. [Figure 20] It is a diagram of the experimental results which are examples.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the disclosed technology will be described in detail with reference to the drawings.

[0011] <Information Processing System of the First Embodiment>

[0012] FIG. shows an information processing system 10 according to the first embodiment. As shown in FIG. , the information processing system 10 of the first embodiment includes a microphone 12, an information processing apparatus 14, and a display device 16.

[0013] The information processing system 10 estimates whether the user has a predetermined disease or symptom (hereinafter simply referred to as "disease, etc.") based on the user's voice collected by the microphone 12. In this embodiment, the information processing system 10 estimates whether the user has a mental illness or neurological illness, or symptoms of mental disorder or cognitive impairment, as an example of a predetermined disease or symptom.

[0014] The information processing device 14 of the information processing system 10 in the first embodiment applies predetermined preprocessing to the voice data, which is time-series data of the voice spoken by the user, and generates preprocessed data. Then, the information processing device 14 determines whether or not the user has a disease or the like based on the result of applying Dynamic Time Warping to the preprocessed data.

[0015] In dynamic time stretching, the distance between one time series of data and another time series of data is calculated. In this embodiment, the processed result data obtained by dynamic time stretching is used to estimate whether or not the user has a disease or other condition.

[0016] The following provides a detailed explanation.

[0017] As shown in Figure 1, the information processing device 14 functionally comprises an acquisition unit 20, an audio data storage unit 22, a reference data storage unit 24, a processing unit 26, a generation unit 28, a calculation unit 30, an estimation unit 32, and an output unit 34. The information processing device 14 is implemented by a computer as described later.

[0018] The acquisition unit 20 acquires audio data, which is time-series data of the voice spoken by the user. The acquisition unit 20 then stores the audio data in the audio data storage unit 22.

[0019] The audio data storage unit 22 stores the audio data acquired by the acquisition unit 20.

[0020] The reference data storage unit 24 stores voice data of reference users whose status (whether or not they have a disease, etc.) is known.

[0021] The processing unit 26 reads the audio data stored in the audio data storage unit 22. Then, the processing unit 26 performs predetermined preprocessing on the audio data to generate preprocessed audio data. The method for generating preprocessed audio data will be described in detail below. Figure 2 shows a diagram illustrating preprocessed audio data.

[0022] (Extraction of the central part of the audio data)

[0023] When estimating whether a user has a disease or other condition based on voice data they have produced, it is preferable to use voice data in which the user's speech is stable.

[0024] In this regard, the initial portion of the time-series data represented by audio data is the time when the user began to speak, and therefore, it is often undesirable to use data from this point for estimating diseases or other conditions. For example, if a user suddenly starts speaking after being silent, it is expected that their voice may become hoarse or quiet due to instability in their vocalization. Even if such data is used to estimate diseases or other conditions, it is unlikely that accurate results will be obtained.

[0025] Furthermore, it is often undesirable to use the parts of the time-series data represented by audio data that are close to the end point for estimating diseases, etc. For example, if a user pronounces a long sound, it is possible that the user will run out of breath and be unable to sustain their voice, or that the end of a word will be pronounced ambiguously.

[0026] Therefore, the processing unit 26 of the information processing device 14 in this embodiment extracts the central portion of the data from the audio data, which is time-series data.

[0027] Specifically, as shown in Figure 2, the processing unit 26 generates data D2, which represents data from the audio data D1 from the start point of audio data D1 to the first time interval T1, and data from the end point of audio data D1 to the second time interval T2. Data D2 corresponds to the time interval T3 of audio data D1. This generates data for the central part where the user's speech is stable.

[0028] (Extraction of data for a predetermined period) Furthermore, the processing unit 26 extracts data for a predetermined period from the extracted central portion of the data. Figure 3 shows a diagram illustrating the data for the predetermined period. As shown in Figure 3, the audio data Df is time-series data, and there is repetition of a predetermined signal. For example, in the example shown in Figure 3, the same signal waveform is repeated for each time interval T.

[0029] As will be described later, when estimating whether or not a user has a disease, etc., the voice data emitted by the user whose disease, etc. is to be estimated may be compared with the voice data of a reference user whose disease, etc. is already known. For this reason, it is preferable that the predetermined period of data extracted from the voice data emitted by the user whose disease, etc. is to be estimated matches the predetermined period of data in the reference user's voice data. For this reason, for example, the processing unit 26 extracts data from the extracted central portion that is the same predetermined period as the period of the reference user's voice data. This predetermined period is, for example, set in advance. Alternatively, for example, the predetermined period may be changed depending on the type of data.

[0030] (Data extraction by shifting along the time axis) Next, the processing unit 26 shifts the extracted data for a predetermined period in the time axis direction. Figure 4 shows a diagram illustrating the shift of data in the time axis direction. As shown in Figure 4, the audio data Ds has a repeating signal with a period Ts, and the audio data D of the reference user Ref Period T RefConsider the case where there is a repetition of signals. In this case, as shown in FIG. 4, when the start portion P1 of the cut-out of the voice data Ds and the start portion P2 of the voice data D of the reference user Ref do not align, even if the voice data Ds and the voice data D of the reference user Ref are similar, the value representing the distance between the voice data D and the voice data D calculated by the dynamic time stretching method Ref may become large.

[0031] Therefore, the processing unit 26 shifts the extracted data for a predetermined period in the time axis direction. For example, the processing unit 26 shifts the data for a predetermined period shown in FIG. 4 by a predetermined time in the time axis direction represented by the arrow S. Note that, for example, the shift amount of this predetermined time is set in advance. Or, for example, the shift amount may be changed according to the type of data.

[0032] (Extraction of data by sampling at a predetermined sampling rate) Next, the processing unit 26 extracts sampling data obtained by sampling from the data that has been subjected to the shift processing in the time axis direction. As described above, in the present embodiment, when estimating whether a user has a disease or the like, the voice data uttered by the user for whom the disease or the like is to be estimated may be compared with the voice data of a reference user for whom it is known whether the user has a disease or the like. Therefore, it is preferable that the sampling rate for the voice data uttered by the user for whom the disease or the like is to be estimated is the same as the sampling rate for the voice data of the reference user.

[0033] For example, as shown in FIG. 5, consider the sampling data D A , D B obtained by sampling the voice data D. In this case, the sampling data D A generated by the sampling rate A and the sampling data D BWhen the distance between two points is calculated using the dynamic time stretching method, a value representing a predetermined distance is calculated even though the original audio data D is the same.

[0034] Therefore, for example, the processing unit 26 generates sampled data extracted at the same sampling rate as the audio data of the reference user. This sampling rate is set in advance. Alternatively, for example, the sampling rate may be changed depending on the type of data. For example, 200 sampling points are extracted per period of data from a predetermined period of data.

[0035] (Data stretching and contraction along the time axis) Next, the processing unit 26 performs a stretching and shrinking process in the time axis direction on the sampled data obtained by sampling from the audio data. As described above, in this embodiment, when estimating whether or not a user has a disease, etc., the audio data spoken by the user whose disease, etc. is to be estimated may be compared with the audio data of a reference user whose presence or absence of disease, etc. is already known.

[0036] Therefore, it is preferable that the time-axis intervals of the voice data emitted by the user whose disease or other condition is being estimated be the same as the time-axis intervals of the voice data of the reference user. For this reason, for example, the processing unit 26 performs a predetermined stretching and shrinking process in the time axis direction on the data D3 shown in Figure 2. The method of the predetermined stretching and shrinking process is set in advance. Alternatively, for example, the method of stretching and shrinking may be changed depending on the type of data.

[0037] (Data stretching in the amplitude direction) Next, the processing unit 26 performs an amplitude-direction stretching process on the data that has undergone a time-axis stretching process. As described above, in this embodiment, when estimating whether or not a user has a disease, etc., the voice data spoken by the user whose disease, etc. is to be estimated may be compared with the voice data of a reference user whose presence or absence of disease, etc. is already known.

[0038] Therefore, it is preferable that the amplitude of the voice data emitted by the user whose disease or other condition is being estimated matches the amplitude of the voice data of the reference user. For this reason, for example, the processing unit 26 performs a predetermined stretching / contracting process in the amplitude direction on the data D4 shown in Figure 2. The method of the predetermined stretching / contracting process is set in advance. Alternatively, for example, the method of stretching / contracting may be changed depending on the type of data.

[0039] The processing unit 26 generates pre-processed audio data by performing the above-described pre-processing steps on the audio data.

[0040] The generation unit 28 generates processed result data by applying dynamic time warping to the preprocessed audio data generated by the processing unit 26. The processed result data obtained by applying dynamic time warping is calculated as a distance matrix representing the distance between each point in one time series of data and each point in another time series of data.

[0041] Specifically, the generation unit 28 reads the reference user's voice data stored in the reference data storage unit 24. Then, as shown in Figure 2, the generation unit 28 processes the pre-processed voice data D5 and the reference user's voice data D Ref By applying dynamic time stretching to the preprocessed audio data D5 and the audio data D of the reference user, Ref This generates processing result data representing the distance between the two points. Note that the audio data of the reference user may also be subjected to the same preprocessing described above.

[0042] The generation unit 28 may generate processing result data using only pre-processed audio data. For example, processing result data representing the distance between the first audio data and the second audio data may be generated by applying a dynamic time stretching method to first audio data representing data in a first time interval within the pre-processed audio data and second audio data representing data in a second time interval within the pre-processed audio data.

[0043] More specifically, as shown in Figure 2, the generation unit 28 generates processed result data representing the distance between the first audio data D5-1 and the second audio data D5-2 by applying a dynamic time stretching method to the first audio data D5-1, which represents the data in a first time interval within the preprocessed audio data D5, and the second audio data D5-1, which represents the data in a second time interval within the preprocessed audio data D5.

[0044] Next, as shown in Figure 2, the generation unit 28 applies a dynamic time stretching method to the second audio data D5-2, which represents the data in the second time interval within the preprocessed audio data D5, and the third audio data D5-3, which represents the data in the third time interval within the preprocessed audio data D5, thereby generating processed result data that represents the distance between the second audio data D5-2 and the third audio data D5-3.

[0045] Furthermore, the generation unit 28 applies a dynamic time stretching method to the first audio data D5-1 and the third audio data D5-3 to generate processed result data representing the distance between the first audio data D5-1 and the third audio data D5-3. In this way, the generation unit 28 generates processed result data for each pair of audio data D5-1 to D5-9 within a predetermined time interval.

[0046] The calculation unit 30, described later, may calculate a score representing the degree to which the user has a disease or other condition, based on the processing result data using only the pre-processed audio data D5 generated in this manner.

[0047] The calculation unit 30 calculates a score representing the degree to which a user has a disease or other condition, based on the processing result data generated by the generation unit 28. For example, the calculation unit 30 uses the mean, maximum, minimum, standard deviation, and median of each element of the distance matrix generated by the generation unit 28 to calculate a score representing the degree to which a user has a predetermined disease or symptom, using a known method.

[0048] The estimation unit 32 estimates whether or not the user has a disease or other condition based on the score calculated by the calculation unit 30. For example, if the score is above a predetermined threshold, the estimation unit 32 estimates that the user has a disease or other condition, and if the score is below the predetermined threshold, it estimates that the user does not have a disease or other condition.

[0049] The output unit 34 outputs the estimation result estimated by the estimation unit 32. Alternatively, the output unit 34 may output the score itself as the estimation result.

[0050] The display device 16 displays the estimation results output from the estimation unit 32.

[0051] A medical professional or user operating the information processing device 14 checks the estimated results output from the display device 16 to determine what diseases or symptoms the user may have.

[0052] The information processing system 10 of this embodiment is envisioned for use in situations such as those shown in Figure 6.

[0053] In the example shown in Figure 6, a medical professional H, such as a doctor, is holding a tablet device, which is an example of an information processing system 10. Medical professional H uses the microphone (not shown) on the tablet device to collect voice data from user U, who is the subject. The tablet device then estimates whether user U has any of the diseases or symptoms based on user U's voice data and outputs the estimation result to the display unit (not shown). Medical professional H uses the estimation result displayed on the tablet device's display unit (not shown) as a reference to determine whether user U has any of the diseases or symptoms.

[0054] The information processing device 14 can be implemented, for example, by the computer 50 shown in Figure 7. The computer 50 includes a CPU 51, a memory 52 as a temporary storage area, and a non-volatile storage unit 53. The computer 50 also includes an input / output interface (I / F) 54 to which external devices and output devices are connected, and a read / write (R / W) unit 55 that controls the reading and writing of data to the recording medium. The computer 50 also includes a network I / F 56 that connects to a network such as the Internet. The CPU 51, memory 52, storage unit 53, input / output I / F 54, R / W unit 55, and network I / F 56 are connected to each other via a bus 57.

[0055] The storage unit 53 can be implemented using a Hard Disk Drive (HDD), Solid State Drive (SSD), flash memory, etc. The storage unit 53, as a storage medium, stores a program that allows the computer 50 to function. The CPU 51 reads the program from the storage unit 53, loads it into memory 52, and sequentially executes the processes contained in the program.

[0056] [Operation of the information processing system of the first embodiment]

[0057] Next, the specific operation of the information processing system 10 of the first embodiment will be described. The information processing device 14 of the information processing system 10 executes each of the processes shown in Figure 8.

[0058] First, in step S100, the acquisition unit 20 acquires the user's voice data collected by the microphone 12. Then, the acquisition unit 20 stores the voice data in the voice data storage unit 22.

[0059] Next, in step S102, the processing unit 26 reads the audio data stored in the audio data storage unit 22. Then, the processing unit 26 extracts the central portion of the audio data, which is data within a predetermined time interval, from the audio data.

[0060] In step S104, the processing unit 26 extracts data for a predetermined period from the central portion of the audio data acquired in step S102.

[0061] In step S105, the processing unit 26 performs a shift process on the audio data for a predetermined period acquired in step S104.

[0062] In step S106, the processing unit 26 generates sampled data by performing a predetermined sampling process on the shifted data for a predetermined period obtained in step S105.

[0063] In step S108, the processing unit 26 performs amplitude stretching / contraction processing on the sampling data generated in step S106.

[0064] In step S110, the processing unit 26 performs a stretching process in the time axis direction on the sampled data obtained in step S108, which has already undergone stretching in the amplitude direction.

[0065] As each of the processes in steps S102 to S110 is executed, pre-processed audio data is generated in which pre-processing has been performed on the audio data.

[0066] In step S112, the estimation unit 32 applies a dynamic time stretching method to the pre-processed audio data and the reference user's audio data stored in the reference data storage unit 24 to generate processing result data representing the distance between the pre-processed audio data and the reference user's audio data.

[0067] Furthermore, the reference users will consist of reference users who have a specified disease or condition and reference users who do not have a specified disease or condition.

[0068] For this reason, for example, the estimation unit 32 generates processing result data between the pre-processed audio data and the audio data of a reference user who has a disease, etc., stored in the reference data storage unit 24. Alternatively, for example, the estimation unit 32 generates processing result data between the pre-processed audio data and the audio data of a reference user who does not have a disease, etc., stored in the reference data storage unit 24.

[0069] In step S114, the calculation unit 30 calculates a score representing the degree to which the user has a disease or other condition, based on the processing result data generated in step S112. The score may be set such that, for example, a higher value indicates a higher degree to which the user has a disease or other condition. Alternatively, the score may be set such that, for example, a lower value indicates a higher degree to which the user has a disease or other condition.

[0070] For example, consider a case where the score takes a larger value the higher the degree to which the user has a disease or other condition. In this case, if the distance between the pre-processed audio data and the audio data of a reference user who has a disease or other condition is small, the calculation unit 30 calculates the score in such a way that the degree to which the user has a disease or other condition is high. On the other hand, if the distance between the pre-processed audio data and the audio data of a reference user who has a disease or other condition is large, the calculation unit 30 calculates the score in such a way that the degree to which the user has a disease or other condition is low.

[0071] Furthermore, for example, if the distance between the pre-processed audio data and the audio data of a reference user who does not have a disease, the calculation unit 30 calculates a score such that the degree to which the user has a disease is lower. On the other hand, if the distance between the pre-processed audio data and the audio data of a reference user who does not have a disease, the calculation unit 30 calculates a score such that the degree to which the user has a disease is higher.

[0072] In step S116, the estimation unit 32 estimates whether the user has a disease or other condition based on the score calculated in step S114. For example, if the score is above a predetermined threshold, the estimation unit 32 estimates that the user has a disease or other condition, and if the score is below the predetermined threshold, it estimates that the user does not have a disease or other condition.

[0073] Furthermore, the estimation unit 32 may estimate which disease or condition a user has based on the processing result data for each of the following: the voice data of a reference user having disease A, the voice data of a reference user having disease B, and the voice data of a reference user having disease C.

[0074] In step S118, the output unit 34 outputs the estimation result estimated in step S116.

[0075] The display device 16 displays the estimation results output from the output unit 34. A medical professional or user operating the information processing device 14 checks the estimation results output from the display device 16 to determine what diseases or symptoms the user may have.

[0076] As described above, the information processing system 10 of the first embodiment acquires audio data, which is time-series data of the voice spoken by the user, and generates pre-processed data. The information processing device 14 then applies Dynamic Time Warping to the generated pre-processed audio data to generate processing result data, and calculates a score representing the degree to which the user has a predetermined disease or symptom based on the generated processing result data. The information processing device 14 then estimates whether or not the user has a predetermined disease or symptom based on the calculated score. In this way, by applying Dynamic Time Warping to the audio data, which is time-series data of the voice spoken by the user, it is possible to accurately estimate whether or not the user has a predetermined disease or symptom.

[0077] The pre-processed audio data is the central portion of the acquired audio data, representing data from the first hour after the start of the audio data, and up to two hours before the end of the audio data. By using the central portion of the audio data as pre-processed audio data, it is possible to accurately estimate whether or not a user has a predetermined disease or predetermined symptoms by using the stable central portion of the voice spoken by the user.

[0078] Furthermore, pre-processed audio data also consists of data for a predetermined period. Pre-processed audio data is also obtained by shifting the data along the time axis. Pre-processed audio data is also obtained by performing a predetermined sampling process. Furthermore, pre-processed audio data is also obtained by performing a stretching / contraction process along the time axis. Furthermore, pre-processed audio data is also obtained by performing a stretching / contraction process along the amplitude axis. By performing these pre-processing steps on audio data, it becomes possible to format the audio data in a way suitable for estimating diseases, etc., enabling accurate estimation of whether or not a user has a disease or the like.

[0079] <Information Processing System of the Second Embodiment>

[0080] Next, a second embodiment will be described. Note that parts of the information processing system configuration in the second embodiment that are the same as those in the first embodiment are denoted by the same reference numerals and their descriptions are omitted.

[0081] Figure 9 shows the information processing system 310 of the second embodiment. As shown in Figure 9, the information processing system 310 comprises a user terminal 18 and an information processing device 314. The information processing device 314 further comprises a communication unit 36.

[0082] The information processing device 314 of the information processing system 310 estimates whether or not the user has a disease or other condition based on the user's voice collected by the microphone 12 installed in the user terminal 18.

[0083] The information processing system 310 of the second embodiment is envisioned to be used, for example, in situations such as those shown in Figures 10 and 11.

[0084] In the example shown in Figure 10, a medical professional H, such as a doctor, operates the information processing device 314, while the subject, user U, operates the user terminal 18. User U collects their own voice data "XXXX" using the microphone 12 on the user terminal 18 that they are operating. The user terminal 18 then transmits the voice data to the information processing device 314 via a network 19 such as the internet.

[0085] The information processing device 314 receives the voice data "XXX" from user U transmitted from the user terminal 18. Based on the received voice data, the information processing device 314 estimates whether or not user U has any disease or symptoms, and outputs the estimation result to the display unit 315 of the information processing device 314. Medical professional H uses the estimation result displayed on the display unit 315 of the information processing device 314 as a reference to determine whether or not user U has any disease or symptoms.

[0086] On the other hand, in the example shown in Figure 11, user U, the subject, collects their own voice data using the microphone 12 of the user terminal 18 that they operate. The user terminal 18 then transmits the voice data to the information processing device 314 via a network 19 such as the internet. The information processing device 314 receives the voice data of user U transmitted from the user terminal 18. Based on the received voice data, the information processing device 314 estimates whether or not user U has any disease or symptoms, and transmits the estimation result to the user terminal 18. The user terminal 18 receives the estimation result transmitted from the information processing device 14 and displays the estimation result on a display unit (not shown). The user checks the estimation result to see what disease or symptoms they are most likely to have.

[0087] The information processing device 314 executes the same information processing routine as shown in Figure 8 above.

[0088] As described above, the information processing system of the second embodiment can estimate whether or not a user has a mental illness, a neurological illness, or symptoms thereof, using an information processing device 214 installed on the cloud. [Examples]

[0089] Next, an example will be described. In this example, experimental results regarding the effects of the pretreatment described in this example are shown.

[0090] Figure 12 is a graph plotting audio data obtained from subjects evaluated as having depression (indicated by squares in Figure 12) and audio data obtained from subjects evaluated as healthy (indicated by circles in Figure 12). Figure 12 shows data obtained using the preprocessing and DTW of this embodiment. In the graph of Figure 12, the horizontal axis dist2 represents the distance from the average reference of healthy individuals, and the vertical axis dist3 represents the distance from the average reference of depressed patients. As shown in Figure 12, the data with squares, which are audio data from depressed patients, tend to be farther from the average reference of healthy individuals and shorter from the average reference of depressed patients. Similarly, the data with circles, which are audio data from healthy individuals, tend to be shorter from the average reference of healthy individuals and longer from the average reference of depressed patients. Furthermore, Figure 12 shows the ROC curve and AUC values. As shown in Figure 12, the AUC value when depression is diagnosed by combining the distance from the average reference of healthy individuals (dist2) and the distance from the average reference of depressed patients (dist3) is 1.0.

[0091] Figure 13 is a table of data from Figure 12 and other experimental results. In Figure 13, HAMD represents a score related to depression assessment. HAMD≧7 means that only those with a score of 7 or higher were evaluated. MDD represents depressed patients, PD represents patients with Parkinson's disease, AD represents patients with Alzheimer's disease, and HE represents healthy individuals. MDD=20 means that data from 20 depressed patients was used, and HE=14 means that data from 14 healthy individuals was used. Intra-Person DTW refers to the case where features are generated by creating pairs of intervals in the speech data of one subject, and then DTW is performed. Figure 13 shows the performance without period adjustment, which is a preprocessing step in this embodiment. The performance value is AUC=0.7893, which is lower than the performance value when preprocessing is performed. As shown in the top row of Figure 13, the AUC when distinguishing between HE and MDD is 0.9643, and as shown in the bottom row of Figure 13, the AUC when distinguishing between HE and PD is 0.9173.

[0092] Figure 14 shows experimental results illustrating the effects of the various pretreatments used in this embodiment. The top row of results in Figure 14 represents the baseline results. The results from the second row onward show the effects of applying each pretreatment to the data, demonstrating that each pretreatment contributes to the accuracy of depression diagnosis. Note that in the bottom row of the table in Figure 14, the performance evaluation (AUC) values ​​are reversed; this will be explained below.

[0093] Figure 15 shows the difference in distance calculated by DTW between the case where amplitude adjustment is not performed (labeled "No Amplitude Normalization" in Figure 15) and the case where amplitude adjustment is performed (labeled "Amplitude Normalization Performed" in Figure 15). The vertical axis of the graph shown in Figure 15 represents the distance calculated by DTW.

[0094] Figure 15 shows the distance values ​​calculated by DTW for HE_HospitalA, which represents data obtained from multiple healthy individuals at Hospital A, and HE_HospitalB, which represents data obtained from multiple depressed patients (MDD) and multiple healthy individuals at Hospital B.

[0095] In Figure 15, under "No Amplitude Normalization," it can be seen that there is a large difference between HE_HospitalA, which represents data obtained from multiple healthy individuals at Hospital A, and HE_HospitalB, which represents data obtained from multiple healthy individuals at Hospital B. This is thought to be due to differences in the recording environment and recording settings of the audio data. When the sound pressure of the recorded audio data is low, the distance value calculated by DTW is small, while when the sound pressure of the audio data is high, the distance value calculated by DTW is large. As a result, as shown in "No Amplitude Normalization" in Figure 15, there is a difference between the data distribution of HE_HospitalA and HE_HospitalB, which were recorded in different environments (significant difference in mean values ​​by t-test: p<0.01).

[0096] In contrast, in Figure 15, under the "amplitude normalization" setting, the difference between the recording conditions in HE_HospitalA and HE_HospitalB is calibrated, and no difference is observed between the data distribution of HE_HospitalA and HE_HospitalB, which were recorded in different environments (no significant difference in mean values ​​by t-test: p>0.1). Thus, by incorporating amplitude normalization into the preprocessing, it is possible to correctly classify whether or not a user has a mental illness, a neurological illness, or symptoms thereof, without being affected by differences in recording conditions.

[0097] Similarly, the diagnosis of Alzheimer's disease and Parkinson's disease can also be estimated. Figure 16 shows the various conditions and the results of the differentiation between healthy individuals (HE) and patients suffering from major depressive disorder (MDD), Alzheimer's disease (AD), and Parkinson's disease (PD) using the Intra-Person DTW of this embodiment. Figure 17 shows the ROC curve corresponding to the performance evaluation AUC shown in Figure 16. Figure 18 shows the DTW value calculated under the conditions shown in Figure 16. Figure 19 shows the actual symptoms (labeled "Actual" in Figure 19) and the prediction results using the method of this embodiment (labeled "Prediction" in Figure 19). Figure 20 shows the results of multiple comparison tests.

[0098] As shown in Figure 16, the AUC value when distinguishing between healthy individuals (HE) and patients suffering from some disease (Sick) is 0.8486. Furthermore, as shown in Figure 20, multiple comparison tests of the mean DTW values ​​show that the distribution of healthy individuals (HE) and each disease (MDD, AD, PD) is different (significant difference in mean values: p<0.01). Note that "E" in the table represents "×10", and the number next to it represents the exponent.

[0099] The results shown in Figures 16 to 20 demonstrate that the method of this embodiment can accurately estimate whether or not a user has a mental illness, a neurological illness, or symptoms thereof.

[0100] Furthermore, the technology disclosed herein is not limited to the embodiments described above, and various modifications and applications are possible without departing from the spirit of this invention.

[0101] For example, although the present specification describes an embodiment in which the program is pre-installed, it is also possible to provide the program by storing it on a computer-readable recording medium.

[0102] In addition, the processing that the CPU reads and executes in the above embodiment may be executed by various processors other than the CPU. Examples of such processors include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits that are processors with circuit configurations specifically designed to execute specific processing, such as ASICs (Application Specific Integrated Circuits). Alternatively, a GPGPU (General-purpose graphics processing unit) may be used as the processor. Furthermore, each processing may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0103] Furthermore, while the above embodiments describe a configuration in which the program is pre-stored (installed) on storage, the invention is not limited to this. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the program may be downloaded from an external device via a network.

[0104] Furthermore, each process in this embodiment may be configured by a computer or server equipped with a general-purpose processing unit and storage device, and each process may be executed by a program. This program is stored in the storage device and can be recorded on a recording medium such as a magnetic disk, optical disk, or semiconductor memory, or provided over a network. Of course, none of the other components have to be implemented by a single computer or server; they may be implemented in a distributed manner across multiple computers connected by a network.

[0105] Furthermore, in the embodiments described above, the estimation of whether or not a person has a mental illness or neurological illness, or symptoms of mental disorder or cognitive impairment, was explained as an example of a predetermined disease or symptom, but the invention is not limited to this. The predetermined disease or symptom may be anything. It is assumed that various diseases or symptoms will be reflected in the voice data. For example, not only respiratory diseases and symptoms, but also mental illnesses, etc., will affect the voice data. Therefore, in the embodiments described above, the estimation of whether or not a person has a mental illness or neurological illness, or symptoms of mental disorder or cognitive impairment, was explained as an example of a predetermined disease or symptom, but the invention is not limited to this, and any disease, etc., may be estimated as long as its effect is reflected in the voice data.

[0106] Furthermore, while the above embodiments have described the case where all preprocessing steps are performed when generating preprocessed audio data, the invention is not limited to this. Preprocessed audio data may be generated using at least one of the preprocessing steps described above.

[0107] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

Claims

1. An acquisition unit that acquires audio data, which is time-series data of the voice spoken by the user, A processing unit that generates pre-processed audio data representing data from the audio data acquired by the acquisition unit, which is data from the first hour after the start point of the audio data and data from the second hour before the end point of the audio data, A generation unit generates processed result data by applying Dynamic Time Warping to the preprocessed audio data generated by the processing unit, A calculation unit calculates a score representing the degree to which the user has a predetermined disease or symptom, based on the processing result data generated by the generation unit. An estimation unit estimates whether the user has a predetermined disease or symptom based on the score calculated by the calculation unit, Information processing device including

2. The processing unit generates pre-processed audio data from data that is from the start point of the audio data for a predetermined period of time, and from data that is from the start point of the audio data for a period of time, and from data that is from the end point of the audio data for a period of time. The information processing apparatus according to claim 1.

3. The processing unit generates pre-processed audio data by performing a predetermined sampling process on data from the first hour after the start point of the audio data and on data prior to the second hour after the end point of the audio data. The information processing apparatus according to claim 1 or claim 2.

4. The processing unit generates the pre-processed audio data by performing a process to stretch or compress data in the time axis direction from the start point of the audio data to the data from the first hour after the start point of the audio data and to the data from the second hour before the end point of the audio data. The information processing apparatus according to claim 1 or claim 2.

5. The processing unit generates the pre-processed audio data by performing a process to expand or contract the amplitude of data from the first hour after the start point of the audio data and from the second hour before the end point of the audio data. The information processing apparatus according to claim 1 or claim 2.

6. The processing unit generates the pre-processed audio data by shifting the data in the time axis direction with respect to the data from the start point of the audio data to the data from the first hour onward and to the data from the end point of the audio data to the data from the second hour onward. The information processing apparatus according to claim 1 or claim 2.

7. The generation unit generates the processing result data representing the distance between the preprocessed audio data and the audio data of the reference user by applying the dynamic time stretching method to the preprocessed audio data and the audio data of the reference user whose presence or absence of the predetermined disease or symptoms is known. The information processing apparatus according to claim 1 or claim 2.

8. The generation unit generates the processing result data representing the distance between the first audio data and the second audio data by applying the dynamic time stretching method to the first audio data representing the data in a first time interval within the preprocessed audio data and the second audio data representing the data in a second time interval within the preprocessed audio data. The information processing apparatus according to claim 1 or claim 2.

9. An information processing system comprising a user terminal equipped with a microphone and an information processing device according to claim 1 or claim 2, The user terminal transmits the audio data acquired by the microphone to the information processing device. The acquisition unit of the information processing device acquires the voice data transmitted from the user terminal, The communication unit of the information processing device transmits the estimation result estimated by the estimation unit to the user terminal. The user terminal receives the estimation result transmitted from the information processing device. Information processing system.

10. We acquire audio data, which is time-series data of the voice spoken by the user. Preprocessed audio data is generated from the acquired audio data, representing data from the first hour after the start point of the audio data, and data from the second hour before the end point of the audio data. By applying Dynamic Time Warping to the generated pre-processed audio data, the resulting processed data is generated. Based on the generated processing result data, a score is calculated that represents the degree to which the user has a predetermined disease or symptom. Based on the calculated score, it is estimated whether the user has a predetermined disease or symptom. An information processing method that involves having a computer perform a task.

11. We acquire audio data, which is time-series data of the voice spoken by the user. Preprocessed audio data is generated from the acquired audio data, representing data from the first hour after the start point of the audio data, and data from the second hour before the end point of the audio data. By applying Dynamic Time Warping to the generated pre-processed audio data, the resulting processed data is generated. Based on the generated processing result data, a score is calculated that represents the degree to which the user has a predetermined disease or symptom. Based on the calculated score, it is estimated whether the user has a predetermined disease or symptom. An information processing program that causes a computer to perform a task.

Images