Information processing device, information processing method, information processing system, and information processing program

Abstract

To estimate a respiratory condition during sleep from a user's voice data without requiring the user to wear a predetermined instrument or the like.SOLUTION: An information processing device 14 extracts an acoustic feature amount from voice data uttered by a user. The information processing device 14 estimates information on a respiratory condition during sleep of the user on the basis of the acoustic feature amount. The information processing device 14 outputs information on the respiratory condition. The sound data is the sound data acquired when the user performs at least one utterance task of an utterance task for uttering at least part of "A, I, U, E, O," an utterance task for uttering at least part of "GA, GI, GU, GE, GO," an utterance task for uttering at least part of "RA, RI, RU, RE, RO," an utterance task for uttering at least part of "PA, PI, PU, PE, PO," and an utterance task for uttering a long vowel at a time point different from during sleep.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The disclosed technology relates to an information processing device, an information processing method, an information processing system, and an information processing program. [Background technology]

[0002] US Patent No. 5,399,636 discloses a system for characterizing the upper airway using voice characteristics. The technology disclosed in US Patent No. 5,399,636 includes a mechanical coupler including a means for restricting a patient's jaw position, a means for recording vocalizations, and a processing means for determining at least one anatomical parameter of the upper airway from the recorded vocalizations and for comparing the recorded vocalizations with a threshold value. The system disclosed in US Patent No. 5,399,636 is used as a diagnostic tool for evaluating obstructive sleep apnea. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2014-532448 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology in Patent Document 1 estimates the state of sleep apnea of ​​a patient based on voice data collected when the patient wears a mechanical coupler. However, when collecting voice data, it is preferable to collect voice data without the patient wearing a mechanical coupler.

[0005] The disclosed technology has been made in consideration of the above circumstances, and provides an information processing device, an information processing method, an information processing system, and an information processing program that can estimate a user's breathing state during sleep from voice data obtained by the user performing a specified vocalization task, without the user having to wear any specified equipment, etc. [Means for solving the problem]

[0006] In order to achieve the above object, a first aspect of the present disclosure includes an acquisition unit that acquires voice data uttered by a user, an extraction unit that extracts acoustic features from the voice data acquired by the acquisition unit, an estimation unit that estimates information regarding a respiratory state of the user during sleep based on the acoustic features acquired by the extraction unit, and an output unit that outputs the information regarding the respiratory state estimated by the estimation unit, wherein the voice data includes a speech task of uttering at least a part of "a, i, u, e, o", and an output unit that outputs information regarding the respiratory state estimated by the estimation unit. an information processing device in which the voice data is obtained by the user executing at least one of the following vocalization tasks at a time other than when sleeping: a vocalization task of vocalizing at least a portion of "ka-ki-ku-ke-ko," a vocalization task of vocalizing at least a portion of "ra-ri-ru-re-ro" or "ta-chi-tsu-te-to-da-ji-zu-de-dona-ni-nu-ne-no," a vocalization task of vocalizing at least a portion of "pa-pi-pu-pe-po" or "ba-bi-bu-be-bo-ma-mi-mu-me-mo," and a vocalization task of vocalizing long vowels.

[0007] A second aspect of the present disclosure includes acquiring voice data uttered by a user, extracting acoustic features from the acquired voice data, estimating information regarding a breathing state of the user during sleep based on the acoustic features, and outputting the estimated information regarding the breathing state, wherein the voice data includes a vocalization task of uttering at least a portion of "a, i, u, e, o," a vocalization task of uttering at least a portion of "ga, gi, gu, ge, go" or "ka, ki, ku, ke, ko," and a vocalization task of uttering "ra, ri, ru, re." The information processing method includes a computer executing processing, the processing being voice data obtained by the user performing at least one of the following vocalization tasks at a time other than when asleep: a vocalization task of vocalizing at least a portion of "pa-pi-pu-pe-po" or "ba-bi-bu-be-bo-ma-mi-mu-me-mo"; and a vocalization task of vocalizing a long vowel.

[0008] A third aspect of the present disclosure includes acquiring voice data uttered by a user, extracting acoustic features from the acquired voice data, estimating information regarding a breathing state of the user during sleep based on the acoustic features, and outputting the estimated information regarding the breathing state, wherein the voice data includes a vocalization task of uttering at least a portion of "a, i, u, e, o," a vocalization task of uttering at least a portion of "ga, gi, gu, ge, go" or "ka, ki, ku, ke, ko," and a vocalization task of uttering at least a portion of "ra, ri, ru, re, ro." or a vocalization task of vocalizing at least a portion of "ta-chi-tsu-te-to-da-ji-zu-de-do-na-ni-nu-ne-no," a vocalization task of vocalizing at least a portion of "pa-pi-pu-pe-po" or "ba-bi-bu-be-bo-ma-mi-mu-me-mo," and a vocalization task of vocalizing a long vowel, which is voice data obtained by the user performing at a time other than when asleep. Effect of the Invention

[0009] According to the disclosed technology, it is possible to obtain the effect of estimating a user's breathing state during sleep from voice data obtained by the user performing a specified vocalization task, without the user having to wear any specified equipment, etc. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of an information processing system according to a first embodiment. [Diagram 2] FIG. 1 is a diagram showing the structure of the human upper airway. [Diagram 3] FIG. 1 is a diagram illustrating an example of a usage pattern of an information processing system according to a first embodiment. [Figure 4] FIG. 1 illustrates an example of a computer constituting an information processing device. [Diagram 5] FIG. 2 illustrates an example of a process executed by the information processing apparatus of the first embodiment. [Figure 6] 1 is a diagram showing the upper airway of a human being in a sitting position and in a supine position. FIG. [Figure 7] FIG. 13 is a diagram illustrating an example of a schematic configuration of an information processing system according to a third embodiment. [Figure 8] FIG. 13 is a diagram illustrating an example of a usage pattern of an information processing system according to a third embodiment. [Figure 9] FIG. 13 is a diagram illustrating an example of a usage pattern of an information processing system according to a third embodiment. [Figure 10] FIG. 1 is a diagram showing experimental results according to an embodiment. [Figure 11] FIG. 1 is a diagram showing experimental results according to an embodiment. [Figure 12] FIG. 1 is a diagram showing experimental results according to an embodiment. [Figure 13] FIG. 1 is a diagram showing experimental results according to an embodiment. [Figure 14] FIG. 1 is a diagram showing experimental results according to an embodiment. [Figure 15] FIG. 1 is a diagram showing experimental results according to an embodiment. [Figure 16] FIG. 1 is a diagram showing experimental results according to an embodiment. [Figure 17] FIG. 1 is a diagram showing experimental results according to an embodiment. [Figure 18] FIG. 1 is a diagram showing experimental results according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the disclosed technology will be described in detail with reference to the drawings.

[0012] <Information Processing System of the First Embodiment>

[0013] 1 shows an information processing system 10 according to the first embodiment. As shown in FIG. 1, the information processing system 10 of the first embodiment includes a microphone 12, an information processing device 14, and a display device 16.

[0014] The information processing system 10 extracts acoustic features from voice data, which is time-series data of the voice uttered by the user and collected by the microphone 12, and estimates the breathing state of the user during sleep based on the acoustic features.

[0015] Specifically, the information processing device 14 of the information processing system 10 collects voice data of the user at times other than when the user is asleep. Then, the information processing device 14 estimates the breathing state of the user while sleeping based on the voice data.

[0016] In this embodiment, the following various vocalization tasks are considered to be useful for estimating the respiratory state during sleep, and voice data obtained by the various vocalization tasks in which the user speaks while in a sitting position is used.

[0017] (1) Speech task to assess the entire airway In this embodiment, the breathing state of the user while sleeping is estimated using voice data when the user utters at least a part of the phrase "a-i-u-e-o."

[0018] Fig. 2 is a diagram showing the structure of the upper airway of a human. When a user utters at least a part of the phrase "a, i, u, e, o", the user utters only vowels without consonants using the lips, teeth, tongue, etc., making it possible to evaluate the morphological characteristics of the entire airway shown in Fig. 2. For this reason, in this embodiment, a task of uttering at least a part of the phrase "a, i, u, e, o" is set as a utterance task for mainly evaluating the morphological characteristics of the entire airway.

[0019] (2) A speech task to assess the posterior tongue and nasopharynx In this embodiment, the breathing state of the user during sleep is estimated using voice data obtained when the user utters at least a part of the phrase "ga, gi, gu, ge, go". When the user utters at least a part of the phrase "ga, gi, gu, ge, go", it is necessary to utilize the back tongue to the nasopharynx. For this reason, the voice data obtained when at least a part of these phrases is uttered is considered to reflect the function and morphological characteristics of the back tongue to the nasopharynx A shown in FIG. 2. For this reason, in this embodiment, a task of uttering at least a part of the phrase "ga, gi, gu, ge, go" is set as a speech task for mainly evaluating the back tongue to the nasopharynx, which is directly connected to securing the airway during sleep. It is considered that the function and morphological characteristics of the back tongue to the nasopharynx are also reflected in at least a part of the voice data of the phrase "ka, ki, ku, ke, ko". For this reason, the speech task for evaluating the nasopharynx from the back tongue may be "ka, ki, ku, ke, ko" instead of "ga, gi, gu, ge, go".

[0020] (3) A speech task to assess the tip of the tongue from the oral cavity In this embodiment, the breathing state of the user during sleep is estimated using voice data when the user utters at least a part of the phrase "ra-ri-ru-re-ro." When the user utters at least a part of the phrase "ra-ri-ru-re-ro," it is necessary to effectively use the tongue tip and hard palate, which is considered to reflect the functional and morphological characteristics of the oral cavity B including the tongue tip and hard palate shown in FIG. 2. For this reason, in this embodiment, a task of uttering at least a part of the phrase "ra-ri-ru-re-ro" is set as a speech task for mainly evaluating the tongue tip from the oral cavity. Note that at least a part of the voice data of the phrase "ta-chi-tsu-te-to-da-ji-zu-de-do-na-ni-nu-ne-no" is also considered to reflect the functional and morphological characteristics of the oral cavity B including the tongue tip and hard palate. For this reason, the speech task to evaluate the oral cavity could be "ta·chi·tsu·te·to·da·ji·zu·de·don·na·ni·nu·ne·no" instead of "ra·ri·ru·re·ro."

[0021] (4) A speech task to evaluate the lips In this embodiment, the breathing state of the user during sleep is estimated using voice data when the user utters at least a part of the phrase "pa-pi-pu-pe-po". When the user utters at least a part of the phrase "pa-pi-pu-pe-po", the user needs to move the lips well and needs to use the lips C shown in FIG. 2 well. For this reason, in this embodiment, a task of uttering at least a part of the phrase "pa-pi-pu-pe-po" is set as a utterance task for mainly evaluating the lips. It is considered that the function and morphological characteristics of are also reflected in at least a part of the voice data of the phrase "ba-bi-bu-be-bo-ma-mi-mu-me-mo". For this reason, the utterance task for evaluating the lips may be "ba-bi-bu-be-bo-ma-mi-mu-me-mo" instead of "pa-pi-pu-pe-po".

[0022] (5) A vocalization task in which participants were asked to produce long vowels. When a user utters the long vowels "a," "i," "u," "e," and "o," it is necessary to effectively utilize part or all of the airway, similar to the above-mentioned utterance task "a, i, u, e, o" for evaluating the entire airway. For this reason, a utterance task for uttering long vowels is considered useful, so a task for uttering long vowels is set.

[0023] The specific details will be explained below.

[0024] 1, the information processing device 14 functionally includes an acquisition unit 20, a voice data storage unit 22, an extraction unit 24, an estimation unit 26, and an output unit 28. The information processing device 14 is realized by a computer as described below.

[0025] The acquiring unit 20 acquires voice data, which is time-series data of a voice uttered by a user. The acquiring unit 20 then stores the voice data in the voice data storage unit 22.

[0026] The voice data in this embodiment is voice data obtained by a user performing at least one of the following vocalization tasks at a time other than when sleeping: a vocalization task of pronouncing at least a part of "a, i, u, e, o" which evaluates the entire airway; a vocalization task of pronouncing at least a part of "ga, gi, gu, ge, go" or "ka, ki, ku, ke, ko" which mainly evaluates the back of the tongue to the nasopharynx; a vocalization task of pronouncing at least a part of "ra, ri, ru, re, ro" or "ta, chi, tsu, te, to, da, ji, zu, de, do na, ni, nu, ne, no" which mainly evaluates the oral cavity including the tip of the tongue and hard palate; a vocalization task of pronouncing at least a part of "pa, pi, pu, pe, po" or "ba, bi, bu, be, bo, ma, mi, mu, me, mo" which mainly evaluates the lips; and a vocalization task of pronouncing a long vowel.

[0027] The voice data storage unit 22 stores the voice data acquired by the acquisition unit 20.

[0028] The extraction unit 24 extracts acoustic features from the speech data. For example, the extraction unit 24 extracts acoustic features from the speech data using a known openSMILE (open-source Speech and Music Interpretation by Large-space Extraction) (Internet URL:<https: / / audeering.github.io / opensmile / > ,Retrieved February 28, 2023), Praat (Internet URL<https: / / www.fon.hum.uva.nl / praat / > ,Retrieved February 28, 2023), pyAudioAnalysis (Internet URL < https: / / github.com / tyiannak / pyAudioAnalysis >,Retrieved February 28, 2023), Pyworld (Internet URL<https: / / pypi.org / project / pyworld / > ,Retrieved March 15, 2023), librosa (Internet URL<https: / / librosa.org / doc / latest / index.html> ,Retrieved March 15, 2023) and phonet (Internet URL<https: / / phonet.readthedocs.io / en / latest / > Acoustic features that can be used in the speech recognition technology (search results on March 15, 2023) may be extracted from the speech data. Examples of acoustic features include fundamental frequency, spectral envelope, non-periodicity index, or phoneme features, such as those disclosed in Japanese Patent Application Laid-Open No. 2021-194527.

[0029] The estimation unit 26 estimates information related to the breathing state of the user during sleep, based on the multiple acoustic features extracted by the extraction unit 24. For example, the information related to the breathing state during sleep is information indicating the risk, degree, or severity of sleep apnea syndrome.

[0030] For example, the estimation unit 26 estimates information related to the user's breathing state during sleep by inputting the multiple acoustic features into a known machine learning model (e.g., a neural network or a decision tree model), a deep learning model, or a statistical model (e.g., logistic regression). In this case, a predetermined score corresponding to the multiple acoustic features is output from the known machine learning model, deep learning model, or statistical model. This score corresponds to information related to the user's breathing state during sleep. The machine learning model, deep learning model, or statistical model is generated in advance based on learning data, etc.

[0031] The data output from the trained model or statistical model is, for example, numerical data representing the breathing state of the user during sleep. For example, the data output from the trained model or statistical model is at least one numerical value representing the risk, degree, or severity of sleep apnea syndrome.

[0032] For example, the estimation unit 26 inputs multiple acoustic features into a trained model that has been previously trained using machine learning or deep learning, or a statistical model that has been previously generated, and regards the data output from the trained model or the statistical model as a score representing the degree to which the user has sleep apnea syndrome.

[0033] The estimation unit 26 may estimate information related to the respiratory state during sleep of the target user by referring to voice data of a reference user whose respiratory state during sleep is known.

[0034] The output unit 28 outputs the estimation result estimated by the estimation unit 26 .

[0035] The display device 16 displays the estimation result output from the estimation unit 26.

[0036] A medical professional or a user operating the information processing device 14 checks the estimation results outputted from the display device 16 and confirms the breathing state of the user while sleeping.

[0037] The information processing system 10 of this embodiment is assumed to be used, for example, under conditions as shown in FIG.

[0038] In the example of FIG. 3, a medical worker H such as a doctor holds a tablet terminal, which is an example of the information processing system 10. The medical worker H collects voice data of a user U, who is a subject, using a microphone (not shown) provided in the tablet terminal. Then, based on the voice data of the user U, the tablet terminal estimates whether the user U has any disease or symptom, and outputs the estimation result to a display unit (not shown). The medical worker H judges the breathing state of the user U during sleep by referring to the estimation result displayed on the display unit (not shown) of the tablet terminal.

[0039] The information processing device 14 can be realized by, for example, a computer 50 shown in Fig. 4. The computer 50 includes a CPU 51, a memory 52 as a temporary storage area, and a non-volatile storage unit 53. The computer 50 also includes an input / output interface (I / F) 54 to which an external device, an output device, etc. are connected, and a read / write (R / W) unit 55 that controls reading and writing of data from and to a recording medium. The computer 50 also includes a network I / F 56 that is connected to a network such as the Internet. The CPU 51, the memory 52, the storage unit 53, the input / output I / F 54, the R / W unit 55, and the network I / F 56 are connected to one another via a bus 57.

[0040] The storage unit 53 can be realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage unit 53 as a storage medium stores a program for operating the computer 50. The CPU 51 reads the program from the storage unit 53, expands it in the memory 52, and sequentially executes the processes contained in the program.

[0041] [Operation of the information processing system according to the first embodiment]

[0042] Next, a specific operation of the information processing system 10 of the first embodiment will be described. The information processing device 14 of the information processing system 10 executes each process shown in FIG.

[0043] First, in step S100, the acquisition unit 20 acquires voice data of the user collected by the microphone 12. Then, the acquisition unit 20 stores the voice data in the voice data storage unit 22.

[0044] Next, in step S 102 , the extraction unit 24 extracts a plurality of acoustic features from the voice data stored in the voice data storage unit 22 .

[0045] In step S104, the estimation unit 26 inputs the multiple acoustic characteristics obtained in step S102 into a trained model previously trained by machine learning or deep learning or a statistical model previously generated, and estimates data output from the trained model or statistical model as a score related to the user's breathing state during sleep. Note that the score takes a larger value, for example, as the breathing state during sleep is more likely to be an apneic state. Alternatively, the score may take a smaller value, for example, as the breathing state during sleep is more likely to be an apneic state.

[0046] Then, in step S106, the output unit 28 outputs the score estimated in step S104 above as an estimation result.

[0047] The display device 16 displays the estimation result output from the output unit 28. A medical professional or a user who operates the information processing device 14 checks the estimation result output from the display device 16 and confirms what state the user's respiratory state during sleep is estimated to be.

[0048] As described above, the information processing system 10 of the first embodiment extracts acoustic features from voice data uttered by a user, and estimates information on the user's breathing state during sleep based on the acoustic features. The voice data here is voice data obtained by the user performing at least one of the following vocalization tasks at a time other than when the user is sleeping: a vocalization task for uttering at least a part of "a, i, u, e, o" to evaluate the entire airway, a vocalization task for uttering at least a part of "ga, gi, gu, ge, go" to evaluate the pharyngeal part including the tongue root, a vocalization task for uttering at least a part of "ra, ri, ru, re, ro" to evaluate the oral cavity including the tongue, a vocalization task for uttering at least a part of "pa, pi, pu, pe, po" to evaluate the lips, and a vocalization task for uttering a long vowel. This makes it possible to estimate the user's breathing state during sleep from the voice data obtained by the user performing a predetermined vocalization task without the user having to wear a predetermined device or the like.

[0049] <Information Processing System of Second Embodiment> Next, a second embodiment will be described. Note that since the configuration of the information processing system of the second embodiment is similar to that of the first embodiment, the same reference numerals are used and the description will be omitted.

[0050] The information processing system of the second embodiment estimates information regarding the breathing state of a user while sleeping, using voice data uttered when the user is in a sitting position and voice data uttered when the user is in a supine position.

[0051] FIG. 6 is a diagram showing the upper airway when a human is in a sitting position and when a human is in a supine position.

[0052] In general, obstructive sleep apnea is more likely to occur when the cross-sectional area of ​​the upper airway is small. In addition, the shape of the airway above the vocal cords changes the resonance characteristics of the sound, resulting in changes in voice quality.

[0053] Comparing the upper airway in the sitting position and the supine position shown in FIG. 6, it can be seen that the cross-sectional area D of the upper airway is smaller and the airway shape E is also changed.

[0054] Therefore, in this embodiment, information regarding the user's respiratory state during sleep is estimated using voice data uttered when the user is in a sitting position and voice data uttered when the user is in a supine position.

[0055] Specifically, the information processing device 14 of the second embodiment analyzes the change between voice data uttered when the user is in a sitting position and voice data uttered when the user is in a supine position, and estimates information related to the breathing state during sleep based on the analysis results of the change.

[0056] More specifically, the estimation unit 26 of the information processing device 14 of the second embodiment calculates the difference, ratio, or rate of change between the voice data uttered when the user is in a sitting position and the voice data uttered when the user is in a supine position as an example of the change between these two voice data.

[0057] Then, the estimation unit 26 of the information processing device 14 of the second embodiment estimates information related to the breathing state of the user during sleep based on the changes in the two pieces of voice data, similarly to the first embodiment.

[0058] Other configurations and operations of the information processing system 10 according to the second embodiment are similar to those of the first embodiment, and therefore will not be described.

[0059] As described above, the information processing system 10 of the second embodiment analyzes the change between the voice data uttered when the user is in a sitting position and the voice data uttered when the user is in a supine position, and estimates information on the breathing state during sleep based on the analysis result of the change. This makes it possible to estimate the breathing state of the user during sleep from the voice data obtained by the user performing a predetermined vocal task without the user having to wear a predetermined device or the like.

[0060] <Modification of the second embodiment> In the second embodiment, the information on the breathing state during sleep is estimated by using the change between the voice data uttered when the user is in a sitting position and the voice data uttered when the user is in a supine position, but the present invention is not limited to this. For example, as a modification of the second embodiment, the information on the breathing state during sleep of the user may be estimated by using the voice data uttered when the user is in a supine position. In this case, the information processing device 14 of the second embodiment analyzes the voice data uttered when the user is in a supine position, and estimates the information on the breathing state during sleep based on the analysis result. By analyzing the voice data uttered when the user is in a supine position, it is possible to evaluate the narrowed airway state, and the breathing state of the user during sleep can be accurately estimated.

[0061] <Information Processing System of Third Embodiment>

[0062] Next, a third embodiment will be described. Note that, among the configuration of the information processing system of the third embodiment, the parts having the same configuration as the first or second embodiment will be given the same reference numerals and the description thereof will be omitted.

[0063] Fig. 7 shows an information processing system 310 according to the third embodiment. As shown in Fig. 7, the information processing system 310 includes a user terminal 18 and an information processing device 314. The information processing device 314 further includes a communication unit .

[0064] The information processing device 314 of the information processing system 310 estimates the breathing state of the user during sleep based on the user's voice collected by the microphone 12 provided in the user terminal 18.

[0065] The information processing system 310 of the third embodiment is assumed to be used, for example, under the circumstances shown in FIG. 8 and FIG.

[0066] 8, a medical professional H such as a doctor operates an information processing device 314, and a user U who is a subject operates a user terminal 18. The user U collects his / her own voice data "XXXX" using the microphone 12 of the user terminal 18 that he / she operates. Then, the user terminal 18 transmits the voice data to the information processing device 314 via a network 19 such as the Internet.

[0067] The information processing device 314 receives the voice data "XXX" of the user U transmitted from the user terminal 18. Then, the information processing device 314 estimates the respiratory state of the user U while sleeping based on the received voice data, and outputs the estimation result to the display unit 315 of the information processing device 314. The medical staff H checks the respiratory state of the user U while sleeping by referring to the estimation result displayed on the display unit 315 of the information processing device 314.

[0068] On the other hand, in the example of FIG. 9, a user U who is a subject collects his / her own voice data using the microphone 12 of a user terminal 18 operated by the user. Then, the user terminal 18 transmits the voice data to an information processing device 314 via a network 19 such as the Internet. The information processing device 314 receives the voice data of the user U transmitted from the user terminal 18. Then, the information processing device 314 estimates the respiratory state of the user U during sleep based on the received voice data, and transmits the estimation result to the user terminal 18. The user terminal 18 receives the estimation result transmitted from the information processing device 14, and displays the estimation result on a display unit (not shown). The user checks the estimation result and confirms what state his / her respiratory state during sleep is likely to be.

[0069] The information processing device 314 executes the same information processing routine as that shown in FIG.

[0070] As described above, the information processing system according to the third embodiment can estimate the respiratory state of the user during sleep by using the information processing device 314 installed on the cloud. EXAMPLES

[0071] Next, a description will be given of Example 1. Example 1 shows experimental results regarding the effects of the technique described in the first embodiment.

[0072] In Example 1, a model for predicting an Apnea-Hypopnea Index (AHI) was generated from collected voice data, and its accuracy was verified. The model in this example is a model for estimating whether AHI is <20 or AHI is ≥20.

[0073] In Example 1, we use voice data when subjects utter "A, I, U, E, O," "Ga, Gi, Gu, Ge, Go," "Ra, Ri, Ru, Re, Ro," and "Pa, Pi, Pu, Pe, Po." In addition, a certain number of conventional features are selected using a known method. In addition, a decision tree is trained using LightGBM, an example of a known machine learning model, and the accuracy is confirmed by 10-fold cross-validation.

[0074] Figure 10 shows the results of estimating whether AHI<20 or AHI≧20 using speech data when the subject uttered "a, i, u, e, o" in a sitting position. As shown in Figure 10, the accuracy rate was 89.9%, and the AUC (Area Under the Curve) was 0.959.

[0075] Figure 11 shows the results of estimating whether AHI is < 20 or >= 20 using speech data when a subject uttered the sounds "ga, gi, gu, ge, go" in a sitting position. As shown in Figure 11, the accuracy rate was 86.0%, and the AUC (Area Under the Curve) was 0.940.

[0076] Figure 12 shows the results of estimating whether AHI is < 20 or >= 20 using speech data when the subject uttered "ra-ri-ru-re-ro" in a sitting position. As shown in Figure 12, the accuracy rate was 85.5%, and the AUC (Area Under the Curve) was 0.923.

[0077] Figure 13 shows the results of estimating whether AHI is < 20 or >= 20 using voice data when a subject utters "pa-pi-pu-pe-po" in a sitting position. As shown in Figure 13, the accuracy rate is 86.0%, and the AUC (Area Under the Curve) is 0.944.

[0078] Fig. 14 shows the results of estimating whether AHI<20 or AHI≧20 using a combination of speech data when the subject uttered all the above phrases in a sitting position. As shown in Fig. 14, the accuracy rate was 98.3%, and the AUC (Area Under the Curve) was 0.997.

[0079] It can be said from the results shown in Figs. 10 to 14 that the method according to the first embodiment makes it possible to estimate the breathing state of the user while sleeping with high accuracy. EXAMPLES

[0080] Next, a description will be given of Example 2. Example 2 shows experimental results regarding the effects of the technique described in the modified example of the second embodiment.

[0081] In Example 2, an acoustic feature obtained from voice data uttered when the subject is in a supine position is used for estimation. In Example 2, a model for predicting an Apnea-Hypopnea Index (AHI) was generated and its accuracy was verified, as in Example 1. The model in this example is a model for estimating whether AHI<20 or AHI≧20.

[0082] In Example 2, speech data of a subject uttering "ga, gi, gu, ge, go" is used. In Example 2, a predetermined number of conventional features are selected using a known method, as in Example 1. A decision tree is trained using LightGBM, an example of a known machine learning model, and the accuracy is confirmed by 10-fold cross-validation.

[0083] Fig. 15 shows the results of Example 2. From the results shown in Fig. 15, it can be said that the method of the modified example of the second embodiment can accurately estimate the breathing state of a user during sleep only from the voice data uttered when the subject is in a supine position. EXAMPLES

[0084] Next, a description will be given of Example 3. Example 3 shows experimental results regarding the effects of the technique described in the second embodiment.

[0085] In Example 3, the difference between the acoustic feature amount obtained from the voice data uttered when the subject is in a sitting position and the acoustic feature amount obtained from the voice data uttered when the subject is in a supine position is calculated, and the difference is used for estimation. In Example 3, as in Examples 1 and 2, a model for predicting the Apnea-Hypopnea Index (AHI) was generated, and its accuracy was verified. The model in this example is a model for estimating whether AHI<20 or AHI≧20.

[0086] In Example 3, speech data is used when a subject utters "A, I, U, E, O," "Ga, Gi, Gu, Ge, Go," "Pa, Pi, Pu, Pe, Po," and "Ra, Ri, Ru, Re, Ro." In Example 3, a predetermined number of conventional features are selected using a known method, as in Examples 1 and 2. A decision tree is trained using LightGBM, an example of a known machine learning model, and the accuracy is confirmed by 10-fold cross-validation.

[0087] Figures 16 to 18 show the results of Example 3. From the results shown in Figures 16 to 18, it can be said that the method of the second embodiment makes it possible to accurately estimate the breathing state of the user while sleeping.

[0088] In addition, it is believed that a vocalization task in which phrases not disclosed in the above embodiments (e.g., "ta-chi-tsu-te-to-da-ji-zu-de-do-na-ni-nu-ne-no" or "ba-bi-bu-be-bo-ma-mi-mu-me-mo", etc.) are vocalized can also accurately estimate the user's breathing state while sleeping.

[0089] Additionally, it is believed that the vocalization task of pronouncing long vowels such as "ah," "i," "uh," "eh," and "oh" can also accurately estimate the user's breathing state while sleeping.

[0090] The technology of the present disclosure is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit and scope of the present invention.

[0091] For example, although the present specification has been described as an embodiment in which a program is pre-installed, the program may also be provided by being stored on a computer-readable recording medium.

[0092] In the above embodiment, the process executed by the CPU after reading the software (program) may be executed by various processors other than the CPU. In this case, examples of the processor include a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing such as an FPGA (Field-Programmable Gate Array), and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) which is a processor having a circuit configuration designed exclusively for executing a specific process. Alternatively, a GPGPU (General-purpose graphics processing unit) may be used as the processor. Each process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, etc.). The hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0093] In addition, in each of the above embodiments, the program is described as being pre-stored (installed) in the storage, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0094] Furthermore, each process of the present embodiment may be implemented by a computer or server equipped with a general-purpose processor and a storage device, and each process may be executed by a program. This program is stored in a storage device, and may be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or may be provided through a network. Of course, any other components do not have to be implemented by a single computer or server, and may be distributed and implemented on multiple computers connected by a network.

[0095] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[0096] (Additional Note) The following additional notes are given regarding aspects of the present disclosure.

[0097] (Appendix 1) An acquisition unit that acquires voice data uttered by a user; an extraction unit that extracts acoustic features from the speech data acquired by the acquisition unit; an estimation unit that estimates information regarding a respiratory state of the user during sleep based on the acoustic feature obtained by the extraction unit; an output unit that outputs information regarding the respiratory state estimated by the estimation unit; Including, The voice data is voice data obtained by the user performing at least one of the following vocalization tasks at a time other than when sleeping: a vocalization task for vocalizing at least a part of "a, i, u, e, o," a vocalization task for vocalizing at least a part of "ga, gi, gu, ge, go" or "ka, ki, ku, ke, ko," a vocalization task for vocalizing at least a part of "ra, ri, ru, re, ro" or "ta, chi, tsu, te, to, da, ji, zu, de, dona, ni, nu, ne, no," a vocalization task for vocalizing at least a part of "pa, pi, pu, pe, po" or "ba, bi, bu, be, bo, ma, mi, mu, me, mo," and a vocalization task for vocalizing a long vowel. Information processing device.

[0098] (Appendix 2) The acquisition unit acquires at least one of the voice data uttered while the user is in a sitting position and the voice data uttered while the user is in a supine position, The estimation unit estimates information related to the breathing state during sleep based on the voice data. 2. An information processing device according to claim 1.

[0099] (Appendix 3) The estimation unit analyzes a change between the voice data uttered when the user is in a sitting position and the voice data uttered when the user is in a supine position, and estimates information related to the breathing state during sleep based on an analysis result of the change. 3. An information processing device according to claim 1 or 2.

[0100] (Appendix 4) The estimation unit inputs the acoustic feature to a trained model that has been trained in advance by machine learning or deep learning, or a statistical model that has been generated in advance, and estimates data output from the trained model or the statistical model as information regarding the respiratory state during sleep. The information processing device according to any one of Supplementary Note 1 to Supplementary Note 3.

[0101] (Appendix 5) the vocalization task of vocalizing at least a part of "a, i, u, e, o" is a vocalization task for evaluating the entire airway, The speech task of uttering at least a part of "ga, gi, gu, ge, go" or "ka, ki, ku, ke, ko" is a speech task for evaluating a region from the back of the tongue to the nasopharynx, The speech task of uttering at least a part of "ra-ri-ru-re-ro" or "ta-chi-tsu-te-to-da-ji-zu-de-do-na-ni-nu-ne-no" is a speech task for evaluating the oral cavity including the tip of the tongue and the hard palate, The vocalization task of vocalizing at least a part of "pa, pi, pu, pe, po" or "ba, bi, bu, be, bo, ma, mi, mu, me, mo" is a vocalization task for evaluating the lips. 3. An information processing device according to claim 1 or 2.

[0102] (Appendix 6) An information processing system including a user terminal equipped with a microphone and the information processing device according to any one of Supplementary Note 1 to Supplementary Note 5, the user terminal transmits the voice data acquired by the microphone to the information processing device; The acquisition unit of the information processing device acquires the voice data transmitted from the user terminal, a communication unit of the information processing device that transmits an estimation result estimated by the estimation unit to a user terminal; The user terminal receives the estimation result transmitted from the information processing device. Information processing system.

[0103] (Appendix 7) Acquire voice data uttered by the user, Extracting acoustic features from the acquired speech data; estimating information about a breathing state of the user during sleep based on the acoustic feature; outputting information relating to the estimated respiratory state; The voice data is voice data obtained by the user performing at least one of the following vocalization tasks at a time other than when sleeping: a vocalization task for vocalizing at least a part of "a, i, u, e, o," a vocalization task for vocalizing at least a part of "ga, gi, gu, ge, go" or "ka, ki, ku, ke, ko," a vocalization task for vocalizing at least a part of "ra, ri, ru, re, ro" or "ta, chi, tsu, te, to, da, ji, zu, de, dona, ni, nu, ne, no," a vocalization task for vocalizing at least a part of "pa, pi, pu, pe, po" or "ba, bi, bu, be, bo, ma, mi, mu, me, mo," and a vocalization task for vocalizing a long vowel. An information processing method in which processing is performed by a computer.

[0104] (Appendix 8) Acquire voice data uttered by the user, Extracting acoustic features from the acquired speech data; estimating information about a breathing state of the user during sleep based on the acoustic feature; outputting information relating to the estimated respiratory state; The voice data is voice data obtained by the user performing at least one of the following vocalization tasks at a time other than when sleeping: a vocalization task for vocalizing at least a part of "a, i, u, e, o," a vocalization task for vocalizing at least a part of "ga, gi, gu, ge, go" or "ka, ki, ku, ke, ko," a vocalization task for vocalizing at least a part of "ra, ri, ru, re, ro" or "ta, chi, tsu, te, to, da, ji, zu, de, dona, ni, nu, ne, no," a vocalization task for vocalizing at least a part of "pa, pi, pu, pe, po" or "ba, bi, bu, be, bo, ma, mi, mu, me, mo," and a vocalization task for vocalizing a long vowel. An information processing program for causing a computer to execute processing. [Explanation of symbols]

[0105] 14 Information processing equipment 20 Acquisition Department 22 Audio data storage unit 24 Extraction part 36 Estimation part 28 Output section 310 Information Processing Systems 314 Information processing equipment

Claims

1. An acquisition unit that acquires voice data uttered by a user; an extraction unit that extracts acoustic features from the speech data acquired by the acquisition unit; an estimation unit that estimates information regarding a respiratory state of the user during sleep based on the acoustic feature obtained by the extraction unit; an output unit that outputs information regarding the respiratory state estimated by the estimation unit; Including, The voice data is voice data obtained by the user executing at least one of the following vocalization tasks at a time other than when sleeping: a vocalization task for vocalizing at least a portion of "a, i, u, e, o," a vocalization task for vocalizing at least a portion of "ga, gi, gu, ge, go" or "ka, ki, ku, ke, ko," a vocalization task for vocalizing at least a portion of "ra, ri, ru, re, ro" or "ta, chi, tsu, te, to, da, ji, zu, de, do, na, ni, nu, ne, no," a vocalization task for vocalizing at least a portion of "pa, pi, pu, pe, po" or "ba, bi, bu, be, bo, ma, mi, mu, me, mo," and a vocalization task for vocalizing a long vowel. Information processing device.

2. The acquisition unit acquires at least one of the voice data uttered while the user is in a sitting position and the voice data uttered while the user is in a supine position, The estimation unit estimates information related to the breathing state during sleep based on the voice data. The information processing device according to claim 1 .

3. The estimation unit analyzes a change between the voice data uttered when the user is in a sitting position and the voice data uttered when the user is in a supine position, and estimates information related to the breathing state during sleep based on an analysis result of the change.

3. The information processing device according to claim 1 or 2.

4. The estimation unit inputs the acoustic feature to a trained model that has been trained in advance by machine learning or deep learning, or a statistical model that has been generated in advance, and estimates data output from the trained model or the statistical model as information regarding the respiratory state during sleep.

3. The information processing device according to claim 1 or 2.

5. The vocalization task of vocalizing at least a part of the sounds "a, i, u, e, o" is a vocalization task for evaluating the entire airway, The speech task of uttering at least a part of "ga, gi, gu, ge, go" or "ka, ki, ku, ke, ko" is a speech task for evaluating the back of the tongue to the nasopharynx, The speech task of uttering at least a part of "ra-ri-ru-re-ro" or "ta-chi-tsu-te-to-da-ji-zu-de-do-na-ni-nu-ne-no" is a speech task for evaluating an oral cavity including a tip of the tongue and a hard palate, The vocalization task of vocalizing at least a part of "pa, pi, pu, pe, po" or "ba, bi, bu, be, bo, ma, mi, mu, me, mo" is a vocalization task for evaluating lips.

3. The information processing device according to claim 1 or 2.

6. An information processing system including a user terminal equipped with a microphone and the information processing device according to claim 1 or 2, the user terminal transmits the voice data acquired by the microphone to the information processing device; The acquisition unit of the information processing device acquires the voice data transmitted from the user terminal, a communication unit of the information processing device that transmits an estimation result estimated by the estimation unit to a user terminal; The user terminal receives the estimation result transmitted from the information processing device. Information processing system.

7. Acquire voice data uttered by the user, Extracting acoustic features from the acquired speech data; estimating information about a breathing state of the user during sleep based on the acoustic feature; outputting information relating to the estimated respiratory state; The voice data is voice data obtained by the user executing at least one of the following vocalization tasks at a time other than when sleeping: a vocalization task for vocalizing at least a portion of "a, i, u, e, o," a vocalization task for vocalizing at least a portion of "ga, gi, gu, ge, go" or "ka, ki, ku, ke, ko," a vocalization task for vocalizing at least a portion of "ra, ri, ru, re, ro" or "ta, chi, tsu, te, to, da, ji, zu, de, do, na, ni, nu, ne, no," a vocalization task for vocalizing at least a portion of "pa, pi, pu, pe, po" or "ba, bi, bu, be, bo, ma, mi, mu, me, mo," and a vocalization task for vocalizing a long vowel. An information processing method in which processing is performed by a computer.

8. Acquire voice data uttered by the user, Extracting acoustic features from the acquired speech data; estimating information about a breathing state of the user during sleep based on the acoustic feature; outputting information relating to the estimated respiratory state; The voice data is voice data obtained by the user executing at least one of the following vocalization tasks at a time other than when sleeping: a vocalization task for vocalizing at least a portion of "a, i, u, e, o," a vocalization task for vocalizing at least a portion of "ga, gi, gu, ge, go" or "ka, ki, ku, ke, ko," a vocalization task for vocalizing at least a portion of "ra, ri, ru, re, ro" or "ta, chi, tsu, te, to, da, ji, zu, de, do, na, ni, nu, ne, no," a vocalization task for vocalizing at least a portion of "pa, pi, pu, pe, po" or "ba, bi, bu, be, bo, ma, mi, mu, me, mo," and a vocalization task for vocalizing a long vowel. An information processing program for causing a computer to execute processing.

Images