Information processing device and information processing method

Abstract

An information processing device according to the present disclosure comprises: an acquiring unit that acquires inhalation information representing an inhalation by a user; and a prediction unit that, on the basis of the inhalation information acquired by the acquiring unit, predicts whether the user will speak after the inhalation by the user.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and an information processing method. [Background technology]

[0002] In recent years, technologies for voice dialogue systems that communicate (dialogue) with a user through voice have been provided. For example, a technology for determining an utterance timing in a voice dialogue system is known (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-211596 A Summary of the Invention [Problem to be solved by the invention]

[0004] According to the conventional technology, the timing of speech is determined in a voice dialogue system based on the timing at which the user's breathing changes from exhalation to inhalation.

[0005] However, in the conventional technology, it is not always possible to appropriately predict whether or not a user will speak. For example, in the conventional technology, the system merely determines the timing at which the user will speak, and it is difficult to predict whether or not the user will speak with such technology. Therefore, it is desirable to appropriately predict whether or not a user will speak.

[0006] Therefore, the present disclosure proposes an information processing device and an information processing method capable of appropriately predicting whether or not a user will speak. [Means for solving the problem]

[0007] In order to solve the above problems, an information processing device of one embodiment according to the present disclosure includes an acquisition unit that acquires inhalation information indicating a user's inhalation, and a prediction unit that predicts whether the user will speak after the user inhales based on the inhalation information acquired by the acquisition unit. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of information processing according to an embodiment of the present disclosure. [Diagram 2] FIG. 11 is a diagram showing an example of inhalation information of a user. [Diagram 3] FIG. 13 is a diagram showing an example of prediction using a user's inhalation. [Figure 4] 1A and 1B are diagrams illustrating an example of a relationship between a user's inhalation and speech. [Diagram 5] FIG. 1 is a diagram illustrating a configuration example of an information processing system according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a diagram illustrating a configuration example of a server device according to an embodiment of the present disclosure. [Figure 7] FIG. 4 is a diagram illustrating an example of a threshold information storage unit according to an embodiment of the present disclosure. [Figure 8] FIG. 2 is a diagram illustrating a configuration example of a terminal device according to an embodiment of the present disclosure. [Figure 9] 1 is a flowchart illustrating a processing procedure of an information processing device according to an embodiment of the present disclosure. [Figure 10] 1 is a sequence diagram showing a processing procedure of an information processing system according to an embodiment of the present disclosure. [Figure 11] FIG. 13 is a diagram illustrating an example of a process using a result of classification of inhalation. [Figure 12] FIG. 11 is a diagram showing an example of inhalation information of a user. [Figure 13] FIG. 13 is a diagram showing an example of prediction using a user's inhalation. [Figure 14] FIG. 13 is a diagram illustrating an example of a process using a result of classification of inhalation. [Figure 15] FIG. 11 is a diagram showing an example of inhalation information of a user. [Figure 16]FIG. 13 is a diagram showing an example of prediction using a user's inhalation. [Figure 17] FIG. 13 is a diagram illustrating an example of a process using a result of classification of inhalation. [Figure 18] FIG. 11 is a diagram showing an example of inhalation information of a user. [Figure 19] FIG. 13 is a diagram showing an example of prediction using a user's inhalation. [Figure 20] FIG. 13 is a diagram illustrating an example of a process using a result of classification of inhalation. [Figure 21] FIG. 13 is a diagram illustrating an example of a process using a result of classification of inhalation. [Figure 22] FIG. 13 is a diagram illustrating an example of a process using a result of classification of inhalation. [Figure 23] FIG. 11 is a diagram showing an example of inhalation information of a user. [Figure 24] FIG. 13 is a diagram showing an example of prediction using a user's inhalation. [Diagram 25] FIG. 13 is a diagram illustrating an example of the relationship between the length of a user's utterance and the number of characters. [Figure 26] FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the information processing device and the information processing method according to the present application are not limited to these embodiments. In addition, in the following embodiments, the same components are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0010] The present disclosure will be described in the following order. 1. Embodiment 1-1. Overview of information processing according to an embodiment of the present disclosure 1-1-1. Background and effects 1-1-2.Sensor examples 1-1-2-1. Contact type 1-1-2-2. Non-contact type 1-2. Configuration of information processing system according to embodiment 1-3. Configuration of the information processing device according to the embodiment 1-4. Configuration of terminal device according to embodiment 1-5. Information Processing Procedure According to the Embodiment 1-5-1. Processing procedure related to information processing device 1-5-2. Processing procedures related to information processing systems 1-6. Examples of processing using classification results 1-6-1. Examples of abbreviated activation words based on respiratory status 1-6-2. Example of switching between local and cloud voice recognition 1-6-3. Example of changing the voice recognition dictionary 1-6-4. Example of UI change selected according to intake state 1-6-5. Example of changing system response 2. Other embodiments 2-1. Example of a configuration for performing prediction processing on the client side 2-2.Other configuration examples 2-3.Other 3. Effects of the present disclosure 4. Hardware Configuration

[0011] [1. Embodiment] [1-1. Overview of information processing according to an embodiment of the present disclosure] 1 is a diagram illustrating an example of information processing according to an embodiment of the present disclosure. The information processing according to the embodiment of the present disclosure is realized by an information processing system 1 (see FIG. 5) including a server device 100 (see FIG. 6) and a terminal device 10 (see FIG. 8).

[0012] The server device 100 is an information processing device that executes information processing according to the embodiment. The server device 100 predicts whether or not the user will speak after the user inhales, based on inhalation information indicating the user's inhalation. The example of FIG. 1 shows a case where sensor information detected by a breathing sensor 171 (see FIG. 8) of a terminal device 10 used by a user is used as the inhalation information. Note that the example of FIG. 1 shows a case where the inhalation information indicating the user's inhalation is detected by the breathing sensor 171 using a millimeter wave radar, but is not limited to a millimeter wave radar, and any sensor may be used as long as it can detect the user's inhalation information. This point will be described later.

[0013] Fig. 1 will be described in detail below. In the following, a case will be described as an example in which the server device 100 performs a prediction process to predict whether or not the user U1 will speak after the user U1 inhales, using inhalation information indicating the inhalation of the user U1 detected by the terminal device 10. Note that, in Fig. 1, a case will be described as an example in which the server device 100 performs the prediction process (information processing), but the terminal device 10 may also perform the prediction process (information processing). This point will be described later.

[0014] First, in the example of Fig. 1, the server device 100 acquires inhalation information BINF1 indicating the inhalation of the user U1. For example, the server device 100 acquires the inhalation information BINF1 indicating the inhalation of the user U1 from the terminal device 10 used by the user U1. In the example of Fig. 1, the terminal device 10 is a smart speaker. Note that the terminal device 10 is not limited to a smart speaker, and may be any device such as a smartphone, which will be described in detail later.

[0015] The server device 100 performs a prediction process using the inhalation information BINF1 indicating the inhalation of the user U1 (step S1). The server device 100 calculates a score using the inhalation information BINF1. The server device 100 then compares the calculated score with a threshold value to predict whether the user U1 will speak after the inhalation corresponding to the inhalation information BINF1. If the calculated score is greater than the threshold value, the server device 100 predicts that the user U1 will speak after the inhalation corresponding to the inhalation information BINF1.

[0016] The above-mentioned prediction process performed by the server device 100 will be specifically described below with reference to Fig. 2 and Fig. 3. Fig. 2 is a diagram showing an example of inhalation information of a user. Fig. 3 is a diagram showing an example of prediction using inhalation information of a user.

[0017] Graph GR1 in FIG. 2 is a graph showing the relationship between time and inhalation volume, with the horizontal axis showing time and the vertical axis showing inhalation volume. The range between lines LN1 and LN2 in graph GR1 shows the range of normal breathing of user U1 (normal breathing range). For example, the breathing volume corresponding to line LN1 shows the lower limit of the inhalation volume during normal breathing. Also, for example, the breathing volume corresponding to line LN2 shows the upper limit of the inhalation volume during normal breathing. For example, when user U1 is normal, he or she repeats breathing with an inhalation volume within the breathing range between lines LN1 and LN2 (normal breathing range).

[0018] In addition, the current intake amount "B_current" in the graph GR1, which is the intake amount at the current time CR1, indicates the latest intake amount at the time of detection (current time) of the intake information BINF1. In addition, for example, the increase amount "B_increase" in the graph GR1 indicates the change (increase) in the intake amount at the current value CR1.

[0019] For example, the intake information BINF1 includes a current intake volume "B_current", which is the intake volume of the current value CR1 in FIG. 2, and an increase amount "B_increase". Note that the intake information BINF1 may also include a change in the intake volume between the intake start point IS1 immediately before the current value CR1 in FIG. 2 and the current value CR1. In this case, the server device 100 may calculate the increase amount "B_increase" from the change in the intake volume. Note that the increase amount "B_increase" may be information indicating the ratio of the increase amount of the intake volume to the passage of time (increase amount). For example, the increase amount "B_increase" may be a slope (rate of change).

[0020] The server device 100 calculates an utterance prediction score "Score_uttr_pr" which is a score used for utterance prediction, using the current intake amount "B_current" and the increase amount "B_increase" and the following formula (1).

[0021]

number

[0022] In the above formula (1), "a" and "b" indicate predetermined constants. Note that the above formula (1) is an example of the calculation of the speech prediction score "Score_uttr_pr", and various formulas may be used to calculate the speech prediction score "Score_uttr_pr".

[0023] Then, the server device 100 predicts whether the user U1 will speak by using the speech presence / absence prediction threshold "Threshold_uttr_pr" which is a threshold used for predicting the presence / absence of an utterance. The server device 100 predicts whether the user U1 will speak by comparing the speech prediction score "Score_uttr_pr" with the speech presence / absence prediction threshold "Threshold_uttr_pr". That is, the server device 100 classifies the speech type according to the value of the speech prediction score "Score_uttr_pr".

[0024] As shown in the prediction result PD1 in FIG. 3, the server device 100 predicts whether or not the user U1 will speak by comparing the speech prediction score "Score_uttr_pr" with the speech presence / absence prediction threshold "Threshold_uttr_pr."

[0025] In the example of FIG. 3, the server device 100 predicts that the user U1 will speak if the speech prediction score "Score_uttr_pr" is greater than the speech presence / absence prediction threshold "Threshold_uttr_pr". In this case, the server device 100 performs pre-processing required for voice recognition, assuming that there is a high possibility that the user U1 will speak after the current inhalation. The server device 100 performs pre-processing required for voice recognition as soon as it predicts that the user U1 will speak after finishing inhalation. In the example of FIG. 2, the server device 100 performs pre-processing required for voice recognition before the user U1 finishes inhalation (before the maximum inhalation volume "B_max" is reached).

[0026] 3, the server device 100 predicts that the user U1 will not speak if the speech prediction score "Score_uttr_pr" is equal to or smaller than the speech presence / absence prediction threshold "Threshold_uttr_pr". In this case, the server device 100 does not perform pre-activation since it is assumed that speech will not be made after the current inhalation.

[0027] In addition, which of the cases includes the case where the score is equal to the threshold value may be appropriately set. The server device 100 may predict that the user U1 will speak when the speech prediction score "Score_uttr_pr" is equal to or greater than the speech presence / absence prediction threshold "Threshold_uttr_pr", and may predict that the user U1 will not speak when the speech prediction score "Score_uttr_pr" is less than the speech presence / absence prediction threshold "Threshold_uttr_pr".

[0028] In this way, the server device 100 predicts whether the user U1 will speak after the inhalation corresponding to the inhalation information BINF1 by comparing the speech prediction score "Score_uttr_pr" with the speech presence / absence prediction threshold "Threshold_uttr_pr". Note that each threshold such as the speech presence / absence prediction threshold "Threshold_uttr_pr" may be increased or decreased in accordance with changes in the normal breathing range due to changes in the user's motion state, etc.

[0029] As described above, the speech prediction score "Score_uttr_pr" is a value that takes into account the current breathing volume and increment. The server device 100 uses the speech prediction score "Score_uttr_pr" to predict the possibility of speech from the current inhalation volume and increment. This allows the server device 100 to determine the possibility of speech even before the maximum inhalation point is reached, making it possible to use it for advance system preparation.

[0030] Here, the relationship between the user's inhalation and speech will be briefly described with reference to FIG. 4. FIG. 4 is a diagram showing an example of the relationship between the user's inhalation and speech. The waveform shown in FIG. 4 shows an example of the transition of the inhalation amount from the user's normal breathing (steady breathing) to the end of speech. In this way, the inhalation amount increases more than the steady breathing when the user speaks, and transitions to decrease with speech. For example, among the inhalation information related to inhalation, the inhalation increase degree is related to the degree of urgency, etc. Also, for example, among the inhalation information related to inhalation, the maximum inhalation amount is related to the amount of speech text, the speech volume, etc. Also, for example, among the inhalation information related to inhalation, the inhalation decrease degree can be used for predicting the end of speech, etc. Also, for example, among the inhalation information related to inhalation, the inhalation amount at the end of speech can be used for predicting the possibility of continuing speech, etc.

[0031] Also, examples of the information that may be predicted (estimated) from breathing related to speech include the following. For example, before speech, examples include whether the speech is for operating a device, the amount of speech, and the degree of urgency. For example, during speech, examples include whether the speech is nearing the end. For example, after speech, examples include whether there is a possibility that the next speech will follow. Note that the above is just an example, and the server device 100 may predict (estimate) various information related to the user's speech using information on the user's inhalation as described above. In this way, the server device 100 can predict (estimate) various information related to the user's speech by using information on the user's inhalation as described above.

[0032] Moreover, the waveform at a time point beyond the current time point CR1 indicated by the two-dot chain line in the graph GR1 in FIG. 2 indicates a predicted value of the respiration volume. The server device 100 may predict the respiration volume at a time point beyond the current time point CR1 based on the transition of the inhalation volume up to the current time point CR1 and the history of the user U1's past respiration. In this case, the server device 100 can predict the maximum inhalation volume "B_max" indicating the maximum inhalation volume reached by the inhalation corresponding to the current time point CR1. This allows the server device 100 to perform a process using the maximum respiration volume (maximum inhalation volume) described later in advance. The server device 100 may also predict whether the user will speak using the inhalation volume (initial inhalation volume) at the time point when the user finishes exhalation (exhalation) and starts inhalation and the increase in the exhalation volume from that time point. In this case, in the example of FIG. 2, the server device 100 predicts whether the user will speak using the inhalation volume (initial inhalation volume) at the inhalation start time point IS1 and the increase from the inhalation start time point IS1. This enables the server device 100 to predict whether or not the user will speak within a short time after the user starts to exhale.

[0033] The process using the above formula (1) is an example, and the server device 100 may perform prediction processing using various methods other than the above process. For example, the server device 100 may perform prediction processing using a technique related to machine learning. In this case, the server device 100 may perform prediction processing using a model that outputs a score when inhalation information is input. The server device 100 may perform prediction processing using a model that outputs a higher score as the likelihood that the user will speak after an inhalation corresponding to the inhalation information increases. For example, the server device 100 may train the model using training data including a combination of inhalation information indicating the user's inhalation and information indicating the presence or absence of speech after the inhalation, or may acquire the model from an external information processing device.

[0034] From here, we will return to FIG. 1 and continue the explanation. If it is predicted in step S1 that there will be speech, the information processing system 1 starts the startup process (step S11). The information processing system 1 performs pre-processing necessary for voice recognition. For example, the information processing system 1 performs processes such as activating the microphone and establishing a communication connection between the client and the server (connection to the cloud). For example, the server device 100 instructs the terminal device 10 to activate the microphone and activate voice recognition. The terminal device 10 activates the microphone and activates voice recognition. In this way, if it is predicted that there will be speech, the information processing system 1 executes pre-processing necessary for voice recognition by looking ahead.

[0035] Then, the information processing system 1 makes the user U1 aware of the activation (step S12). The information processing system 1 performs a process of clearly indicating the activation of voice recognition and the like by outputting sound or light from the terminal device 10. The terminal device 10 performs a WakeUpResponse (hereinafter also simply referred to as "activation notification") by emitting a notification sound or light indicating the activation of voice recognition. In the example of FIG. 1, the terminal device 10 makes the user U1 aware of the activation by turning on the light source unit 18. This allows the user U1 to recognize that voice input is now possible.

[0036] Then, the user U1 speaks (step S13). For example, the user U1 performs a voice input to request predetermined information to the terminal device 10. The user U1 performs a voice input to request the terminal device 10 to perform a search process.

[0037] Then, the information processing system 1 performs conventional processing (processing of a voice dialogue system) on the input by the user U1 (step S14). For example, the information processing system 1 interprets the user's input by natural language understanding (NLU) and executes a corresponding process (action). The server device 100 or the terminal device 10 interprets the user's input by natural language understanding (NLU) and executes a corresponding process (action). In the example of FIG. 1, the terminal device 10 performs processing of voice output in response to the request of the user U1, "OK. Here's the result...".

[0038] On the other hand, if it is predicted in step S1 that there will be no speech, the information processing system 1 does not start the startup process (step S11). The information processing system 1 does not start voice recognition. The terminal device 10 does not start voice recognition.

[0039] In this way, the information processing system 1 can use the speech prediction result based on the user's inhalation to determine whether or not to activate voice recognition or the like, thereby enabling the omission of activation words based on the pre-speech breathing state.

[0040] [1-1-1. Background and effects] Existing voice user interfaces (UIs) do not take into account the state of the user who is speaking, and therefore may impose various burdens on the user. For example, the user needs to input a start word (by voice) or operate a start button.

[0041] On the other hand, the information processing system 1 detects the breathing state before the user speaks and dynamically changes the voice UI system according to the state. This allows the information processing system 1 to omit the activation word when breathing before making a request to the system. Therefore, the information processing system 1 can improve usability.

[0042] [1-1-2. Sensor examples] 1, a millimeter wave radar is used as an example of the breathing sensor 171 that detects the inhalation information indicating the user's inhalation, but the breathing sensor 171 is not limited to the millimeter wave radar and may be any sensor that can detect the inhalation information of the user. An example of this point will be described below.

[0043] [1-1-2-1. Contact type] 1, the respiration sensor 171 using a millimeter wave radar, that is, a non-contact type sensor is used to detect the inhalation information, but the sensor used to detect (obtain) the inhalation information is not limited to the non-contact type and may be a contact type. An example of a contact type sensor will be described below.

[0044] The respiratory sensor 171 may be a wearable sensor. The respiratory sensor 171 may be a contact sensor of various types, such as a band type, a jacket type, or a mask type.

[0045] When a band-type sensor is used for the breathing sensor 171, the information processing system 1 acquires the amount of breathing displacement from the expansion and contraction of a band wrapped around the chest or abdomen of the user. When a jacket-type sensor is used for the breathing sensor 171, the information processing system 1 embeds the band in a jacket worn by the user. Furthermore, by providing sensors in multiple locations (directions), the accuracy of breathing detection can be improved.

[0046] Furthermore, when an acceleration sensor is used for the respiration sensor 171, the information processing system 1 may estimate the respiration volume by observing the movement of the chest using an acceleration sensor mounted on a wearable device such as a neck-hanging device or a smartphone worn on the upper body of the user. Furthermore, when a mask-type sensor is used for the respiration sensor 171, the information processing system 1 detects the speed of exhalation and inhalation using an air flow sensor or air pressure sensor mounted on the mask, and estimates the depth and period from the accumulated displacement.

[0047] Furthermore, a VR (Virtual Reality) headset that covers the user's mouth may be used for the breathing sensor 171. In this case, the breathing sensor 171 that performs breathing sensing with a noise blocking microphone allows disadvantages in the real world to be ignored since VR is being used. Furthermore, when a close microphone is used for the breathing sensor 171, the information processing system 1 recognizes the sound of exhaled breath with the close microphone, recognizes the amount of change in exhaled breath over time, and estimates the depth and speed of breathing. For example, the information processing system 1 recognizes the sound of noise generated when the exhaled breath hits the microphone with the close microphone, recognizes the amount of change in exhaled breath over time, and estimates the depth and speed of breathing.

[0048] [1-1-2-2. Non-contact type] Furthermore, the non-contact sensor is not limited to the millimeter wave radar, and various non-contact sensors may be used for the respiration sensor 171. Below, examples of non-contact sensors other than the millimeter wave radar will be described.

[0049] The respiration sensor 171 may use an image sensing method, a method for detecting respiration from the temperature around the nose, a proximity sensor, or a radar other than a millimeter wave radar.

[0050] When image sensing is used for the breathing sensor 171, the information processing system 1 recognizes the amount of change over time of inhalation and exhalation, which have different temperatures, using a thermo camera, and estimates the depth, cycle, and speed of breathing. The information processing system 1 may also recognize the amount of change over time of exhalation by image sensing, which turns white when it is cold, and estimate the depth, cycle, and speed of breathing.

[0051] For example, the following disclosure is made regarding a technique for detecting respiration from the temperature around the nose. - Non-intrusive breathing sensing method<https: / / shingi.jst.go.jp / past_abst / abst / p / 09 / 919 / tama2.pdf>

[0052] For example, the following disclosure is made regarding breath detection using a proximity sensor. Capacitive film-type proximity sensor that monitors human movement and breathing<https: / / www.aist.go.jp / aist_j / press_release / pr2016 / pr20160125 / pr20160125.html>

[0053] For example, the following disclosure is made regarding respiration detection using radar. Heart rate and respiration detection sensor "GZS-350 series"<https: / / www.ipros.jp / product / detail / 2000348329 / >

[0054] 1, when a millimeter wave radar is used as the breathing sensor 171, the information processing system 1 detects the movement of the user's chest from the phase difference of the reception signal of the millimeter wave radar and estimates the breathing volume. For example, the terminal device 10 uses the sensor information detected by the breathing sensor 171 to detect the movement of the user's chest from the phase difference of the reception signal of the millimeter wave radar and estimates the breathing volume, thereby generating the inhalation information of the user. Then, the terminal device 10 transmits the generated inhalation information of the user to the server device 100.

[0055] Alternatively, the server device 100 may generate the inhalation information of the user. In this case, the terminal device 10 transmits sensor information detected by the breathing sensor 171 to the server device 100. The server device 100 may then use the received sensor information to detect the movement of the user's chest from the phase difference of the received signal of the millimeter wave radar and estimate the breathing volume to generate the inhalation information of the user.

[0056] The above sensors are merely examples of sensors used to acquire inhalation information, and any sensor may be used as long as it is capable of acquiring inhalation information. The information processing system 1 may use any sensor to detect inhalation information indicating the user's inhalation.

[0057] The sensor unit 17 of the terminal device 10 has at least one of the above-mentioned sensors, and detects the inhalation information by the sensor. The information processing system 1 may generate the inhalation information by using the sensor information detected by the sensor of the sensor unit 17. For example, the terminal device 10 or the server device 100 may generate the inhalation information by using the sensor information (point cloud data) detected by the breathing sensor 171 (millimeter wave radar). In this case, the terminal device 10 or the server device 100 may generate the inhalation information from the sensor information (point cloud data) detected by the breathing sensor 171 (millimeter wave radar) by appropriately using various techniques.

[0058] [1-2. Configuration of information processing system according to embodiment] An information processing system 1 shown in FIG. 5 will be described. As shown in FIG. 5, the information processing system 1 includes a terminal device 10 and a server device 100. The terminal device 10 and the server device 100 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other by wire or wirelessly. FIG. 5 is a diagram showing an example of the configuration of an information processing system according to an embodiment. Note that the information processing system 1 shown in FIG. 5 may include a plurality of terminal devices 10 and a plurality of server devices 100.

[0059] The server device 100 is a computer that predicts whether the user will speak after the user inhales, based on inhalation information indicating the user's inhalation. The server device 100 classifies the user's inhalation, based on the user's inhalation information. The server device 100 is also a computer that transmits various types of information to the terminal device 10. The server device 100 is a server device used to provide services related to various functions.

[0060] Furthermore, the server device 100 may have software modules such as speech signal processing, speech recognition, speech semantic analysis, and dialogue control. The server device 100 may have a speech recognition function. For example, the server device 100 may have a natural language understanding (NLU) or automatic speech recognition (ASR) function. For example, the server device 100 may estimate information about a user's intent or an entity (target) from input information by the user's speech. The server device 100 functions as a speech recognition server having the functions of natural language understanding and automatic speech recognition.

[0061] The terminal device 10 is a terminal device that detects inhalation information indicating the inhalation of a user by a sensor. For example, the terminal device 10 detects inhalation information indicating the inhalation of a user by a breathing sensor 171. The terminal device 10 is an information processing device that transmits the inhalation information of the user to a server device such as the server device 100. The terminal device 10 may also have a voice recognition function such as natural language understanding or automatic voice recognition. For example, the terminal device 10 may estimate information related to the user's intent (intention) or entity (target) from input information by the user's speech.

[0062] The terminal device 10 is a device used by a user. The terminal device 10 accepts input from the user. The terminal device 10 accepts voice input by the user's speech and input by the user's operation. The terminal device 10 displays information according to the user's input. The terminal device 10 may be any device as long as it can realize the processing in the embodiment. The terminal device 10 may be any device as long as it has a function of detecting the user's inhalation information and transmitting it to the server device 100. For example, the terminal device 10 may be a device such as a smart speaker, a television, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, or a PDA (Personal Digital Assistant). The terminal device 10 may be a wearable device worn by a user. For example, the terminal device 10 may be a wristwatch terminal, a glasses terminal, or the like.

[0063] [1-3. Configuration of the information processing device according to the embodiment] Next, a configuration of the server device 100, which is an example of an information processing device that executes information processing according to an embodiment, will be described. Fig. 6 is a diagram illustrating an example of the configuration of the server device 100 according to an embodiment of the present disclosure.

[0064] 6, the server device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The server device 100 may also include an input unit (e.g., a keyboard, a mouse, etc.) for receiving various operations from an administrator of the server device 100, and a display unit (e.g., a liquid crystal display, etc.) for displaying various information.

[0065] The communication unit 110 is realized by, for example, a network interface card (NIC) or the like. The communication unit 110 is connected to a network N (see FIG. 5) by wire or wirelessly, and transmits and receives information to and from other information processing devices such as the terminal device 10. The communication unit 110 may also transmit and receive information to and from a user terminal (not shown) used by a user.

[0066] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 6, the storage unit 120 according to the embodiment includes an intake information storage unit 121, a user information storage unit 122, a threshold information storage unit 123, and a function information storage unit 124.

[0067] The storage unit 120 stores various information in addition to the above. The storage unit 120 stores information on a voice recognition application (program) that realizes a voice recognition function. For example, the server device 100 can execute voice recognition by starting a voice recognition application (also simply called "voice recognition"). The storage unit 120 stores various information used for voice recognition. The storage unit 120 stores information on a dictionary (voice recognition dictionary) used in the voice recognition dictionary. The storage unit 120 stores information on a plurality of voice recognition dictionaries. The storage unit 120 stores information on a voice recognition dictionary for long sentences (dictionary for long sentences), a voice recognition dictionary for Chinese sentences (dictionary for Chinese sentences), a voice recognition dictionary for short sentences (dictionary for words and phrases), etc.

[0068] The inhalation information storage unit 121 according to the embodiment stores various information related to the user's inhalation. The inhalation information storage unit 121 stores various information such as inhalation information of each user in association with the identification information (user ID) of each user. The inhalation information storage unit 121 stores inhalation information including an increase in the user's inhalation. The inhalation information storage unit 121 stores inhalation information including the inhalation volume of the user's inhalation. The inhalation information storage unit 121 stores inhalation information including an initial inhalation volume at the start of the user's inhalation. The inhalation information storage unit 121 stores inhalation information including a maximum inhalation volume of the user's inhalation. The inhalation information storage unit 121 stores time point information indicating the start time of the user's speech after inhalation. The inhalation information storage unit 121 stores speech information including the length and number of characters of the user's speech after inhalation.

[0069] The inhalation information storage unit 121 may store various information according to the purpose, not limited to the above. For example, the inhalation information storage unit 121 may store information related to the user's breathing, not limited to the inhalation information. The inhalation information storage unit 121 may store information related to the user's inhalation. For example, the inhalation information storage unit 121 may store various information required for generating the graphs GR1 to GR5. For example, the inhalation information storage unit 121 may store various information shown in the graphs GR1 to GR5.

[0070] The user information storage unit 122 according to the embodiment stores various pieces of information related to users. For example, the user information storage unit 122 stores various pieces of information such as attribute information of each user.

[0071] The user information storage unit 122 stores information about the user, such as a user ID, age, sex, and place of residence. For example, the user information storage unit 122 stores information about the user U1, such as the age, sex, and place of residence of the user U1, in association with a user ID “U1” that identifies the user U1.

[0072] The user information storage unit 122 also stores information for identifying a device (such as a television or a smartphone) used by each user in association with the user. The user information storage unit 122 also stores information for identifying a terminal device 10 used by each user (such as a terminal ID) in association with the user.

[0073] The user information storage unit 122 may store various information according to the purpose, not limited to the above. For example, the user information storage unit 122 may store other demographic attribute information and psychographic attribute information in addition to age and gender. For example, the user information storage unit 122 may store information such as name, home, place of work, interests, family structure, income, and lifestyle.

[0074] The threshold information storage unit 123 according to the embodiment stores various information related to thresholds. The threshold information storage unit 123 stores various information related to thresholds used in prediction processing and classification processing. Fig. 7 is a diagram illustrating an example of the threshold information storage unit according to the embodiment. The threshold information storage unit 123 illustrated in Fig. 7 includes items such as "threshold ID", "purpose", "threshold name", and "value".

[0075] "Threshold ID" indicates identification information for identifying a threshold. "Use" indicates the use of the threshold. "Threshold name" indicates the name (character string) of the threshold (variable) used as the threshold identified by the corresponding threshold ID. "Value" indicates the specific value of the threshold identified by the corresponding threshold ID.

[0076] In the example of FIG. 7, the use of the threshold (threshold TH1) identified by the threshold ID "TH1" is indicated as speech presence / absence prediction. The threshold TH1 is indicated as being used with the threshold name "Threshold_uttr_pr". The value of threshold TH1 is indicated as "VL1". Note that in FIG. 7, the value is indicated as an abstract code such as "VL1", but the value is assumed to be a concrete numerical value such as "0.5" or "1.8".

[0077] 7 illustrates only the threshold TH1 used for speech presence / absence prediction, various thresholds used for classification processing are also stored in the threshold information storage unit 123. For example, various thresholds corresponding to Threshold_uttr, Threshold_ask, and the like shown in FIG. 13 are stored in the threshold information storage unit 123. Furthermore, the threshold information storage unit 123 may store various information according to the purpose, without being limited to the above.

[0078] The function information storage unit 124 according to the embodiment stores various information related to functions. The function information storage unit 124 stores information related to each function executed in response to a user's input. The function information storage unit 124 stores information related to an input required for executing a function. The function information storage unit 124 stores input items required for executing each function. Note that the function information storage unit 124 is not limited to the above, and may store various information according to the purpose.

[0079] Returning to Fig. 6, the explanation will be continued. The control unit 130 is realized, for example, by a central processing unit (CPU), a micro processing unit (MPU), or the like executing a program (for example, an information processing program according to the present disclosure) stored inside the server device 100 using a random access memory (RAM) or the like as a working area. The control unit 130 is also realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0080] As shown in Fig. 6, the control unit 130 has an acquisition unit 131, a prediction unit 132, a selection unit 133, an execution unit 134, and a transmission unit 135, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Fig. 6, and may be other configurations as long as they perform the information processing described below. Also, the connection relationship between the processing units in the control unit 130 is not limited to the connection relationship shown in Fig. 6, and may be other connection relationships.

[0081] The acquisition unit 131 acquires various information. The acquisition unit 131 acquires various information from an external information processing device. The acquisition unit 131 acquires various information from the terminal device 10. The acquisition unit 131 acquires various information from the terminal device 10, that is, information detected by the sensor unit 17 of the terminal device 10. The acquisition unit 131 acquires various information from the terminal device 10, that is, information detected by the respiratory sensor 171 of the sensor unit 17.

[0082] The acquisition unit 131 acquires various pieces of information from the storage unit 120. The acquisition unit 131 acquires various pieces of information from the inhalation information storage unit 121, the user information storage unit 122, the threshold information storage unit 123, and the function information storage unit 124. The acquisition unit 131 acquires various pieces of information predicted by the prediction unit 132. The acquisition unit 131 acquires various pieces of information selected by the selection unit 133.

[0083] The acquisition unit 131 acquires inhalation information indicating the user's inhalation. The acquisition unit 131 acquires inhalation information including an increase in the user's inhalation. The acquisition unit 131 acquires inhalation information including the inhalation volume of the user's inhalation. The acquisition unit 131 acquires inhalation information including an initial inhalation volume at the start of the user's inhalation. The acquisition unit 131 acquires inhalation information including a maximum inhalation volume of the user's inhalation. The acquisition unit 131 acquires time point information indicating the start time of the user's speech after the user's inhalation. The acquisition unit 131 acquires speech information including the length and number of characters of the speech after the user's inhalation. For example, the acquisition unit 131 acquires inhalation information BINF1 indicating the inhalation of the user U1.

[0084] The prediction unit 132 predicts various information. The prediction unit 132 classifies various information. The prediction unit 132 calculates various information. The prediction unit 132 determines various information. The prediction unit 132 makes various judgments. The prediction unit 132 judges various information. For example, the prediction unit 132 predicts various information based on information from an external information processing device or information stored in the storage unit 120. The prediction unit 132 predicts various information based on information from another information processing device such as the terminal device 10. The prediction unit 132 predicts various information based on information stored in the intake information storage unit 121, the user information storage unit 122, the threshold information storage unit 123, or the function information storage unit 124. The prediction unit 132 classifies various information. For example, the prediction unit 132 classifies various information based on information from an external information processing device or information stored in the storage unit 120.

[0085] The prediction unit 132 predicts various pieces of information based on various pieces of information acquired by the acquisition unit 131. The prediction unit 132 predicts various pieces of information based on various pieces of information selected by the selection unit 133. The prediction unit 132 makes various decisions based on the predictions. The prediction unit 132 makes various decisions based on the information acquired by the acquisition unit 131. The prediction unit 132 calculates a score based on the inhalation information. The prediction unit 132 calculates a score used for predicting the user's speech based on the inhalation information.

[0086] The prediction unit 132 predicts whether the user will speak after the user inhales, based on the inhalation information acquired by the acquisition unit 131. The prediction unit 132 predicts whether the user will speak after the user inhales, based on the increase amount. The prediction unit 132 predicts whether the user will speak after the user inhales, based on the inhalation amount, and the prediction unit 132 predicts whether the user will speak after the user inhales, based on the initial inhalation amount.

[0087] The prediction unit 132 calculates a score using the inhalation information and a predetermined formula. The prediction unit 132 predicts whether the user will speak after inhalation using the score calculated based on the inhalation information. The prediction unit 132 compares the score with a threshold, and predicts whether the user will speak after inhalation based on the comparison result. If the comparison result between the score and the threshold satisfies a predetermined condition, the prediction unit 132 predicts that the user will speak after inhalation. If the score is greater than the threshold, the prediction unit 132 predicts that the user will speak after inhalation. For example, the prediction unit 132 calculates a speech prediction score "Score_uttr_pr", which is a score used for speech prediction, using the current inhalation volume "B_current", the increase amount "B_increase", and formula (1). For example, the prediction unit 132 predicts that the user U1 will speak if the speech prediction score "Score_uttr_pr" is greater than the speech presence / absence prediction threshold "Threshold_uttr_pr". For example, when the speech prediction score "Score_uttr_pr" is equal to or smaller than the speech presence / absence prediction threshold "Threshold_uttr_pr", the prediction unit 132 predicts that the user U1 will not speak.

[0088] The prediction unit 132 classifies the user's inhalation based on the inhalation information. The prediction unit 132 classifies the user's inhalation based on the maximum inhalation amount. The prediction unit 132 classifies the user's inhalation based on the interval between the time of the maximum inhalation amount and the time of the start of the speech. The prediction unit 132 classifies the user's inhalation based on the length of the speech and the number of characters.

[0089] The prediction unit 132 classifies the user's inhalation into one of a plurality of types including at least a requested inhalation and a non-requested inhalation. The prediction unit 132 classifies the user's inhalation into one of a plurality of types including at least a long sentence inhalation and a short sentence inhalation. The prediction unit 132 classifies the user's inhalation into one of a plurality of types including at least a normal processing desired inhalation and a short processing desired inhalation.

[0090] The selection unit 133 selects various information. The selection unit 133 extracts various information. The selection unit 133 identifies various information. The selection unit 133 selects various information based on information from an external information processing device or information stored in the storage unit 120. The selection unit 133 selects various information based on information from another information processing device such as the terminal device 10. The selection unit 133 selects various information based on information stored in the intake information storage unit 121, the user information storage unit 122, the threshold information storage unit 123, or the function information storage unit 124.

[0091] The selection unit 133 selects various information based on the various information acquired by the acquisition unit 131. The selection unit 133 selects various information based on the various information predicted by the prediction unit 132. The selection unit 133 selects various information based on the processing executed by the execution unit 134.

[0092] The selection unit 133 performs a selection process according to the classification result by the prediction unit 132. The selection unit 133 selects a process to be executed according to the classification result by the prediction unit 132. The selection unit 133 selects information to be used in a process for a user's utterance according to the classification result by the prediction unit 132.

[0093] The execution unit 134 executes various processes. The execution unit 134 decides to execute various processes. The execution unit 134 executes various processes based on information from an external information processing device. The execution unit 134 executes various processes based on information stored in the storage unit 120. The execution unit 134 executes various processes based on information stored in the intake information storage unit 121, the user information storage unit 122, the threshold information storage unit 123, and the function information storage unit 124.

[0094] The execution unit 134 executes various processes based on the various information acquired by the acquisition unit 131. The execution unit 134 executes various processes based on the various information predicted by the prediction unit 132. The execution unit 134 executes various processes based on the various information selected by the selection unit 133.

[0095] The execution unit 134 generates various information. The execution unit 134 generates various information based on information from an external information processing device or information stored in the storage unit 120. The execution unit 134 generates various information based on information from another information processing device such as the terminal device 10. The execution unit 134 generates various information based on information stored in the intake information storage unit 121, the user information storage unit 122, the threshold information storage unit 123, and the function information storage unit 124.

[0096] The execution unit 134 executes processing according to the prediction result by the prediction unit 132. When the prediction unit 132 predicts that the user will speak after inhalation, the execution unit 134 executes pre-processing related to voice recognition. The execution unit 134 executes the pre-processing before the user finishes inhaling.

[0097] The execution unit 134 executes activation of voice recognition when the prediction unit 132 predicts that the user will speak after inhalation. When the prediction unit 132 predicts that the user will speak after inhalation, the execution unit 134 executes pre-processing to activate voice recognition before the user's inhalation ends. For example, when it is predicted that the user U1 will speak, the execution unit 134 executes pre-processing necessary for voice recognition, assuming that there is a high possibility that speech will be made after the current inhalation.

[0098] When the prediction unit 132 predicts that the user will speak after inhalation, the execution unit 134 executes a pre-processing to instruct the terminal device 10 to start voice recognition. When the prediction unit 132 predicts that the user will speak after inhalation, the execution unit 134 executes a pre-processing to instruct the terminal device 10 to start voice recognition before the user finishes inhaling.

[0099] The transmitting unit 135 transmits various information. The transmitting unit 135 transmits various information to an external information processing device. The transmitting unit 135 provides various information to an external information processing device. For example, the transmitting unit 135 transmits various information to another information processing device such as the terminal device 10. The transmitting unit 135 provides information stored in the memory unit 120. The transmitting unit 135 transmits information stored in the memory unit 120.

[0100] The transmission unit 135 provides various information based on information from other information processing devices such as the terminal device 10. The transmission unit 135 provides various information based on information stored in the storage unit 120. The transmission unit 135 provides various information based on information stored in the inhalation information storage unit 121, the user information storage unit 122, the threshold information storage unit 123, and the function information storage unit 124.

[0101] The transmission unit 135 transmits, to the terminal device 10, information indicating a function to be executed by the terminal device 10. The transmission unit 135 transmits, to the terminal device 10, information indicating a function determined to be executed by the execution unit 134. The transmission unit 135 transmits various information to the terminal device 10 in response to an instruction from the execution unit 134. The transmission unit 135 transmits information instructing the terminal device 10 to start a voice recognition application. When the prediction unit 132 predicts that the user will speak after inhalation, the transmission unit 135 transmits information instructing the terminal device 10 to start voice recognition. When the prediction unit 132 predicts that the user will speak after inhalation, the transmission unit 135 transmits information instructing the terminal device 10 to start voice recognition before the user's inhalation ends.

[0102] [1-4. Configuration of terminal device according to embodiment] Next, a configuration of the terminal device 10, which is an example of an information processing device that executes information processing according to an embodiment, will be described. Fig. 8 is a diagram illustrating an example of the configuration of the terminal device according to an embodiment of the present disclosure.

[0103] As shown in FIG. 8, the terminal device 10 includes a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, a control unit 15, a display unit 16, a sensor unit 17, and a light source unit 18.

[0104] The communication unit 11 is realized by, for example, a NIC, a communication circuit, etc. The communication unit 11 is connected to a network N (such as the Internet) by wire or wirelessly, and transmits and receives information to and from other devices such as the server device 100 via the network N.

[0105] The input unit 12 accepts various inputs. The input unit 12 accepts detection by the sensor unit 17 as input. The input unit 12 accepts input of inhalation information indicating the user's inhalation. The input unit 12 accepts input of inhalation information detected by the sensor unit 17. The input unit 12 accepts input of inhalation information detected by the breathing sensor 171. The input unit 12 accepts input of inhalation information based on point cloud data detected by the breathing sensor 171. The input unit 12 accepts input of user's speech information. The input unit 12 accepts input of inhalation information of the user who performs input by physical movement. The input unit 12 accepts the user's gestures and gaze as input.

[0106] Various operations are input to the input unit 12 by the user. The input unit 12 receives sound as input by a sensor unit 17 having a function of detecting sound. The input unit 12 receives voice information detected by a microphone (sound sensor) that detects sound as input information. The input unit 12 receives voice uttered by the user as input information.

[0107] The input unit 12 may also receive an operation (user operation) on the terminal device 10 used by the user as an operation input by the user. The input unit 12 may receive information on a user operation using a remote controller via the communication unit 11. The input unit 12 may also have a button provided on the terminal device 10, or a keyboard or a mouse connected to the terminal device 10.

[0108] For example, the input unit 12 may have a touch panel that can realize the same functions as a remote control, a keyboard, or a mouse. In this case, various information is input to the input unit 12 via the display unit 16. The input unit 12 accepts various operations from the user via the display screen by the function of the touch panel realized by various sensors. That is, the input unit 12 accepts various operations from the user via the display unit 16 of the terminal device 10. For example, the input unit 12 accepts operations such as a user's designated operation via the display unit 16 of the terminal device 10. For example, the input unit 12 functions as a reception unit that accepts user operations by the function of the touch panel. In this case, the input unit 12 and the reception unit 153 may be integrated. Note that, in tablet terminals, the electrostatic capacitance method is mainly used as a method for detecting user operations by the input unit 12, but any method may be used as long as it can detect user operations and realize the function of a touch panel, such as a resistive film method, a surface acoustic wave method, an infrared method, or an electromagnetic induction method.

[0109] For example, the input unit 12 accepts an utterance of the user U1 as an input. The input unit 12 accepts an utterance of the user U1 detected by the sensor unit 17 as an input. The input unit 12 accepts an utterance of the user U1 detected by the sound sensor of the sensor unit 17 as an input.

[0110] The output unit 13 outputs various information. The output unit 13 has a function of outputting sound. For example, the output unit 13 has a speaker that outputs sound. The output unit 13 outputs various information by sound in response to control by the execution unit 152. The output unit 13 outputs information by sound to the user. The output unit 13 outputs information displayed on the display unit 16 by sound.

[0111] The memory unit 14 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The memory unit 14 stores information on a voice recognition application (program) that realizes a voice recognition function. For example, the terminal device 10 can execute voice recognition by starting a voice recognition application. The memory unit 14 stores various information used to display information. The memory unit 14 stores various information used for voice recognition. The memory unit 14 stores information on a dictionary used in a voice recognition dictionary (voice recognition dictionary).

[0112] Returning to Fig. 8, the description will be continued. The control unit 15 is realized, for example, by a CPU, an MPU, or the like executing a program (for example, an information processing program according to the present disclosure) stored inside the terminal device 10 using a RAM or the like as a working area. The control unit 15 may also be realized, for example, by an integrated circuit such as an ASIC or an FPGA.

[0113] 8, control unit 15 has receiving unit 151, executing unit 152, accepting unit 153, and transmitting unit 154, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of control unit 15 is not limited to the configuration shown in FIG. 8, and may be other configurations as long as they perform information processing described below.

[0114] The receiving unit 151 receives various information. The receiving unit 151 receives various information from an external information processing device. The receiving unit 151 receives various information from another information processing device such as the server device 100.

[0115] The receiving unit 151 receives information instructing the start of voice recognition from the server device 100. The receiving unit 151 receives information instructing the start of a voice recognition application from the server device 100.

[0116] The receiving unit 151 receives execution instructions for various functions from the server device 100. For example, the receiving unit 151 receives information designating a function from the server device 100 as an execution instruction for the function. The receiving unit 151 receives content. The receiving unit 151 receives content to be displayed from the server device 100.

[0117] The execution unit 152 executes various processes. The execution unit 152 decides to execute various processes. The execution unit 152 executes various processes based on information from an external information processing device. The execution unit 152 executes various processes based on information from the server device 100. The execution unit 152 executes various processes in response to instructions from the server device 100. The execution unit 152 executes various processes based on information stored in the storage unit 14. The execution unit 152 starts up voice recognition.

[0118] The execution unit 152 controls various outputs. The execution unit 152 controls audio output by the output unit 13. The execution unit 152 controls the lighting of the light source unit 18. The execution unit 152 controls various displays. The execution unit 152 controls the display of the display unit 16. The execution unit 152 controls the display of the display unit 16 in response to reception by the receiving unit 151. The execution unit 152 controls the display of the display unit 16 based on the information received by the receiving unit 151. The execution unit 152 controls the display of the display unit 16 based on the information accepted by the accepting unit 153. The execution unit 152 controls the display of the display unit 16 in response to acceptance by the accepting unit 153.

[0119] The reception unit 153 receives various types of information. The reception unit 153 receives input by the user via the input unit 12. The reception unit 153 receives speech by the user as input. The reception unit 153 receives operations by the user. The reception unit 153 receives operations by the user on information displayed by the display unit 16. The reception unit 153 receives character input by the user.

[0120] The transmission unit 154 transmits various information to an external information processing device. For example, the transmission unit 154 transmits various information to another information processing device such as the terminal device 10. The transmission unit 154 transmits information stored in the storage unit 14.

[0121] The transmitting unit 154 transmits various types of information based on information from other information processing devices such as the server device 100. The transmitting unit 154 transmits various types of information based on information stored in the storage unit .

[0122] The transmission unit 154 transmits sensor information detected by the sensor unit 17 to the server device 100. The transmission unit 154 transmits inhalation information of the user U1 detected by the breathing sensor 171 of the sensor unit 17 to the server device 100.

[0123] The transmission unit 154 transmits input information input by a user to the server device 100. The transmission unit 154 transmits input information input by voice by the user to the server device 100. The transmission unit 154 transmits input information input by a user's operation to the server device 100.

[0124] The transmitting unit 154 transmits inhalation information indicating the user's inhalation to the server device 100. The transmitting unit 154 transmits inhalation information including an increased amount of the user's inhalation to the server device 100. The transmitting unit 154 transmits inhalation information including an inhalation amount of the user's inhalation to the server device 100. The transmitting unit 154 transmits inhalation information including an initial inhalation amount at the start of the user's inhalation to the server device 100. The transmitting unit 154 transmits inhalation information including a maximum inhalation amount of the user's inhalation to the server device 100. The transmitting unit 154 transmits time point information indicating a start time point of the user's speech after the user's inhalation to the server device 100. The transmitting unit 154 transmits speech information including a length and number of characters of the user's speech after the user's inhalation to the server device 100.

[0125] The display unit 16 is provided in the terminal device 10 and displays various information. The display unit 16 is realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 16 may be realized by any means as long as it is capable of displaying information provided by the server device 100. The display unit 16 displays various information in response to control by the execution unit 152.

[0126] The display unit 16 displays the content. The display unit 16 displays the content received by the receiving unit 151.

[0127] The sensor unit 17 detects predetermined information. The sensor unit 17 detects inhalation information of the user. The sensor unit 17 has a breathing sensor 171 as a means for detecting inhalation information indicating the inhalation of the user. The sensor unit 17 detects the inhalation information by the breathing sensor 171.

[0128] The sensor unit 17 detects inhalation information by the breathing sensor 171. The sensor unit 17 detects inhalation information by the breathing sensor 171 using a millimeter wave radar. The sensor unit 17 is not limited to a millimeter wave radar, and may have a breathing sensor 171 of any configuration as long as it can detect the user's inhalation information. The breathing sensor 171 may be an image sensor. The breathing sensor 171 may be a wearable sensor. The breathing sensor 171 may be either a contact type or a non-contact type sensor.

[0129] The sensor unit 17 is not limited to the above and may have various sensors. The sensor unit 17 may have a sensor (position sensor) that detects position information such as a GPS (Global Positioning System) sensor. The sensor unit 17 is not limited to the above and may have various sensors.

[0130] The light source unit 18 has a light source such as an LED (Light Emitting Diode). The light source unit 18 emits light. The light source unit 18 realizes a desired lighting mode. The light source unit 18 realizes a desired lighting mode in response to control by the execution unit 152. The light source unit 18 turns on in response to control by the execution unit 152. The light source unit 18 turns off in response to control by the execution unit 152. The light source unit 18 blinks in response to control by the execution unit 152.

[0131] [1-5. Information processing procedure according to the embodiment] Next, the procedure of various information processes according to the embodiment will be described with reference to FIG. 9 and FIG.

[0132] [1-5-1. Processing procedure related to information processing device] First, a process flow of the information processing device according to the embodiment of the present disclosure will be described with reference to Fig. 9. Fig. 9 is a flowchart showing a process procedure of the information processing device according to the embodiment of the present disclosure. Specifically, Fig. 9 is a flowchart showing the procedure of information processing by the server device 100.

[0133] 9, the server device 100 acquires inhalation information indicating an inhalation of a user (step S101). Then, the server device 100 predicts whether the user will speak after the user inhales based on the inhalation information (step S102).

[0134] [1-5-2. Processing procedures related to information processing systems] Next, a process flow related to the information processing system according to the embodiment of the present disclosure will be described with reference to Fig. 10. Fig. 10 is a sequence diagram showing a process procedure of the information processing system according to the embodiment of the present disclosure.

[0135] 10, the terminal device 10 detects inhalation information indicating the inhalation of the user (step S201). For example, the terminal device 10 acquires the inhalation information of the user detected by the breathing sensor 171. Then, the terminal device 10 transmits the inhalation information indicating the inhalation of the user to the server device 100 (step S202).

[0136] The server device 100 predicts whether the user will speak after the user inhales, based on the inhalation information acquired from the terminal device 10 (step S203). In the example of Fig. 10, the server device 100 predicts that the user will speak after the user inhales, based on the inhalation information acquired from the terminal device 10.

[0137] Then, since the server device 100 predicts that the user will speak, it instructs the terminal device 10 to start voice recognition (step S204). The server device 100 instructs the terminal device 10 to start voice recognition by transmitting information instructing the terminal device 10 to start voice recognition. Then, the terminal device 10 executes voice recognition start processing in response to the instruction from the server device 100 (step S205). Then, the terminal device 10 outputs a sound corresponding to the start of voice recognition (step S206). The terminal device 10 outputs a sound corresponding to the start of voice recognition or emits light. For example, the terminal device 10 outputs a WakeUpResponse (start notification) corresponding to the start of voice recognition by sound from the output unit 13. Also, for example, the terminal device 10 executes a point or the like corresponding to the start of voice recognition by the light source unit 18.

[0138] [1-6. Examples of processing using classification results] The server device 100 may perform various processes other than the above-mentioned prediction process. For example, the server device 100 may perform a classification process for classifying the inhalation of the user. This will be described below. In the following description, the same points as those in FIG. 1 will be omitted as appropriate.

[0139] [1-6-1. Examples of omission of activation words based on respiratory status] An example of the classification process will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of a process using the classification result of inspiration. Fig. 11 shows an example of omission of activation words depending on the respiratory state.

[0140] 11, the server device 100 acquires inhalation information indicating an inhalation of the user U1 before speaking. For example, the server device 100 acquires the inhalation information indicating an inhalation of the user U1 from the terminal device 10 used by the user U1.

[0141] The server device 100 performs a classification process using inhalation information indicating the inhalation of the user U1 (step S301). The server device 100 calculates a score using the inhalation information. The server device 100 then classifies the inhalation of the user U1 by comparing the calculated score with a threshold value. The server device 100 classifies the inhalation of the user U1 based on the magnitude relationship between the calculated score and each threshold value.

[0142] The above-mentioned classification process performed by the server device 100 will be specifically described below with reference to Fig. 12 and Fig. 13. Fig. 12 is a diagram showing an example of inhalation information of a user. Fig. 13 is a diagram showing an example of prediction using inhalation information of a user.

[0143] Graph GR2 in Fig. 12 is a graph showing the relationship between time and the amount of intake air, with the horizontal axis showing time and the vertical axis showing the amount of intake air. Note that, regarding graph GR2, explanations of the same points as those of graph GR1 in Fig. 2 will be omitted.

[0144] For example, the maximum breathing volume "B_max" in graph GR2 indicates the maximum breathing volume (maximum breathing volume) reached by inhalation before speaking. The maximum breathing-to-speech time "T_bmax_uttr" indicates the interval (pre-speech time) from the time when the maximum breathing volume is reached to the time when speaking starts (speech start time). In the example of FIG. 12, the maximum breathing-to-speech time "T_bmax_uttr" indicates the difference between time t2 indicating the time when speaking starts (speech start time) and time t1 indicating the time when the maximum breathing volume is reached.

[0145] For example, the increase amount "B_increase" in the graph GR2 indicates the change (increase) in the inhalation volume before reaching the maximum respiratory volume "B_max." Note that the increase amount "B_increase" may be the change (increase) in the inhalation volume at the time when the inhalation information is acquired (the current time).

[0146] For example, the inhalation information includes the increase amount "B_increase", the maximum breathing volume "B_max", and the maximum inhalation-utterance time "T_bmax_uttr" in FIG. 12. The inhalation information does not need to include the maximum inhalation-utterance time "T_bmax_uttr". In this case, the score may be calculated with "c*(1 / T_bmax_uttr)", which is the term (third term) related to the maximum inhalation-utterance time "T_bmax_uttr", set to "0". Furthermore, the maximum breathing volume "B_max" may not be included. In this case, the server device 100 may predict the maximum breathing volume "B_max" as described in FIG. 2, and calculate the score using the predicted maximum breathing volume "B_max".

[0147] The server device 100 calculates the speech score "Score_uttr" which is a score used for speech prediction, using the increase amount "B_increase", the maximum respiratory volume "B_max", the maximum inspiration-to-speech time "T_bmax_uttr" and the following formula (2).

[0148]

number

[0149] In the above formula (2), "a," "b," and "c" represent predetermined constants. Note that the above formula (2) is an example of the calculation of the speech score "Score_uttr," and various formulas may be used to calculate the speech score "Score_uttr."

[0150] Then, the server device 100 classifies the intake of the user U1 using two thresholds, the speech presence / absence threshold "Threshold_uttr" and the request type speech threshold "Threshold_ask". The server device 100 classifies the intake of the user U1 by comparing the speech score "Score_uttr" with each threshold. That is, the server device 100 classifies the speech type according to the value of the speech score "Score_uttr".

[0151] The server device 100 classifies the inhalation of the user U1 by comparing the speech score "Score_uttr" with the speech presence / absence threshold "Threshold_uttr" and the request-type utterance threshold "Threshold_ask" as shown in the classification result PD2 in Fig. 13. In the example of Fig. 13, the speech presence / absence threshold "Threshold_uttr" is smaller than the request-type utterance threshold "Threshold_ask".

[0152] 13, when the speech score "Score_uttr" is greater than the request-type speech threshold "Threshold_ask", the server device 100 classifies the intake of the user U1 as an intake that is highly likely to be a request utterance (also called a "request intake"). In this case, the information processing system 1, for example, starts a voice UI (voice recognition) with an explicit start-up notification, and performs a normal flow (processing).

[0153] 13, when the speech score "Score_uttr" is equal to or less than the request-type speech threshold "Threshold_ask" and is greater than the speech presence / absence threshold "Threshold_uttr", the server device 100 classifies the intake of the user U1 as an intake that may be a request utterance but is not highly certain (also called an "intermediate value"). In this case, the information processing system 1, for example, starts the voice recognition without an explicit start-up notification, and starts the notification-type response flow.

[0154] 13, when the speech score "Score_uttr" is equal to or less than the speech threshold "Threshold_uttr", the server device 100 classifies the inhalation of the user U1 as an inhalation in which speech is not expected (also called "completely unsolicited inhalation"). In this case, the information processing system 1 does not start up a voice UI (voice recognition).

[0155] In this way, the server device 100 classifies the inhalation of the user U1 by comparing the speech score "Score_uttr" with the speech threshold "Threshold_uttr" and the request-type speech threshold "Threshold_ask". Note that each threshold such as the speech threshold "Threshold_uttr" and the request-type speech threshold "Threshold_ask" may be increased or decreased in accordance with changes in the normal breathing range due to changes in the user's motion state, etc.

[0156] As described above, the speech score "Score_uttr" is a value that takes into account the maximum respiration volume, the increase volume, and the time from maximum respiration to speech (maximum inhalation-speech time). The server device 100 classifies the user's breathing by using the speech score "Score_uttr". This enables the server device 100 to appropriately classify the user's inhalation and use it for selecting processing.

[0157] From here, we will return to Fig. 11 to continue the explanation. If the inhalation of the user U1 is classified as requested inhalation in step S301, the information processing system 1 performs a startup process and makes the user U1 aware of the startup (step S311). In the example of Fig. 11, the terminal device 10 turns on the light source unit 18 to make the user U1 aware of the startup. This allows the user U1 to recognize that voice input is now possible.

[0158] Then, the user U1 speaks (step S312). For example, the user U1 performs a voice input to the terminal device 10 requesting predetermined information.

[0159] Then, the information processing system 1 performs conventional processing (processing of a voice dialogue system) on the input by the user U1 (step S313). For example, the information processing system 1 interprets the user's input by natural language understanding (NLU) and executes a corresponding processing (action). In the example of Fig. 11, the terminal device 10 performs processing of voice output in response to the request of the user U1, "OK. Here's the result...".

[0160] Moreover, when the inhalation of the user U1 is classified as an intermediate value in step S301, the information processing system 1 starts voice recognition without a response (step S321).

[0161] Then, the user U1 speaks (step S322). For example, the user U1 performs a voice input to the terminal device 10 requesting predetermined information.

[0162] Then, the information processing system 1 executes acquisition of an Intent by natural language understanding (NLU) without a Response (step S323).

[0163] Then, the information processing system 1 performs a notification-type response (step S324). In the example of Fig. 11, the terminal device 10 performs a voice output process to the user U1, saying "I have an idea for your...". Note that the information processing system 1 may determine whether or not to notify by determining whether the user is continuing the same topic or is having an ongoing conversation.

[0164] On the other hand, if the inhalation of the user U1 is classified as a completely unsolicited inhalation in step S301, the information processing system 1 does not start the startup process (step S331). The information processing system 1 does not start voice recognition. The terminal device 10 does not start voice recognition.

[0165] In this way, the information processing system 1 can select a process using the classification result of the user's inhalation, and can perform an appropriate process according to the pre-utterance breathing state. For example, the information processing system 1 can enable the omission of an activation word depending on the pre-utterance breathing state using the classification result of the user's inhalation.

[0166] [1-6-2. Example of switching between local / cloud voice recognition] An example of the classification process will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of a process using the result of the classification of the inhalation. Fig. 14 shows an example of switching between local / cloud voice recognition.

[0167] 14, the server device 100 acquires inhalation information indicating an inhalation of the user U1 before speaking. For example, the server device 100 acquires the inhalation information indicating an inhalation of the user U1 from the terminal device 10 used by the user U1.

[0168] The server device 100 performs a classification process using inhalation information indicating the inhalation of the user U1 (step S401). The server device 100 calculates a score using the inhalation information. The server device 100 then classifies the inhalation of the user U1 by comparing the calculated score with a threshold value. The server device 100 classifies the inhalation of the user U1 based on the magnitude relationship between the calculated score and each threshold value.

[0169] The above-mentioned classification process performed by the server device 100 will be specifically described below with reference to Fig. 15 and Fig. 16. Fig. 15 is a diagram showing an example of inhalation information of a user. Fig. 16 is a diagram showing an example of prediction using inhalation information of a user.

[0170] Graph GR3 in Fig. 15 is a graph showing the relationship between time and the intake air volume, with the horizontal axis showing time and the vertical axis showing the intake air volume. Note that, regarding graph GR3, explanations of the same points as those of graph GR1 in Fig. 2 and graph GR2 in Fig. 12 will be omitted.

[0171] For example, the maximum breathing volume "B_max" in the graph GR3 indicates the maximum breathing volume (maximum breathing volume) reached by breathing in before speaking. The maximum breathing-to-speech time "T_bmax_uttr" indicates the interval from the time when the maximum breathing volume is reached to the time when speaking starts (speech start time).

[0172] For example, the increase amount "B_increase" in the graph GR3 indicates the change (increase) in the inhalation volume before reaching the maximum respiratory volume "B_max." Note that the increase amount "B_increase" may be the change (increase) in the inhalation volume at the time when the inhalation information is acquired (the current time).

[0173] For example, the inhalation information includes the increase amount "B_increase", the maximum breathing volume "B_max", and the maximum inhalation-utterance time "T_bmax_uttr" in FIG. 15. The inhalation information does not need to include the maximum inhalation-utterance time "T_bmax_uttr". In this case, the score may be calculated with "c*(1 / T_bmax_uttr)", which is the term (third term) related to the maximum inhalation-utterance time "T_bmax_uttr", set to "0". Furthermore, the maximum breathing volume "B_max" may not be included. In this case, the server device 100 may predict the maximum breathing volume "B_max" as described in FIG. 2, and calculate the score using the predicted maximum breathing volume "B_max".

[0174] The server device 100 calculates the speech length expected score "Score_uttr_length", which is a score used for speech prediction, using the increase amount "B_increase", the maximum respiratory volume "B_max", and the maximum inspiration-to-speech time "T_bmax_uttr" and the following formula (3).

[0175]

number

[0176] "a", "b", and "c" in the above formula (3) indicate predetermined constants. The above formula (3) is similar to the above formula (2), but the values ​​of "a", "b", and "c" are different, and it is considered that the correlation with the maximum amount of intake is large, so the coefficient "a" is relatively large, for example, compared to the above formula (2). Note that the above formula (3) is an example of calculation of the utterance length expected score "Score_uttr_length", and various formulas may be used to calculate the utterance length expected score "Score_uttr_length".

[0177] Then, the server device 100 classifies the intake of the user U1 using two thresholds, the short sentence utterance threshold "Threshold_uttr_short" and the long sentence utterance threshold "Threshold_uttr_long". The server device 100 classifies the intake of the user U1 by comparing the utterance length expected score "Score_uttr_length" with each threshold. That is, the server device 100 classifies the utterance type according to the value of the utterance length expected score "Score_uttr_length".

[0178] The server device 100 classifies the inhalation of the user U1 by comparing the utterance length expected score "Score_uttr_length" with the short sentence utterance threshold "Threshold_uttr_short" and the long sentence utterance threshold "Threshold_uttr_long" as shown in the classification result PD3 in Fig. 16. In the example of Fig. 16, the short sentence utterance threshold "Threshold_uttr_short" is smaller than the long sentence utterance threshold "Threshold_uttr_long".

[0179] In the example of FIG. 16, when the utterance length expected score "Score_uttr_length" is greater than the long sentence utterance threshold "Threshold_uttr_long", the server device 100 classifies the utterance as having a high possibility of being a long sentence utterance (also called "long sentence type utterance"). In this case, the information processing system 1 prepares to start cloud speech recognition and performs processing on the utterance. For example, the information processing system 1 starts the speech recognition of the server device 100 and processes the utterance by the server device 100.

[0180] Also, in the example of FIG. 16, when the speech length expected score "Score_uttr_length" is equal to or less than the long sentence speech threshold "Threshold_uttr_long" and is greater than the short sentence speech threshold "Threshold_uttr_short", the server device 100 classifies the utterance as an utterance that is difficult to guess as being clearly long or short (also called "medium sentence utterance"). In this case, the information processing system 1 prepares, for example, both a cloud type and a local type. For example, the information processing system 1 activates the voice recognition of the server device 100 and the terminal device 10. In this case, for example, the information processing system 1 uses local recognition at the beginning of the utterance and a cloud recognition result as necessary.

[0181] In the example of FIG. 16, when the utterance length expected score "Score_uttr_length" is equal to or less than the short sentence utterance threshold "Threshold_uttr_short", the server device 100 classifies the utterance as having a high possibility of being a short sentence utterance (also called "short sentence utterance"). In this case, the information processing system 1 prepares to start local voice recognition and performs processing on the utterance. For example, the information processing system 1 starts voice recognition of the terminal device 10 and performs processing on the utterance by the terminal device 10.

[0182] In this way, the server device 100 classifies the inhalation of the user U1 by comparing the speech length expected score "Score_uttr_length" with the short sentence speech threshold "Threshold_uttr_short" and the long sentence speech threshold "Threshold_uttr_long". Note that each threshold such as the short sentence speech threshold "Threshold_uttr_short" and the long sentence speech threshold "Threshold_uttr_long" may be increased or decreased in accordance with changes in the normal breathing range due to changes in the user's physical activity, etc.

[0183] As described above, the utterance length expected score "Score_uttr_length" is a value that takes into account the maximum respiration volume, the increase volume, and the time from maximum respiration to speech (maximum inhalation-speech time). The server device 100 classifies the user's breathing by using the utterance length expected score "Score_uttr_length". This enables the server device 100 to appropriately classify the user's inhalation and use it for selecting a process.

[0184] From here, the description will be continued by returning to Fig. 14. When the intake of the user U1 is classified as a long-sentence intake in step S401, the information processing system 1 starts cloud CL (server-based) large vocabulary speech recognition (step S411). In the example of Fig. 14, the information processing system 1 establishes a connection between the server device 100 and the terminal device 10 by WebSocket or the like.

[0185] Then, the user U1 speaks (step S412). Then, the information processing system 1 processes the user U1's speech using cloud (server-based) large vocabulary speech recognition (step S413). For example, the information processing system 1 processes the user's speech using cloud (server-based) large vocabulary speech recognition. This allows the information processing system 1 to improve long-text performance using large vocabulary speech recognition.

[0186] Also, when the breath of the user U1 is classified as Chinese-style breath in step S401, the information processing system 1 prepares both the cloud CL and the local (step S421). For example, the information processing system 1 activates the voice recognition of the server device 100 and the terminal device 10.

[0187] Then, the user U1 speaks (step S422). Then, the information processing system 1 responds with a high-response local result as an initial response during the speech (step S423). For example, the information processing system 1 performs the initial response by using the voice recognition of the terminal device 10.

[0188] Then, when the speech length exceeds a certain length, the information processing system 1 changes to a cloud CL result (step S424). For example, when the speech length exceeds a certain length, the information processing system 1 changes to a response by voice recognition of the server device 100. In this way, the information processing system 1 processes the initial response locally, and when the speech is long, the processing is performed on the cloud CL side where the amount of backup data is large.

[0189] In addition, when the breath of the user U1 is classified as a short sentence type breath in step S401, the information processing system 1 performs a process of starting up the local terminal voice recognition (step S431). In the example of Fig. 14, the information processing system 1 performs a process of starting up the voice recognition of the terminal device 10.

[0190] Then, the user U1 speaks (step S432). Then, the information processing system 1 processes the speech using the voice recognition of the terminal device 10 (step S433). In this case, the information processing system 1 can provide a high response and does not require data communication.

[0191] In this way, the information processing system 1 can select a process using the classification result of the user's inhalation, and can perform appropriate processing according to the pre-speech breathing state. For example, the information processing system 1 can switch (select) between local and cloud speech recognition using the classification result of the user's inhalation. This allows the information processing system 1 to take into account the high response and no need for data communication of local speech recognition, and to appropriately use the long-text recognition performance of cloud speech recognition according to the conditions.

[0192] [1-6-3. Example of changing the voice recognition dictionary] An example of the classification process will be described with reference to Fig. 17. Fig. 17 is a diagram showing an example of a process using the result of the classification of the inhalation. Fig. 17 shows an example of a change in the voice recognition dictionary.

[0193] 17, the server device 100 acquires inhalation information indicating an inhalation of the user U1 before speaking. For example, the server device 100 acquires the inhalation information indicating an inhalation of the user U1 from the terminal device 10 used by the user U1.

[0194] The server device 100 performs a classification process using inhalation information indicating the inhalation of the user U1 (step S501). The server device 100 calculates a score using the inhalation information. The server device 100 then classifies the inhalation of the user U1 by comparing the calculated score with a threshold value. The server device 100 classifies the inhalation of the user U1 based on the magnitude relationship between the calculated score and each threshold value.

[0195] The above-mentioned classification process performed by the server device 100 will be specifically described below with reference to Fig. 18 and Fig. 19. Fig. 18 is a diagram showing an example of inhalation information of a user. Fig. 19 is a diagram showing an example of prediction using inhalation information of a user.

[0196] Graph GR4 in Fig. 18 is a graph showing the relationship between time and the intake air volume, with the horizontal axis showing time and the vertical axis showing the intake air volume. Note that with respect to graph GR4, explanations of the same points as those of graph GR1 in Fig. 2 and graph GR2 in Fig. 12 will be omitted.

[0197] For example, the maximum breathing volume "B_max" in the graph GR4 indicates the maximum breathing volume (maximum breathing volume) reached by breathing in before speaking. The maximum breathing-to-speech time "T_bmax_uttr" indicates the interval from the time when the maximum breathing volume is reached to the time when speaking starts (speech start time).

[0198] For example, the increase amount "B_increase" in the graph GR4 indicates the change (increase) in the inhalation volume before reaching the maximum respiratory volume "B_max." Note that the increase amount "B_increase" may be the change (increase) in the inhalation volume at the time when the inhalation information is acquired (the current time).

[0199] For example, the inhalation information includes the increase amount "B_increase", the maximum breathing volume "B_max", and the maximum inhalation-utterance time "T_bmax_uttr" in FIG. 18. The inhalation information does not need to include the maximum inhalation-utterance time "T_bmax_uttr". In this case, the score may be calculated with "c*(1 / T_bmax_uttr)", which is the term (third term) related to the maximum inhalation-utterance time "T_bmax_uttr", set to "0". Furthermore, the maximum breathing volume "B_max" may not be included. In this case, the server device 100 may predict the maximum breathing volume "B_max" as described in FIG. 2, and calculate the score using the predicted maximum breathing volume "B_max".

[0200] The server device 100 calculates the speech length expected score "Score_uttr_length", which is a score used for speech prediction, using the increase amount "B_increase", the maximum respiratory volume "B_max", and the maximum inspiration-to-speech time "T_bmax_uttr" and the following formula (4).

[0201]

number

[0202] In the above formula (4), "a", "b", and "c" indicate predetermined constants. The above formula (4) is similar to the above formula (2), but the values ​​of "a", "b", and "c" are different, and it is considered that the correlation with the maximum amount of intake is large, so the coefficient "a" is relatively large, for example, compared to the above formula (2). Note that the above formula (4) is an example of calculation of the utterance length expected score "Score_uttr_length", and various formulas may be used to calculate the utterance length expected score "Score_uttr_length".

[0203] Then, the server device 100 classifies the intake of the user U1 using two thresholds, the short sentence utterance threshold "Threshold_uttr_short" and the long sentence utterance threshold "Threshold_uttr_long". The server device 100 classifies the intake of the user U1 by comparing the utterance length expected score "Score_uttr_length" with each threshold. That is, the server device 100 classifies the utterance type according to the value of the utterance length expected score "Score_uttr_length".

[0204] The server device 100 classifies the inhalation of the user U1 by comparing the utterance length expected score "Score_uttr_length" with the short sentence utterance threshold "Threshold_uttr_short" and the long sentence utterance threshold "Threshold_uttr_long" as shown in the classification result PD4 in Fig. 19. In the example of Fig. 19, the short sentence utterance threshold "Threshold_uttr_short" is smaller than the long sentence utterance threshold "Threshold_uttr_long".

[0205] In the example of FIG. 19, when the speech length expected score "Score_uttr_length" is greater than the long sentence speech threshold "Threshold_uttr_long", the server device 100 classifies the intake as having a high possibility of being a long sentence intake (also called "long sentence intake"). In this case, the information processing system 1 prepares a long sentence type voice recognition dictionary. For example, the server device 100 acquires information of a dictionary for long sentences from among the dictionaries stored in the storage unit 120, and performs voice recognition processing using the acquired information.

[0206] In the example of FIG. 19, when the speech length expected score "Score_uttr_length" is equal to or less than the long sentence speech threshold "Threshold_uttr_long" and is greater than the short sentence speech threshold "Threshold_uttr_short", the server device 100 classifies the intake as an intake that is difficult to guess as being clearly long or short (also called "Chinese-type intake"). In this case, the information processing system 1 prepares a Chinese-type voice recognition dictionary. For example, the server device 100 acquires information of a Chinese dictionary from among the dictionaries stored in the storage unit 120, and performs voice recognition processing using the acquired information.

[0207] In the example of FIG. 19, when the speech length expected score "Score_uttr_length" is equal to or less than the short sentence speech threshold "Threshold_uttr_short", the server device 100 classifies the intake as having a high possibility of being a short sentence intake (also called "short sentence intake"). In this case, the information processing system 1 prepares a short sentence speech recognition dictionary (words and phrases). For example, the server device 100 acquires information of a dictionary for words and phrases from among the dictionaries stored in the storage unit 120, and performs speech recognition processing using the acquired information.

[0208] In this way, the server device 100 classifies the inhalation of the user U1 by comparing the speech length expected score "Score_uttr_length" with the short sentence speech threshold "Threshold_uttr_short" and the long sentence speech threshold "Threshold_uttr_long". Note that each threshold such as the short sentence speech threshold "Threshold_uttr_short" and the long sentence speech threshold "Threshold_uttr_long" may be increased or decreased in accordance with changes in the normal breathing range due to changes in the user's physical activity, etc.

[0209] As described above, the utterance length expected score "Score_uttr_length" is a value that takes into account the maximum respiration volume, the increase volume, and the time from maximum respiration to speech (maximum inhalation-speech time). The server device 100 classifies the user's breathing by using the utterance length expected score "Score_uttr_length". This enables the server device 100 to appropriately classify the user's inhalation and use it for selecting a process.

[0210] 17. When the inhalation of the user U1 is classified as a long-sentence type inhalation in step S501, the information processing system 1 selects a dictionary for long sentences (step S511). For example, the server device 100 selects information of a dictionary for long sentences from among the dictionaries stored in the storage unit 120.

[0211] Then, the user U1 speaks (step S541). Then, the information processing system 1 acquires a speech recognition result using the selected dictionary (step S542). For example, the information processing system 1 acquires a speech recognition result using a dictionary for long sentences.

[0212] Also, when the inhalation of the user U1 is classified as Chinese-type inhalation in step S501, a dictionary for Chinese is selected (step S521). For example, the server device 100 selects information of a dictionary for Chinese from among the dictionaries stored in the storage unit 120.

[0213] Then, the user U1 speaks (step S541). Then, the information processing system 1 acquires a speech recognition result using the selected dictionary (step S542). For example, the information processing system 1 acquires a speech recognition result using a Chinese dictionary.

[0214] Also, when the breath of the user U1 is classified as a short-sentence type breath in step S501, a word / phrase dictionary is selected (step S521). For example, the server device 100 selects information of a word / phrase dictionary for short sentences from among the dictionaries stored in the storage unit 120.

[0215] Then, the user U1 speaks (step S541). Then, the information processing system 1 acquires a speech recognition result using the selected dictionary (step S542). For example, the information processing system 1 acquires a speech recognition result using a word / phrase dictionary.

[0216] In this way, the information processing system 1 can change the dictionary used in the voice recognition depending on the length of the utterance, thereby improving the voice recognition performance.

[0217] When the characteristics of the speech content and the recognition engine do not match, for example, the accuracy may decrease when speaking a single word or a word at a time. For example, when a speech recognition dictionary for long sentences is used for a word speech, the recognition performance may be significantly reduced.

[0218] However, the information processing system 1 estimates whether the utterance is short or long from the breathing state and changes the voice recognition engine dictionary. In this way, the information processing system 1 can suppress the above-mentioned performance degradation by selecting a dictionary according to the classification of inhalation.

[0219] [1-6-4. Example of UI changes selected according to intake state] The information processing system 1 may select various information and processes based on the classification of the intakes from long to short, not limited to the above example. This will be described with reference to Fig. 20. Fig. 20 is a diagram showing an example of a process using the intake classification result.

[0220] Specifically, Fig. 20 shows a case where the UI selected is changed based on the estimation of the speech length based on the inhalation state. As shown in content CT1 in Fig. 20, the information processing system 1 selects an appropriate UI element according to the expected speech amount, even if the UI elements are randomly laid out.

[0221] In content CT1 in FIG. 20, for example, it is displayed on the display unit 16 (screen) of the terminal device 10. In content CT1, element EL1 corresponding to ID, element EL2 corresponding to Title, and element EL3 corresponding to MessageBody (main text) are randomly arranged. For ID, short input such as numbers is expected. For Title, input of Chinese text of about a few words is expected. For MessageBody, input of long text such as free text is expected. Therefore, as shown in FIG. 20, the area occupied by element EL1 becomes larger, followed by element EL2 and element EL3.

[0222] The server device 100 performs classification processing using inhalation information indicating the inhalation of the user U1 (step S601). Note that step S601 is similar to step S501, and therefore a description thereof will be omitted.

[0223] When the intake of the user U1 is classified as a long-sentence intake in step S601, the information processing system 1 selects the element EL3 corresponding to the MessageBody from among the elements EL1 to EL3 in the content CT1 (step S611). For example, the terminal device 10 selects the element EL3 corresponding to the MessageBody as the input target.

[0224] When the intake of the user U1 is classified as Chinese intake in step S601, the information processing system 1 selects the element EL2 corresponding to the Title from among the elements EL1 to EL3 in the content CT1 (step S621). For example, the terminal device 10 selects the element EL2 corresponding to the Title as the input target.

[0225] When the intake of the user U1 is classified as a short-sentence intake in step S601, the information processing system 1 selects the element EL1 corresponding to the ID from among the elements EL1 to EL3 in the content CT1 (step S631). For example, the terminal device 10 selects the element EL1 corresponding to the ID as the input target.

[0226] Note that the above is an example, and the information processing system 1 may determine the element by appropriately using various information. The information processing system 1 automatically determines the input destination when the UI element is uniquely determined, but when there are multiple UI elements that are expected to receive input of approximately the same length or when the system cannot automatically determine the input element, the information processing system 1 may perform processing such as inquiring of the user to determine the input element.

[0227] [1-6-5. Example of changing system response] An example of the classification process will be described with reference to Fig. 21 and Fig. 22. Fig. 21 and Fig. 22 are diagrams showing an example of a process using the result of the classification of the inhalation. First, the example of Fig. 21 will be described. Fig. 21 shows an example of a change in the system response (Text-To-Speech). Specifically, Fig. 21 shows an example of a response change at the time of Wuw (Wake up Word).

[0228] 21, the server device 100 acquires inhalation information indicating an inhalation of the user U1 before speaking. For example, the server device 100 acquires the inhalation information indicating an inhalation of the user U1 from the terminal device 10 used by the user U1.

[0229] User U1 takes a pre-speech inhale (step S701), and then user U1 speaks WUW (step S702). Server device 100 performs classification processing using inhale information of user U1. Server device 100 calculates a score using the inhale information. Then, server device 100 classifies user U1's inhale by comparing the calculated score with a threshold. Server device 100 classifies user U1's inhale based on the magnitude relationship between the calculated score and each threshold.

[0230] The above-mentioned classification process performed by the server device 100 will be specifically described below with reference to Fig. 23 and Fig. 24. Fig. 23 is a diagram showing an example of inhalation information of a user. Fig. 24 is a diagram showing an example of prediction using inhalation information of a user.

[0231] Graph GR5 in Fig. 23 is a graph showing the relationship between time and the intake air volume, with the horizontal axis showing time and the vertical axis showing the intake air volume. Note that, regarding graph GR5, explanations of the same points as those of graph GR1 in Fig. 2 and graph GR2 in Fig. 12 will be omitted.

[0232] For example, the maximum breathing volume "B_max" in the graph GR5 indicates the maximum breathing volume (maximum breathing volume) reached by breathing in before speaking. The maximum breathing-to-speech time "T_bmax_uttr" indicates the interval from the time when the maximum breathing volume is reached to the time when speaking starts (speech start time).

[0233] For example, the increase amount "B_increase" in the graph GR5 indicates the change (increase) in the inhalation volume before reaching the maximum respiratory volume "B_max." Note that the increase amount "B_increase" may be the change (increase) in the inhalation volume at the time when the inhalation information is acquired (the current time).

[0234] For example, the inhalation information includes the increase amount "B_increase", the maximum breathing volume "B_max", and the maximum inhalation-utterance time "T_bmax_uttr" in FIG. 23. The inhalation information does not need to include the maximum inhalation-utterance time "T_bmax_uttr". In this case, the score may be calculated with "c*(1 / T_bmax_uttr)", which is the term (third term) related to the maximum inhalation-utterance time "T_bmax_uttr", set to "0". Furthermore, the maximum breathing volume "B_max" may not be included. In this case, the server device 100 may predict the maximum breathing volume "B_max" as described in FIG. 2, and calculate the score using the predicted maximum breathing volume "B_max".

[0235] The server device 100 calculates the speech urgency score "Score_hurry", which is a score used for speech prediction, using the increase amount "B_increase", the maximum respiratory volume "B_max", the maximum inspiration-to-speech time "T_bmax_uttr" and the following formula (5).

[0236]

number

[0237] In the above formula (5), "a", "b", and "c" indicate predetermined constants. The above formula (5) is similar to the above formulas (2) to (4), but the values ​​of "a", "b", and "c" are different, and the coefficient "b" is relatively large compared to the above formulas (2) to (4) in order to more significantly reflect the sudden rise in inhalation. Note that the above formula (5) is an example of calculation of the speech urgency score "Score_hurry", and various formulas may be used to calculate the speech urgency score "Score_hurry".

[0238] Then, the server device 100 classifies the inhalation of the user U1 using two thresholds, a low hurry threshold "Threshold_hurry_low" and a high hurry threshold "Threshold_hurry_high". The server device 100 classifies the inhalation of the user U1 by comparing the speech hurriedness score "Score_hurry" with each threshold. That is, the server device 100 classifies the speech type according to the value of the speech hurriedness score "Score_hurry".

[0239] The server device 100 classifies the inhalation of the user U1 by comparing the utterance urgency score "Score_hurry" with the hurry low threshold "Threshold_hurry_low" and the hurry high threshold "Threshold_hurry_high" as shown in the classification result PD5 in Fig. 24. In the example of Fig. 24, the hurry low threshold "Threshold_hurry_low" is smaller than the hurry high threshold "Threshold_hurry_high".

[0240] In the example of FIG. 24, when the speech urgency score "Score_hurry" is greater than the high hurry threshold "Threshold_hurry_high", the server device 100 classifies the intake as having a high possibility of a long sentence (also called "shortest intake"). In this case, the information processing system 1 predicts that the user desires the shortest possible processing, and executes the shortest possible processing. For example, when the user is trying to execute a task in a hurry, the information processing system 1 shortens the TTS (Text-To-Speech) utterance and outputs SE (sound effect).

[0241] Also, in the example of FIG. 24, when the speech urgency score "Score_hurry" is equal to or less than the high hurry threshold "Threshold_hurry_high" and is greater than the low hurry threshold "Threshold_hurry_low", the server device 100 classifies the intake as an intake in which it is difficult to guess whether it is clearly a long sentence or a short sentence (also called "medium intake"). In this case, the information processing system 1 predicts that the user desires intermediate processing between the shortest and normal, and executes intermediate processing. For example, the information processing system 1 summarizes and presents the TTS speech sentence according to the value of the speech urgency score "Score_hurry". The intermediate processing will be described in detail later.

[0242] In the example of FIG. 24, when the utterance haste score "Score_hurry" is equal to or less than the low hurry threshold "Threshold_hurry_low", the server device 100 classifies the utterance as having a high possibility of being a short sentence utterance (also called "normal utterance"). In this case, the information processing system 1 predicts that the user desires normal processing and executes the shortest processing. For example, since the user is not particularly in a hurry, the information processing system 1 executes a TTS utterance that conveys the most detailed information to the user.

[0243] In this way, the server device 100 classifies the inhalation of the user U1 by comparing the speech urgency score "Score_hurry" with the hurry low threshold "Threshold_hurry_low" and the hurry high threshold "Threshold_hurry_high". Note that each threshold such as the hurry low threshold "Threshold_hurry_low" and the hurry high threshold "Threshold_hurry_high" may be increased or decreased in accordance with changes in the normal breathing range due to changes in the user's motion state, etc.

[0244] As described above, the speech urgency score "Score_hurry" is a value that takes into account the maximum breathing volume, the increase volume, and the time from maximum breathing to speech (maximum inhalation-speech time). The server device 100 classifies the user's breathing by using the speech urgency score "Score_hurry". This enables the server device 100 to appropriately classify the user's inhalation and use it for selecting processing. Note that the breathing state and the speech speed may be judged together, but this point will be described in detail later.

[0245] From here, we will return to Fig. 21 to continue the explanation. If the inhalation of the user U1 is classified as normal inhalation, the information processing system 1 predicts that the user U1 desires normal processing and selects normal processing (step S711). In the example of Fig. 21, the terminal device 10 outputs "How can I help?" in normal processing. Then, the user U1 speaks (step S731).

[0246] Also, if the inhalation of the user U1 is classified as the shortest type inhalation, the information processing system 1 predicts that the user U1 desires the shortest processing and selects the shortest processing (step S721). In the example of Fig. 21, the terminal device 10 outputs only a predetermined SE (sound effect). Then, the user U1 speaks (step S731).

[0247] Also, if the inhalation of the user U1 is classified as a medium type inhalation, it is predicted that the user desires a process intermediate between the shortest and normal processes, and the intermediate process is executed. This point will be described below.

[0248] When the inhalation of the user U1 is classified as a medium inhalation, the information processing system 1 summarizes and presents the TTS utterance sentence according to the value of the utterance haste score "Score_hurry". For example, the information processing system 1 summarizes the TTS utterance using the value of the utterance haste score "Score_hurry".

[0249] Note that the information processing system 1 may use the following formula (6) instead of the above formula (5) to calculate the speech urgency score "Score_hurry", which is a score used for speech prediction.

[0250]

number

[0251] In the above formula (6), "a," "b," "c," and "d" represent predetermined constants. "V_uttr" in the above formula (6) represents an index of how many characters are spoken per unit time (number of utterances per unit time), and is calculated, for example, using the following formula (7).

[0252]

number

[0253] The above formula (7) will be explained with reference to FIG. 25. FIG. 25 is a diagram showing an example of the relationship between the length of a user's utterance and the number of characters. Utterance UT in FIG. 25 conceptually shows an utterance by a user. FIG. 25 shows that utterance UT was made from the start time "T_uttr_start" to the end time "T_uttr_end". In other words, "T_uttr_end-T_uttr_start" in formula (7), which is the value obtained by subtracting the start time "T_uttr_start" from the end time "T_uttr_end", indicates the length of the utterance. "Character number of the utterance" in formula (7) indicates the number of words included in utterance UT.

[0254] In this way, "V_uttr" in the above formula (7) indicates an index of how many characters are spoken per time in the utterance UT (number of utterances per unit time). For example, a large "V_uttr" indicates fast speech, and a small "V_uttr" indicates slow speech. The information processing system 1 may also take the speech speed into account in calculating the speech hurriedness score "Score_hurry" by using the above formula (6).

[0255] The information processing system 1 summarizes the TTS utterance using the speech hurry score "Score_hurry" calculated using either the above formula (5) or the above formula (6). Note that the information processing system 1 may summarize the text, or may use an API (Application Programming Interface) provided by an external service.

[0256] The information processing system 1 may calculate the reduction target value using the following formula (8).

[0257]

number

[0258] "e" in the above formula (8) represents a predetermined constant. The value of "Abbrev_target" in the above formula (6) is the shortening target value. "Abbrev_target" is calculated so that it becomes shorter (smaller) as the user is in a hurry.

[0259] Then, the information processing system 1 summarizes the TTS utterance based on the value of "Abbrev_target." For example, the information processing system 1 summarizes the TTS utterance using the following formula (9).

[0260]

number

[0261] In the above formula (9), "Shorten_API" indicates a predetermined function (API) used to generate a summary. In addition, in the above formula (9), "original_response" indicates the TTS response before summary. In the above formula (9), "Response_abbrev" indicates the summary of the TTS utterance output by Shorten_API. In this case, the information processing system 1 uses "Response_abbrev" output by Shorten_API as the TTS summary.

[0262] For example, when the inhalation of the user U1 is classified as a medium inhalation, the information processing system 1 outputs the "Response_abbrev" output by the Shorten_API. For example, the terminal device 10 outputs a TTS summary corresponding to the "Response_abbrev".

[0263] In this way, the information processing system 1 estimates the turn-taking speed desired by the user from the inhalation state before speaking, and adjusts the length of the TTS response. Also, when the user wants to complete a task quickly, the information processing system 1 switches to a short TTS response or a response with SE, thereby shortening the task completion time. This allows the information processing system 1 to improve usability. Note that, depending on the text, it may not be possible to summarize it to the expected length, and in that case, the information processing system 1 may adjust the playback speed of the TTS to shorten the time as well.

[0264] Next, an example of Fig. 22 will be described. Fig. 22 shows an example of a change in a system response (Text-To-Speech). Specifically, Fig. 22 shows an example of an action response change when an utterance is accepted.

[0265] 22, the server device 100 acquires inhalation information indicating an inhalation of the user U1 before speaking. For example, the server device 100 acquires the inhalation information indicating an inhalation of the user U1 from the terminal device 10 used by the user U1.

[0266] User U1 takes a pre-speech inhale (step S801), and then user U1 speaks WUW (step S802). Server device 100 performs a classification process using the inhale information of user U1. Server device 100 calculates a score using the inhale information. Server device 100 then classifies user U1's inhale by comparing the calculated score with a threshold. Server device 100 classifies user U1's inhale based on the magnitude relationship between the calculated score and each threshold. Note that the classification process is similar to that in FIG. 21, and therefore description thereof will be omitted.

[0267] If the inhalation of the user U1 is classified as normal inhalation, the information processing system 1 predicts that the user U1 desires normal processing and selects normal processing (step S811). In the example of Fig. 22, the terminal device 10 displays information on the display DP (display unit 16) and outputs "OK, here's the result. One new movie and two music" in normal processing. Thus, in normal processing, the terminal device 10 displays information in response to the user's request and also outputs audio related to that information (TTS speech).

[0268] Also, if the inhalation of the user U1 is classified as the shortest type inhalation, the information processing system 1 predicts that the user U1 desires the shortest processing, and selects the shortest processing (step S821). In the example of Fig. 22, the terminal device 10 displays information on the display DP (display unit 16) and outputs only a predetermined SE (sound effect). Thus, in normal processing, the terminal device 10 displays information in response to the user's request and outputs only a notification sound to the user.

[0269] Also, when the inhalation of the user U1 is classified as a medium type inhalation, it is predicted that the user desires an intermediate process between the shortest and normal processes, and the intermediate process as described above is executed. For example, when the inhalation of the user U1 is classified as a medium type inhalation, the terminal device 10 displays information in response to the user's request and also outputs a summary of the TTS utterance in voice.

[0270] As mentioned above, if the terminal has a display DP, the shortest response is SE. However, if the terminal does not have a display, the shortest response may be the smallest TTS utterance that can indicate the state.

[0271] In this way, the information processing system 1 estimates the speed at which the user desires to take turns from the inhalation state before speaking, and when the user is in a hurry, summarizes and shortens the TTS response after the action is executed, or notifies the user with SE. This allows the information processing system 1 to improve usability.

[0272] [2. Other embodiments] The processing according to each of the above-described embodiments may be implemented in various different forms (variations) other than the above-described embodiments and variations.

[0273] [2-1. Example of a configuration for performing prediction processing etc. on the client side] In the embodiment, as an example of the system configuration, the server device 100 performs the prediction process, the classification process, etc., but the terminal device 10 may perform the prediction process or the classification process. That is, the terminal device 10, which is a client-side device, may be an information processing device that performs the above-mentioned prediction process or the classification process. In this way, the system configuration of the information processing system 1 is not limited to a configuration in which the server device 100, which is a server-side device, performs the prediction process or the classification process, but may be a configuration in which the terminal device 10, which is a client-side device, performs the above-mentioned prediction process or the classification process.

[0274] When the terminal device 10 is an information processing device that performs the above-mentioned prediction processing and classification processing, in the information processing system 1, the client side (terminal device 10) predicts utterance and classifies inhalation. Then, the server side (server device 100) acquires information on the prediction results and classification results from the terminal device 10 and performs various processes. In this case, the terminal device 10 may have a prediction unit that realizes a function similar to the prediction unit 132 described above, and a selection unit that realizes a function similar to the selection unit 133. In this case, the server device 100 may not have the prediction unit 132 or the selection unit 133.

[0275] Furthermore, the information processing system 1 may have a system configuration in which the client side (terminal device 10) predicts the speech and the server side (server device 100) classifies the inhalation. In this case, the terminal device 10, which is the client side device, may be an information processing device that performs the above-mentioned prediction processing, and the server device 100, which is the server side device, may be an information processing device that performs the above-mentioned classification processing. In this case, the prediction unit of the terminal device 10 performs the prediction processing, and the prediction unit 132 of the server device 100 performs the classification processing.

[0276] The above is just an example, and each process may be performed by any device in the information processing system 1. In this way, the information processing system 1 may have a system configuration in which each process is performed by either the client side device (terminal device 10) or the server side device (server device 100).

[0277] [2-2. Other configuration examples] In the above example, the server device 100 and the terminal device 10 are separate devices, but these devices may be integrated.

[0278] [2-3.Other] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0279] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0280] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory.

[0281] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0282] [3. Effects of this disclosure] As described above, the information processing device according to the present disclosure (server device 100 in the embodiment) includes an acquisition unit (acquisition unit 131 in the embodiment) and a prediction unit (prediction unit 132 in the embodiment). The acquisition unit acquires inhalation information indicating an inhalation of a user. The prediction unit predicts whether or not the user will speak after the user inhales, based on the inhalation information acquired by the acquisition unit.

[0283] In this way, the information processing device according to the present disclosure predicts whether or not the user will speak after the user inhales, based on the inhalation information indicating the user's inhalation. In this way, the information processing device can appropriately predict whether or not the user will speak by predicting whether or not the user will speak after the user inhales, based on the state of the user's inhalation.

[0284] The acquisition unit acquires inhalation information including an increase in the amount of inhalation of the user. The prediction unit predicts whether the user will speak after inhalation based on the increase. In this manner, the information processing device can accurately predict whether the user will speak by using the increase in the amount of inhalation of the user.

[0285] The acquisition unit acquires inhalation information including an amount of inhaled air of the user. The prediction unit predicts whether the user will speak after inhaling based on the amount of inhaled air. In this manner, the information processing device can accurately predict whether the user will speak by using the amount of inhaled air of the user.

[0286] The acquisition unit acquires inhalation information including an initial inhalation volume at the start of the user's inhalation. The prediction unit predicts whether the user will speak after inhalation based on the initial inhalation volume. In this manner, the information processing device can accurately predict whether the user will speak by using the initial inhalation volume of the user.

[0287] Furthermore, the prediction unit predicts whether or not the user will speak after inhaling, using the score calculated based on the inhalation information. In this manner, the information processing device can appropriately predict whether or not the user will speak, by using the score calculated based on the inhalation information.

[0288] Furthermore, when a comparison result between the score and the threshold satisfies a predetermined condition, the prediction unit predicts that the user will speak after inhaling. In this manner, the information processing device can appropriately predict whether or not the user will speak by predicting the user's speech based on the comparison between the score and the threshold.

[0289] Moreover, the information processing device according to the present disclosure includes an execution unit (execution unit 134 in the embodiment). The execution unit executes processing according to the prediction result by the prediction unit. In this manner, the information processing device executes processing according to the prediction result of whether or not the user will speak, thereby enabling the information processing device to execute appropriate processing according to whether or not the user will speak.

[0290] Furthermore, when the prediction unit predicts that the user will speak after inhalation, the execution unit executes pre-processing related to voice recognition before the end of the user's inhalation. In this way, when the information processing device predicts that the user will speak, it executes pre-processing related to voice recognition before the end of inhalation, thereby making it possible to prepare for voice recognition prior to the user's speech, thereby improving usability.

[0291] Furthermore, the execution unit executes pre-processing related to voice recognition when the prediction unit predicts that the user will speak after inhalation. In this way, the information processing device executes pre-processing related to voice recognition when it predicts that the user will speak, thereby making it possible to prepare for voice recognition in accordance with the prediction and improving usability.

[0292] Furthermore, the execution unit executes pre-processing before the end of the user's inhalation. In this way, when it is predicted that the user will speak, the information processing device executes pre-processing related to voice recognition before the end of inhalation, thereby making it possible to prepare for voice recognition prior to the user's speech, thereby improving usability.

[0293] In addition, the prediction unit classifies the user's inhalation based on the inhalation information. In this manner, the information processing device can execute subsequent processing using the result of classifying the user's inhalation state by classifying the user's inhalation.

[0294] The acquisition unit acquires the inhalation information including the maximum inhalation amount of the user's inhalation. The prediction unit classifies the user's inhalation based on the maximum inhalation amount. In this manner, the information processing device can accurately classify the user's inhalation by using the maximum inhalation amount of the user's inhalation.

[0295] The acquisition unit also acquires time point information indicating the start time of speech after the user's inhalation. The prediction unit classifies the user's inhalation based on the interval between the time of the maximum inhalation amount and the start time of speech. In this way, the information processing device can accurately classify the user's inhalation by using the information on the interval between the time of the maximum inhalation amount and the start time of speech.

[0296] The acquisition unit also acquires speech information including a length and a number of characters of the speech after the user's inhalation. The prediction unit classifies the user's inhalation based on the length and the number of characters of the speech. In this way, the information processing device can accurately classify the user's inhalation by using the length and the number of characters of the speech after the user's inhalation.

[0297] In addition, the prediction unit classifies the user's inhalation into one of a plurality of types including at least requested inhalation and unrequested inhalation. In this manner, the information processing device can appropriately classify the user's inhalation status by classifying the user's inhalation into one of a plurality of types including requested inhalation and unrequested inhalation.

[0298] In addition, the prediction unit classifies the user's inhalation into one of a plurality of types including at least long-sentence inhalation and short-sentence inhalation. In this way, the information processing device can appropriately classify the user's inhalation status by classifying the user's inhalation into one of a plurality of types including long-sentence inhalation and short-sentence inhalation.

[0299] In addition, the prediction unit classifies the user's inhalation into one of a plurality of types including at least a normal processing desired type of inhalation and a reduced processing desired type of inhalation. In this manner, the information processing device can appropriately classify the user's inhalation status by classifying the user's inhalation into one of a plurality of types including a normal processing desired type of inhalation and a reduced processing desired type of inhalation.

[0300] Moreover, the information processing device according to the present disclosure includes a selection unit (selection unit 133 in the embodiment). The selection unit performs a selection process according to the classification result by the prediction unit. In this manner, the information processing device can make an appropriate selection according to whether or not the user will speak by performing a selection process according to the prediction result of whether or not the user will speak.

[0301] Furthermore, the selection unit selects a process to be executed according to the classification result by the prediction unit. In this manner, the information processing device can appropriately select a process to be executed according to whether or not the user speaks by selecting a process to be executed according to the classification result by the prediction unit.

[0302] Furthermore, the selection unit selects information to be used in processing the user's utterance according to the classification result by the prediction unit. In this manner, the information processing device selects information to be used in processing the user's utterance according to the classification result by the prediction unit, thereby allowing the information processing device to appropriately select information to be used according to whether the user will speak.

[0303] [4. Hardware configuration] Information devices such as the server device 100 and the terminal device 10 according to each of the above-described embodiments are realized by a computer 1000 having a configuration as shown in FIG. 26, for example. FIG. 26 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of an information processing device. The server device 100 according to the embodiment will be described below as an example. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

[0304] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.

[0305] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.

[0306] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.

[0307] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0308] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0309] For example, when the computer 1000 functions as the server device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize functions of the control unit 130, etc. Also, the information processing program according to the present disclosure and data in the storage unit 120 are stored in the HDD 1400. Note that the CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0310] The present technology can also be configured as follows. (1) An acquisition unit that acquires inhalation information indicating an inhalation of a user; a prediction unit that predicts whether the user will speak after the user inhales based on the inhalation information acquired by the acquisition unit; An information processing device comprising: (2) The acquisition unit is acquiring the inhalation information including an increase in the inhalation of the user; The prediction unit is predicting whether the user will speak after the inhalation based on the increase; An information processing device as described in (1). (3) The acquisition unit is acquiring the inhalation information including an inhalation amount of the user; The prediction unit is predicting whether the user will speak after the inhalation based on the inhalation volume; An information processing device according to (1) or (2). (4) The acquisition unit is acquiring the inhalation information including an initial inhalation amount at a time point when the inhalation of the user starts; The prediction unit is predicting whether the user will speak after the inhalation based on the initial inhalation volume; The information processing device according to any one of (1) to (3). (5) The prediction unit is predicting whether the user will speak after the inhalation using a score calculated based on the inhalation information; The information processing device according to any one of (1) to (4). (6) The prediction unit is predicting that the user will speak after the inhalation if a comparison result between the score and a threshold satisfies a predetermined condition; (5) An information processing device according to the present invention. (7) an execution unit that executes a process according to a result of the prediction by the prediction unit; The information processing device according to any one of (1) to (6). (8) The execution unit is When the prediction unit predicts that the user will speak after the inhalation, a pre-processing related to voice recognition is performed. An information processing device according to (7). (9) The execution unit is The pre-processing is performed before the inhalation of the user is completed. An information processing device according to (8). (10) The prediction unit is classifying the inhalation of the user based on the inhalation information; The information processing device according to any one of (1) to (9). (11) The acquisition unit is acquiring the inhalation information including a maximum inhalation amount of the user; The prediction unit is classifying the inhalation of the user based on the maximum inhalation volume; An information processing device according to (10). (12) The acquisition unit is acquiring time point information indicating a speech start time point after the user's inhalation; The prediction unit is classifying the inhalation of the user based on an interval between the time of the maximum inhalation and the time of the start of speech; An information processing device according to (11). (13) The acquisition unit is Acquire speech information including a length and a number of characters of the user's post-inhalation speech; The prediction unit is classifying the user's inhalation based on the length and number of characters of the utterance; The information processing device according to any one of (10) to (12). (14) The prediction unit is classifying the user's inhalation into one of a plurality of types including at least requested inhalation and unsolicited inhalation; The information processing device according to any one of (10) to (13). (15) The prediction unit is Classifying the user's inhalation into one of a plurality of types including at least a long-sentence inhalation and a short-sentence inhalation; The information processing device according to any one of (10) to (13). (16) The prediction unit is Classifying the user's inhalation into one of a plurality of types including at least a normal processing desired inhalation and a shortened processing desired inhalation. The information processing device according to any one of (10) to (13). (17) a selection unit that performs a selection process according to a classification result by the prediction unit; The information processing device according to any one of (10) to (16), further comprising: (18) The selection unit is Selecting a process to be executed according to a classification result by the prediction unit. An information processing device according to (17). (19) The selection unit is selecting information to be used in processing the user's utterance according to a classification result by the prediction unit; The information processing device according to (17) or (18). (20) Acquire inhalation information indicating an inhalation of a user; predicting whether the user will speak after the user's inhalation based on the acquired inhalation information; An information processing method for performing a process. [Explanation of symbols]

[0311] 1. Information Processing Systems 100 Server device (information processing device) 110 Communications Department 120 Storage section 121 Intake information storage unit 122 User information storage unit 123 Threshold information storage unit 124 Functional information storage unit 130 Control section 131 Acquisition Department 132 Prediction Department 133 Selection Section 134 Executive Department 135 Transmitter 10 Terminal Equipment 11 Communications Department 12 Input section 13 Output section 14 Storage section 15 Control section 151 Receiving unit 152 Executive Department 153 Reception 154 Transmitter 16 Display 17 Sensor section 171 Respiration Sensor 18 Light source section

Claims

1. An acquisition unit that acquires inhalation information indicating an inhalation of a user; a prediction unit that predicts whether the user will speak after the user's inhalation based on the inhalation information acquired by the acquisition unit, and classifies the user's inhalation based on the inhalation information; Equipped with The acquisition unit is Acquire speech information including a length and a number of characters of the user's post-inhalation speech; The prediction unit is An information processing device that classifies the inhalation of the user based on the length and number of characters of the utterance.

2. An acquisition unit that acquires inhalation information indicating an inhalation of a user; a prediction unit that predicts whether the user will speak after the user's inhalation based on the inhalation information acquired by the acquisition unit, and classifies the user's inhalation based on the inhalation information; Equipped with The prediction unit is An information processing device that classifies the inhalation of the user into one of a plurality of types including at least requested inhalation and unrequested inhalation.

3. An acquisition unit that acquires inhalation information indicating an inhalation of a user; a prediction unit that predicts whether the user will speak after the user's inhalation based on the inhalation information acquired by the acquisition unit, and classifies the user's inhalation based on the inhalation information; Equipped with The prediction unit is An information processing device that classifies the user's intake into one of a plurality of types including at least a long-sentence intake and a short-sentence intake.

4. An acquisition unit that acquires inhalation information indicating an inhalation of a user; a prediction unit that predicts whether the user will speak after the user's inhalation based on the inhalation information acquired by the acquisition unit, and classifies the user's inhalation based on the inhalation information; Equipped with The prediction unit is An information processing device that classifies the inhalation of the user into one of a plurality of types including at least a normal processing desired inhalation and a shortened processing desired inhalation.

5. An acquisition unit that acquires inhalation information indicating an inhalation of a user; a prediction unit that predicts whether the user will speak after the user's inhalation based on the inhalation information acquired by the acquisition unit, and classifies the user's inhalation based on the inhalation information; A selection unit that performs a selection process according to a classification result by the prediction unit; Equipped with The selection unit is An information processing device that selects information to be used in processing the user's utterance according to a classification result by the prediction unit.

6. The acquisition unit is acquiring the inhalation information including an increase in the inhalation of the user; The prediction unit is predicting whether the user will speak after the inhalation based on the increase; The information processing device according to claim 1 .

7. The acquisition unit is acquiring the inhalation information including an inhalation amount of the user; The prediction unit is predicting whether the user will speak after the inhalation based on the inhalation volume; The information processing device according to claim 1 .

8. The acquisition unit is acquiring the inhalation information including an initial inhalation amount at a time point when the inhalation of the user starts; The prediction unit is predicting whether the user will speak after the inhalation based on the initial inhalation volume; The information processing device according to claim 1 .

9. The prediction unit is predicting whether the user will speak after the inhalation using a score calculated based on the inhalation information; The information processing device according to claim 1 .

10. The prediction unit is predicting that the user will speak after the inhalation if a comparison result between the score and a threshold satisfies a predetermined condition; The information processing device according to claim 9.

11. an execution unit that executes a process according to a result of the prediction by the prediction unit; The information processing device according to claim 1 .

12. The execution unit is When the prediction unit predicts that the user will speak after the inhalation, a pre-processing related to voice recognition is performed. The information processing device according to claim 11.

13. The execution unit is The pre-processing is performed before the inhalation of the user is completed. The information processing device according to claim 12.

14. The acquisition unit is acquiring the inhalation information including a maximum inhalation amount of the user; The prediction unit is classifying the inhalation of the user based on the maximum inhalation volume; The information processing device according to claim 1 .

15. The acquisition unit is acquiring time point information indicating a speech start time point after the user's inhalation; The prediction unit is classifying the inhalation of the user based on an interval between the time of the maximum inhalation and the time of the start of speech; The information processing device according to claim 14.

16. Acquire inhalation information indicating an inhalation of a user; predicting whether the user will speak after the user's inhalation based on the acquired inhalation information, and classifying the user's inhalation based on the inhalation information; Execute the process, Acquire speech information including a length and a number of characters of the user's post-inhalation speech; An information processing method for classifying the inhalation of the user based on the length and number of characters of the utterance.

17. Acquiring inhalation information indicating an inhalation of a user, predicting whether the user will speak after the user's inhalation based on the acquired inhalation information, and classifying the user's inhalation based on the inhalation information; Execute the process, An information processing method for classifying the user's inhalation into one of a plurality of types including at least requested inhalation and unsolicited inhalation.

18. Acquiring inhalation information indicating an inhalation of a user, predicting whether the user will speak after the user's inhalation based on the acquired inhalation information, and classifying the user's inhalation based on the inhalation information; Execute the process, An information processing method for classifying the user's intake into one of a plurality of types including at least a long-sentence intake and a short-sentence intake.

19. Acquiring inhalation information indicating an inhalation of a user, predicting whether the user will speak after the user's inhalation based on the acquired inhalation information, and classifying the user's inhalation based on the inhalation information; Execute the process, An information processing method for classifying the user's inhalation into one of a plurality of types including at least a normal processing desired inhalation and a shortened processing desired inhalation.

20. Acquiring inhalation information indicating an inhalation of a user, Based on the acquired inhalation information, predict whether the user will speak after the inhalation of the user, and classify the inhalation of the user based on the inhalation information; Selective processing according to the classification results Execute the process, An information processing method for selecting information to be used in processing the user's speech depending on a classification result.

Images