Information processing device and information processing method

Abstract

This information processing device comprises: an acquisition unit that acquire respiration information indicating the respiration of a user; and a determination unit that determines an operation amount in relation to an operation performed by the user, such determination being made on the basis of the respiration of the user as indicated by the respiration information acquired by the acquisition unit.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and an information processing method. [Background technology]

[0002] In recent years, technologies for voice dialogue systems that communicate (dialogue) with a user through voice have been provided. For example, a technology for determining an utterance timing in a voice dialogue system is known (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-211596 A Summary of the Invention [Problem to be solved by the invention]

[0004] According to the conventional technology, the timing of speech is determined in a voice dialogue system based on the timing at which the user's breathing changes from exhalation to inhalation.

[0005] However, the conventional technology does not necessarily reduce the operational load on the user. For example, the conventional technology only determines the timing at which the system speaks to the user, and does not take into consideration the load on the user when performing operations by speaking, etc. Therefore, it is desirable to reduce the operational load on the user.

[0006] Therefore, the present disclosure proposes an information processing device and an information processing method that can reduce the operational burden on the user. [Means for solving the problem]

[0007] In order to solve the above problems, an information processing device of one embodiment according to the present disclosure includes an acquisition unit that acquires respiratory information indicating a user's breathing, and a decision unit that determines an amount of operation related to an operation by the user based on the user's breathing indicated by the respiratory information acquired by the acquisition unit. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of information processing according to an embodiment of the present disclosure. [Diagram 2] 13A and 13B are diagrams illustrating an example of how a user's breathing is reflected in an operation amount. [Diagram 3] FIG. 13 is a diagram showing an example of a display based on an operation amount. [Figure 4] FIG. 13 is a diagram showing an example of a display based on an operation amount. [Diagram 5] FIG. 1 is a diagram illustrating a configuration example of an information processing system according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a diagram illustrating a configuration example of an information processing device according to an embodiment of the present disclosure. [Figure 7] FIG. 4 is a diagram illustrating an example of a threshold information storage unit according to an embodiment of the present disclosure. [Figure 8] FIG. 2 is a diagram illustrating a configuration example of a terminal device according to an embodiment of the present disclosure. [Figure 9] 1 is a flowchart illustrating a processing procedure of an information processing device according to an embodiment of the present disclosure. [Figure 10] 1 is a sequence diagram showing a processing procedure of an information processing system according to an embodiment of the present disclosure. [Figure 11] 1 is a flowchart illustrating processing of an information processing system according to an embodiment of the present disclosure. [Figure 12] 13A and 13B are diagrams illustrating an example of how a user's breathing is reflected in an operation amount. [Figure 13] 13A and 13B are diagrams illustrating an example of how a user's breathing is reflected in an operation amount. [Figure 14] 13A and 13B are diagrams illustrating an example of how a user's breathing is reflected in an operation amount. [Figure 15] 13A and 13B are diagrams illustrating an example of how a user's breathing is reflected in an operation amount. [Figure 16] FIG. 13 is a diagram illustrating an example of prediction of an operation amount. [Figure 17] FIG. 13 is a diagram showing an example of a display of a predicted operation amount. [Figure 18] FIG. 13 is a diagram illustrating an example of use of an operation amount. [Figure 19] FIG. 13 is a diagram showing an example of switching between display and non-display of an indicator. [Figure 20] FIG. 11 is a diagram showing an example of switching of a GUI operation target. [Figure 21] FIG. 13 is a diagram illustrating a configuration example of an information processing device according to a modified example of the present disclosure. [Figure 22] FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the information processing device and the information processing method according to the present application are not limited to these embodiments. In addition, in the following embodiments, the same components are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0010] The present disclosure will be described in the following order. 1. Embodiment 1-1. Overview of information processing according to an embodiment of the present disclosure 1-1-1. Background and effects 1-1-2. Example of display based on operation amount 1-1-2-1. Example of absolute value display 1-1-2-2. Example of relative value display 1-1-3. Target determination using pre-breath speech 1-1-4.Sensor examples 1-1-4-1. Contact type 1-1-4-2. Non-contact type 1-2. Configuration of information processing system according to embodiment 1-3. Configuration of the information processing device according to the embodiment 1-4. Configuration of terminal device according to embodiment 1-5. Information Processing Procedure According to the Embodiment 1-5-1. Processing procedure related to information processing device 1-5-2. Processing procedures related to information processing systems 1-5-3. Specific examples of processing related to information processing systems 1-6. Example of reflection on respiratory control volume 1-6-1. Example of reflection during sudden inspiration 1-6-2. Example of reflecting exhaust control amount 1-6-3. Example of not reflecting exhaust during intake 1-6-4. Example of not reflecting intake during exhaust 1-7. Prediction processing 1-7-1. Example of operation volume prediction 1-7-2. Example of predicted operation amount display 1-8.Other application examples 1-8-1.Selector indicator operation 1-8-2. Example of turning indicators on and off 1-8-3.Example of GUI operation target switching 2. Other embodiments 2-1. Modification (Example of decision processing on the client side) 2-2. Configuration example of information processing device according to the modified example 2-3.Other 3. Effects of the present disclosure 4. Hardware Configuration

[0011] [1. Embodiment] [1-1. Overview of information processing according to an embodiment of the present disclosure] 1 is a diagram illustrating an example of information processing according to an embodiment of the present disclosure. The information processing according to the embodiment of the present disclosure is realized by an information processing system 1 (see FIG. 5) including an information processing device 100 (see FIG. 6) and a terminal device 10 (see FIG. 8).

[0012] The information processing device 100 is an information processing device that executes information processing according to the embodiment. In the example of FIG. 1, the information processing device 100 determines an amount of operation by a user (hereinafter also referred to as an "operation amount") based on the user's breathing, and determines an operation target based on the user's subsequent speech. Note that after determining an operation target based on the user's speech, the amount of operation by the user may be determined based on the user's subsequent breathing, but this point will be described later. Also, in the example of FIG. 1, a case is shown in which sensor information detected by a breathing sensor 171 (see FIG. 8) of a terminal device 10 used by a user is used as breathing information. Note that in the example of FIG. 1, a case is shown in which breathing information indicating the user's breathing is detected by the breathing sensor 171 using a millimeter wave radar, but any sensor may be used as long as it can detect the user's breathing information, not limited to a millimeter wave radar. This point will be described later.

[0013] FIG. 1 will be specifically described below. Each process shown in FIG. 1 may be performed by any of the information processing device 100 and the terminal device 10 of the information processing system 1. The process described in which the information processing system 1 is the subject of the process may be performed by any of the devices included in the information processing system 1. In the following, a case will be described as an example in which the information processing device 100 performs a process (determination process) for determining an amount of operation related to an operation by the user U1 using breathing information indicating the breathing of the user U1 detected by the terminal device 10. Note that, in FIG. 1, a case will be described as an example in which the information processing device 100 performs the determination process (information processing), but the terminal device 10 may perform the determination process (information processing). This point will be described later.

[0014] First, in the example of FIG. 1, the user U1 utters a WUW (Wake up Word) (step S11). For example, the user U1 utters a speech to activate the speech recognition of the information processing system 1. The information processing system 1 activates the speech recognition in response to the utterance of the WUW by the user U1. The information processing device 100 or the terminal device 10 activates the speech recognition in response to the utterance of the WUW by the user U1. In the example of FIG. 1, the terminal device 10 is a tablet terminal having a display unit 16 which is a display (display screen) and an output unit 13 which outputs a voice (see FIG. 8). Note that the terminal device 10 is not limited to a tablet terminal as shown in FIG. 1, and may be any device such as a smart speaker or a smartphone, which will be described in detail later.

[0015] In the example of FIG. 1, the terminal device 10 starts voice recognition in response to an utterance of WUW by the user U1. The terminal device 10 performs a predetermined voice output process when the voice recognition is started. In the example of FIG. 1, as shown in the terminal device 10-1, a voice output of "How can I help?" is output. Thus, in FIG. 1, the terminal device 10 may be distinguished and described as terminal devices 10-1, 10-2, and 10-3 depending on the change in the display of the terminal device 10. Note that the terminal devices 10-1, 10-2, and 10-3 are the same terminal device 10, and when they are described without distinction, they will be described as terminal device 10.

[0016] In the example of FIG. 1, the display unit 16 of the terminal device 10 displays an indicator ID1, which is an indicator (intake volume indicator) corresponding to the intake volume of the user U1, as shown in the terminal device 10-1. The indicator ID1 is a display object whose size changes according to the breathing of the user U1. That is, the indicator ID1 is a display object whose size indicates the amount of operation determined based on the user's breathing. For example, the indicator ID1-1 shown in the terminal device 10-1 indicates the initial state of the indicator ID1. Thus, in FIG. 1, the indicator ID1 may be distinguished and described as indicators ID1-1 and ID1-2 according to changes in the display of the indicator ID1. Note that when the indicators ID1-1 and ID1-2 are described without distinction, they are described as indicator ID1.

[0017] The user U1 inhales before speaking (step S12). For example, the user U1 breathes before speaking, and the information processing device 100 acquires breathing information indicating the breathing of the user U1. For example, the information processing device 100 acquires breathing information indicating the breathing of the user U1 from the terminal device 10 used by the user U1. The information processing device 100 performs a determination process for determining an amount of operation related to an operation by breathing of the user U1, using the acquired breathing information.

[0018] Hereinafter, the above-mentioned determination process performed by the information processing device 100 will be specifically described with reference to Fig. 2. Fig. 2 is a diagram showing an example of how the user's breathing is reflected in the amount of operation.

[0019] Graph GR1 in Fig. 2 is a graph showing the relationship between the user's breathing and the amount of operation. Specifically, graph GR1 in Fig. 2 is a graph showing the relationship between the user's inhalation amount and the size of the indicator based on the amount of operation (indicator size). In graph GR1 in Fig. 2, the vertical axis shows the amount of inhalation or the size of the indicator, and the horizontal axis shows time. In graph GR1 in Fig. 2, a line showing the amount of breathing (line LN11) is shown by a solid line, and a line showing the indicator size (line LN12) is shown by a dashed dotted line.

[0020] In addition, each value used in the following description will be described. The current intake volume "B_current" indicates the user's intake volume at the present time. For example, the current intake volume "B_current" indicates the latest intake volume of the user acquired at the time of the determination process. The maximum intake volume "B_max" indicates the maximum intake volume reached in one breath. The maximum intake volume "B_max" indicates the maximum intake volume reached during one successive inhalation and exhalation.

[0021] The intake increase amount "B_increase" indicates the increase amount of intake air during intake. That is, the intake increase amount "B_increase" indicates the change amount (increase amount) of the intake amount during intake. Note that the intake increase amount "B_increase" may be information indicating the ratio of the increase amount of the intake amount to the passage of time (increase amount). For example, the intake increase amount "B_increase" may be a slope (rate of change). The increase amount threshold "B_increase_thres" indicates a threshold used for comparison with the intake increase amount. The exhaust increase amount "B_decrease" indicates the increase amount of exhaust during exhaust. That is, the exhaust increase amount "B_decrease" indicates the change amount (decrease amount) of the intake amount during exhaust. Note that the exhaust increase amount "B_decrease" may be information indicating the ratio of the decrease amount of the intake amount to the passage of time (increase amount). For example, the exhaust increase amount "B_decrease" may be a slope (rate of change). The exhaust volume threshold "B_decrease_thres" indicates a threshold used for comparison with the exhaust volume increase.

[0022] The indicator parameter "P_indicator" indicates a parameter of the indicator. The indicator parameter "P_indicator" indicates the size of the indicator. The indicator parameter "P_indicator" may indicate an amount of manipulation. In the following, the indicator parameter "P_indicator" is described as indicating an amount of manipulation. The time variable "Time" indicates a value based on time. The time variable "Time" indicates a value that changes according to the elapsed time from the time of the maximum intake volume "B_max". For example, the time variable "Time" takes a larger value as the difference from the time of the maximum intake volume "B_max" increases.

[0023] The indicator parameter "P_indicator" increases until the maximum inspiration is reached. In the example of Fig. 2, the indicator parameter "P_indicator" increases in proportion to the inspiration amount until time t11 when the maximum inspiration amount "B_max" is reached.

[0024] For example, the information processing device 100 calculates an indicator parameter "P_indicator" indicating the amount of operation, using the current amount of intake "B_current" and the following formula (1) until the maximum intake. For example, the information processing device 100 calculates the indicator parameter "P_indicator" indicating the amount of operation, using formula (1) while the user is inhaling.

[0025]

number

[0026] "a" and "b" in the above formula (1) indicate predetermined constants. In this way, the information processing device 100 increases the amount of operation in proportion to the amount of intake during intake. The information processing system 1 increases the amount of operation in proportion to the amount of intake during intake. As a result, the information processing system 1 increases the indicator size in proportion to the amount of intake during intake. Note that the information processing device 100 may calculate the indicator parameter "P_indicator" indicating the amount of operation, and then determine the indicator size using the value of the indicator parameter "P_indicator" and a predetermined conversion formula.

[0027] The above formula (1) is an example of calculation of the indicator parameter "P_indicator" when the intake increase amount "B_increase" is less than the increase amount threshold "B_increase_thres" until the maximum intake, and various formulas may be used to calculate the indicator parameter "P_indicator". For example, when the intake increase amount "B_increase" is equal to or greater than the increase amount threshold "B_increase_thres", a different calculation formula (the following formula (3)) is used, which will be described in detail later.

[0028] In addition, which of the following includes the case where the intake increase amount "B_increase" is equal to the threshold value may be appropriately set. When the intake increase amount "B_increase" is equal to or less than the increase amount threshold value "B_increase_thres", the information processing device 100 may use the above formula (1), and when the intake increase amount "B_increase" becomes larger than the increase amount threshold value "B_increase_thres", the information processing device 100 may use the following formula (3).

[0029] Furthermore, the indicator parameter "P_indicator" changes at a constant rate after the maximum inspiration. In the example of Fig. 2, the indicator parameter "P_indicator" decreases at a constant rate over time until the time t11 at which the maximum inspiration amount "B_max" is reached.

[0030] For example, after the maximum intake, the information processing device 100 calculates an indicator parameter "P_indicator" indicating the amount of operation using the maximum intake amount "B_max", a time variable "Time", and the following formula (2). For example, while the user is exhausting, the information processing device 100 calculates the indicator parameter "P_indicator" indicating the amount of operation using formula (2).

[0031]

number

[0032] In the above formula (2), "c" indicates a predetermined constant. In this way, the information processing device 100 reduces the manipulated variable at a constant rate during exhaust and increases the manipulated variable in proportion to the intake volume during intake. The information processing system 1 reduces the indicator size at a constant rate during exhaust.

[0033] The above formula (2) is an example of calculation of the indicator parameter "P_indicator" when the exhaust increase amount "B_decrease" after the maximum intake is greater than the exhaust amount threshold "B_decrease_thres", and various formulas may be used to calculate the indicator parameter "P_indicator". For example, when the exhaust increase amount "B_decrease" becomes equal to or less than the exhaust amount threshold "B_decrease_thres", a different calculation formula (the following formula (6)) is used, which will be described in detail later.

[0034] In addition, which of the following includes the case where the exhaust gas increase amount "B_decrease" is equal to the threshold value may be appropriately set. When the exhaust gas increase amount "B_decrease" is equal to or greater than the exhaust gas amount threshold value "B_decrease_thres", the information processing device 100 may use the above formula (2), and when the exhaust gas increase amount "B_decrease" is less than the exhaust gas amount threshold value "B_decrease_thres", the information processing device 100 may use the following formula (6).

[0035] The information processing device 100 determines the amount of operation related to the operation by the user U1 by calculating the amount of operation (indicator parameter) through the above-mentioned process. In this manner, the information processing device 100 can appropriately calculate the amount of operation by determining the amount of operation using information on the user's breathing.

[0036] The information processing device 100 may perform the decision process by various methods, not limited to the above-mentioned process, and the process using the above-mentioned formulas (1) and (2) is an example. For example, the information processing device 100 may perform the decision process by using a technique related to machine learning. In this case, the information processing device 100 may perform the decision process by using a model that outputs an operation amount when breathing information is input. The information processing device 100 may perform the decision process by using a model that learns the relationship between the breathing amount indicated by the breathing information and the operation amount, and outputs an operation amount corresponding to the breathing amount according to the input of the breathing amount. For example, the information processing device 100 may learn the model by using learning data including a combination of breathing information indicating the user's breathing amount and the operation amount corresponding to the breathing amount, or may acquire the model from an external information processing device.

[0037] 1, the explanation will be continued. The information processing system 1 changes the size of the indicator ID1 displayed on the terminal device 10 by using the operation amount (indicator parameter) determined by the information processing device 100 (step S13). For example, the information processing device 100 transmits information on the determined operation amount (indicator parameter) to the terminal device 10. The terminal device 10, which has received the information on the operation amount (indicator parameter) from the information processing device 100, changes the size of the displayed indicator ID1.

[0038] In the example of FIG. 1, the terminal device 10 changes the display of indicator ID1 from terminal device 10-1 to terminal device 10-2. The terminal device 10 changes the display from indicator ID1-1 in the initial state to indicator ID1-2, which is an enlarged size, using the operation amount (indicator parameter) received from the information processing device 100. As a result, the display unit 16 of the terminal device 10 displays indicator ID1, which has been resized from indicator ID1-1 to indicator ID1-2, as shown in terminal device 10-2. In this way, the display mode of indicator ID1 changes depending on the inhalation amount of user U1. As a result, the information processing system 1 can display an inhalation guide UI (User Interface) that serves as a guide for the amount.

[0039] The information processing device 100 may transmit the content (corresponding to the display screen of the terminal device 10-2) in which the size of the indicator ID1 has been changed to the terminal device 10, and the terminal device 10 may display the received content.

[0040] Then, the user U1 makes an utterance (step S14). For example, the user U1 makes a voice input to the terminal device 10 requesting processing using the operation amount indicated by the indicator ID1. In the example of FIG. 1, the user U1 makes a voice input to the terminal device 10 requesting an operation to turn up the volume of the device by saying "Turn up volume" as shown in the speech information IN11. For example, the user U1 makes a voice input to the terminal device 10 requesting an operation to turn up the volume of the device by saying "Turn up volume".

[0041] Then, the information processing system 1 performs processing using the spoken input by the user U1 and the amount of operation (step S15). The information processing system 1 performs processing of a voice dialogue system on the input by the user U1. For example, the information processing system 1 interprets the user's input by natural language understanding (NLU) and executes a corresponding process (action). The information processing device 100 or the terminal device 10 interprets the user's input by natural language understanding (NLU) and executes a corresponding process (action).

[0042] In the example of FIG. 1, the terminal device 10 performs a process of increasing the volume using the speech information IN11 and the operation amount. The terminal device 10 performs a process of increasing the volume by the operation amount so that the size of the indicator ID1-2 is increased. The terminal device 10 performs a process of increasing the volume as shown in the terminal device 10-3. This enables the information processing system 1 to change the amount according to the amount of inhalation by the user. The display unit 16 of the terminal device 10 displays information corresponding to the process of increasing the volume as shown in the terminal device 10-3.

[0043] In this way, the information processing system 1 determines the operation amount using the user's breathing, and uses the operation amount in the process corresponding to the user's operation, thereby eliminating the need for the user to input the operation amount by voice or the like, and thus reducing the user's operation load. In the above example, the user's inhalation is associated with an increase in the operation amount or the indicator size, and the user's exhalation is associated with a decrease in the operation amount or the indicator size, but the association between the user's inhalation and the operation amount or the indicator size is not limited thereto. The information processing system 1 may also associate the user's inhalation with a decrease in the operation amount or the indicator size, and the user's exhalation with an increase in the operation amount or the indicator size. In this case, the information processing system 1 may decrease the operation amount or the indicator size when the user inhales, and increase the operation amount or the indicator size when the user exhales. The information processing system 1 may decrease the operation amount or the indicator size according to an increase in the user's inhalation amount, and increase the operation amount or the indicator size according to a decrease in the user's inhalation amount.

[0044] [1-1-1. Background and effects] In existing voice UIs, when adjusting the amount or position as desired by the user, the user needs to include specific information that the system can understand in the utterance. This makes the user's utterance longer and increases the speech load. For example, in existing voice UIs, the user needs to include specific numerical values ​​or other operation amounts in the utterance, such as "Set the volume to 50" or "Move 3 cm to the left," which makes the user's utterance longer and increases the speech load.

[0045] On the other hand, the information processing system 1 determines the amount of operation related to the user's operation based on the user's breathing. This allows the information processing system 1 to omit the user's effort of inputting information indicating the amount of operation, such as a specific numerical value, by speech. Therefore, the information processing system 1 can reduce the user's operation load and improve usability. In the example of FIG. 1, the information processing system 1 uses the breathing state before speaking as a parameter associated with the voice operation, and enables the user to perform the expected operation without entering specific numerical information in the speech. For example, when the user takes a large inhalation and then waits a little while before uttering a command, such information processing system 1 can determine the amount of operation at the time of inhalation and determine the processing content of the operation target, etc. based on the command utterance. For example, the information processing system 1 can appropriately determine the amount of operation related to the user's operation by using breathing information such as the breathing state, the amount of inhalation, the increase in the amount of inhalation, the amount of exhalation, and the increase in the amount of exhalation.

[0046] [1-1-2. Example of display based on operation amount] Note that the display based on the amount of operation is not limited to the example shown in Fig. 1, and may be in various forms. This will be described with reference to Fig. 3 and Fig. 4. Fig. 3 and Fig. 4 are diagrams showing an example of a display based on the amount of operation.

[0047] [1-1-2-1. Example of absolute value display] First, a case where the operation amount is displayed as an absolute value will be described with reference to Fig. 3. For example, the terminal device 10 may display, together with the indicator ID1, an object corresponding to a device 50 that is a candidate for an operation target (candidate for an operation target) and the operation amount corresponding to the device 50 as an absolute value. In the example of Fig. 3, the terminal device 10 displays objects OB1, OB2, and OB3 corresponding to three devices, a lighting device, a music player, and a television, respectively, as the devices 50 that are candidate for an operation target. The object OB1 corresponds to the lighting device, the object OB2 corresponds to the music player, and the object OB3 corresponds to the television. For example, the terminal device 10 displays the operation amount as an absolute value within a range from the lower limit value to the upper limit value of the settable value of each operation target.

[0048] On the display unit 16 of the terminal device 10, "20" indicating the operation amount as an absolute value is displayed below the object OB1 corresponding to the lighting device. In this case, when the lighting device is selected as the operation target, for example, it indicates that the brightness of the lighting device is set to "20".

[0049] On the display unit 16 of the terminal device 10, "10" indicating the operation amount as an absolute value is displayed below the object OB2 corresponding to the music player. In this case, when the music player is selected as the operation target, it indicates that, for example, the volume of the music player is set to "10".

[0050] On the display unit 16 of the terminal device 10, "32" indicating the operation amount as an absolute value is displayed below the object OB3 corresponding to the television. In this case, when the television is selected as the operation target, for example, it indicates that the volume of the television is set to "32".

[0051] Thus, in the example of Fig. 3, the terminal device 10 displays the amount of operation for each operation target in absolute values. For example, the user speaks "Change TV volume" as shown in the speech information IN21, and performs a voice input to request the information processing system 1 to perform an operation to change the television volume. The information processing system 1 performs a process to change the television volume (Volume) using the speech information IN21 and the amount of operation. Specifically, the information processing system 1 performs a process to change the television volume (Volume) to "32" using the speech information IN21 and the amount of operation.

[0052] [1-1-2-2. Example of relative value display] First, a case where the operation amount is displayed as a relative value will be described with reference to Fig. 4. Note that the description of the same points as in Fig. 3 will be omitted as appropriate. For example, the terminal device 10 may display an object corresponding to a device 50 that is a candidate for an operation target (candidate for an operation target) and an operation amount corresponding to the device 50 as a relative value together with the indicator ID1. The terminal device 10 may display the operation amount as a relative value indicating how much to change the operation amount from the current value. For example, the terminal device 10 may display the operation amount as a change amount from the current value. For example, the terminal device 10 displays the operation amount as a relative value within a range from the lower limit value to the upper limit value that can be set for each operation target.

[0053] On the display unit 16 of the terminal device 10, a "5" indicating the amount of operation as a relative value is displayed below the object OB1 corresponding to the lighting device. In this case, when a lighting device is selected as an operation target, it indicates that, for example, the brightness of the lighting device is changed by "5".

[0054] On the display unit 16 of the terminal device 10, "10" indicating the amount of operation as a relative value is displayed below the object OB2 corresponding to the music player. In this case, when the music player is selected as the operation target, it indicates that, for example, the volume of the music player is changed by "10".

[0055] On the display unit 16 of the terminal device 10, "20" indicating the amount of operation as a relative value is displayed below the object OB3 corresponding to the television. In this case, when the television is selected as the operation target, for example, it indicates that the volume of the television is changed by "20".

[0056] 4, the terminal device 10 displays the operation amount for each operation target as a relative value. When the relative value is displayed in this way, an increase or decrease is determined from a spoken command. That is, when the relative value is displayed, the information processing system 1 determines whether the parameter (setting value) is an increase (increase) or a decrease (decrease) based on the user's utterance.

[0057] For example, when the user utters "Turn up TV volume" as shown in the speech information IN31, the information processing system 1 determines that a voice input has been made requesting an operation to turn up the TV volume. The information processing system 1 performs a process to turn up the TV volume (Volume) using the speech information IN31 and the amount of operation. Specifically, the information processing system 1 performs a process to increase the TV volume (Volume) by "20" using the speech information IN31 and the amount of operation.

[0058] Also, for example, when the user utters "Turn down music" as shown in the speech information IN32, the information processing system 1 determines that a voice input has been made requesting an operation to turn down the volume of the music player. The information processing system 1 performs a process to turn down the volume (Volume) of the music player using the speech information IN32 and the amount of operation. Specifically, the information processing system 1 performs a process to decrease the volume (Volume) of the music player by "10" using the speech information IN32 and the amount of operation.

[0059] The above-mentioned absolute value display and relative value display are merely examples, and the information processing system 1 may not limit the use of the indicator to either an absolute value or a relative value when the indicator is displayed, and may determine which value to use based on a spoken command.

[0060] [1-1-3. Determining the target using pre-breathing speech] In the example of FIG. 1, the process is performed in the order of determining the amount of operation and then determining the operation target, but the information processing system 1 may perform the process in the order of determining the operation target and then determining the amount of operation.

[0061] For example, the information processing device 100 may determine an operation target based on the user's speech, and then determine an operation amount related to the operation by the user based on the user's breathing thereafter. In this case, the information processing system 1 may change the parameters of the operation target in real time according to the operation amount by the user's breathing. For example, when the brightness (parameters) of a lighting device is determined as the operation target, the information processing system 1 may change the brightness of the lighting device in real time according to the operation amount by the user's breathing. In this way, the information processing system 1 allows the user to intuitively grasp the change due to the operation by the user's breathing.

[0062] [1-1-4. Sensor examples] 1, a case where a millimeter wave radar is used as an example of the breathing sensor 171 that detects breathing information indicating the user's breathing has been described, but the breathing sensor 171 is not limited to a millimeter wave radar and may be any sensor that can detect the user's breathing information. An example of this point will be described below.

[0063] [1-1-4-1. Contact type] 1, the respiratory sensor 171 using a millimeter wave radar, that is, a non-contact type sensor is used to detect respiratory information, but the sensor used to detect (acquire) respiratory information is not limited to the non-contact type and may be a contact type. An example of a contact type sensor will be described below.

[0064] The respiratory sensor 171 may be a wearable sensor. The respiratory sensor 171 may be a contact sensor of various types, such as a band type, a jacket type, or a mask type.

[0065] When a band-type sensor is used for the breathing sensor 171, the information processing system 1 acquires the amount of breathing displacement from the expansion and contraction of a band wrapped around the chest or abdomen of the user. When a jacket-type sensor is used for the breathing sensor 171, the information processing system 1 embeds the band in a jacket worn by the user. Furthermore, by providing sensors in multiple locations (directions), the accuracy of breathing detection can be improved.

[0066] Furthermore, when an acceleration sensor is used for the respiration sensor 171, the information processing system 1 may estimate the respiration volume by observing the movement of the chest using an acceleration sensor mounted on a wearable device such as a neck-hanging device or a smartphone worn on the upper body of the user. Furthermore, when a mask-type sensor is used for the respiration sensor 171, the information processing system 1 detects the speed of exhalation and inhalation using an air flow sensor or air pressure sensor mounted on the mask, and estimates the depth and period from the accumulated displacement.

[0067] Furthermore, a VR (Virtual Reality) headset that covers the user's mouth may be used for the breathing sensor 171. In this case, the breathing sensor 171 that performs breathing sensing with a noise blocking microphone allows disadvantages in the real world to be ignored since VR is being used. Furthermore, when a close microphone is used for the breathing sensor 171, the information processing system 1 recognizes the sound of exhaled breath with the close microphone, recognizes the amount of change in exhaled breath over time, and estimates the depth and speed of breathing. For example, the information processing system 1 recognizes the sound of noise generated when the exhaled breath hits the microphone with the close microphone, recognizes the amount of change in exhaled breath over time, and estimates the depth and speed of breathing.

[0068] [1-1-4-2. Non-contact type] Furthermore, the non-contact sensor is not limited to the millimeter wave radar, and various non-contact sensors may be used for the respiration sensor 171. Below, examples of non-contact sensors other than the millimeter wave radar will be described.

[0069] The respiration sensor 171 may use an image sensing method, a method for detecting respiration from the temperature around the nose, a proximity sensor, or a radar other than a millimeter wave radar.

[0070] When image sensing is used for the breathing sensor 171, the information processing system 1 recognizes the amount of change over time of inhalation and exhalation, which have different temperatures, using a thermo camera, and estimates the depth, cycle, and speed of breathing. The information processing system 1 may also recognize the amount of change over time of exhalation by image sensing, which turns white when it is cold, and estimate the depth, cycle, and speed of breathing.

[0071] For example, the following disclosure is made regarding a technique for detecting respiration from the temperature around the nose. - Non-intrusive breathing sensing method<https: / / shingi.jst.go.jp / past_abst / abst / p / 09 / 919 / tama2.pdf>

[0072] For example, the following disclosure is made regarding breath detection using a proximity sensor. Capacitive film-type proximity sensor that monitors human movement and breathing<https: / / www.aist.go.jp / aist_j / press_release / pr2016 / pr20160125 / pr20160125.html>

[0073] For example, the following disclosure is made regarding respiration detection using radar. Heart rate and respiration detection sensor "GZS-350 series"<https: / / www.ipros.jp / product / detail / 2000348329 / >

[0074] 1, when a millimeter wave radar is used as the breathing sensor 171, the information processing system 1 detects the movement of the user's chest from the phase difference of the reception signal of the millimeter wave radar and estimates the breathing amount. For example, the terminal device 10 uses the sensor information detected by the breathing sensor 171 to detect the movement of the user's chest from the phase difference of the reception signal of the millimeter wave radar and estimates the breathing amount, thereby generating the user's breathing information. Then, the terminal device 10 transmits the generated user's breathing information to the information processing device 100.

[0075] Alternatively, the information processing device 100 may generate the respiratory information of the user. In this case, the terminal device 10 transmits sensor information detected by the respiratory sensor 171 to the information processing device 100. Then, the information processing device 100 may use the received sensor information to detect the movement of the user's chest based on the phase difference of the received signal of the millimeter wave radar and estimate the respiratory volume to generate the respiratory information of the user.

[0076] The above sensors are merely examples of sensors used to acquire respiratory information, and any sensor may be used as long as it can acquire respiratory information. The information processing system 1 may use any sensor to detect respiratory information indicating the user's breathing.

[0077] The sensor unit 17 of the terminal device 10 has at least one of the above-mentioned sensors and detects respiratory information by the sensor. The information processing system 1 may generate respiratory information by using sensor information detected by the sensor of the sensor unit 17. For example, the terminal device 10 or the information processing device 100 may generate respiratory information by using sensor information (point cloud data) detected by the respiratory sensor 171 (millimeter wave radar). In this case, the terminal device 10 or the information processing device 100 may generate respiratory information from the sensor information (point cloud data) detected by the respiratory sensor 171 (millimeter wave radar) by appropriately using various techniques.

[0078] [1-2. Configuration of information processing system according to embodiment] An information processing system 1 shown in FIG. 5 will be described. As shown in FIG. 5, the information processing system 1 includes a terminal device 10, an information processing device 100, and a plurality of devices 50-1, 50-2, and 50-3. In the following, when the devices 50-1 to 50-3 are not distinguished from each other, they may be referred to as devices 50. Although three devices 50-1, 50-2, and 50-3 are illustrated in FIG. 5, the information processing system 1 may include more than three devices 50 (for example, 20 or 100 or more). The terminal device 10, the information processing device 100, and the devices 50 are connected to each other via a predetermined communication network (network N) so as to be communicable by wire or wirelessly. FIG. 5 is a diagram showing a configuration example of an information processing system according to an embodiment. The information processing system 1 shown in FIG. 5 may include a plurality of terminal devices 10 and a plurality of information processing devices 100.

[0079] The information processing device 100 is a computer that predicts whether or not the user will speak after the user inhales, based on breathing information indicating the user's breathing. The information processing device 100 classifies the user's inhalation, based on the user's breathing information. The information processing device 100 is also a computer that transmits various information to the terminal device 10. The information processing device 100 is a server device used to provide services related to various functions.

[0080] Furthermore, the information processing device 100 may have software modules such as speech signal processing, speech recognition, speech semantic analysis, and dialogue control. The information processing device 100 may have a speech recognition function. For example, the information processing device 100 may have a natural language understanding (NLU) or automatic speech recognition (ASR) function. For example, the information processing device 100 may estimate information related to a user's intent or an entity (target) from input information by the user's speech. The information processing device 100 functions as a speech recognition server having the functions of natural language understanding and automatic speech recognition.

[0081] The terminal device 10 is a terminal device that detects respiratory information indicating the user's breathing by a sensor. For example, the terminal device 10 detects respiratory information indicating the user's breathing by a respiratory sensor 171. The terminal device 10 is an information processing device that transmits the user's respiratory information to a server device such as an information processing device 100. The terminal device 10 may also have a voice recognition function such as natural language understanding or automatic voice recognition. For example, the terminal device 10 may estimate information related to a user's intent (intention) or an entity (target) from input information by the user's speech.

[0082] The terminal device 10 is a device used by a user. The terminal device 10 accepts input from the user. The terminal device 10 accepts voice input by the user's speech and input by the user's operation. The terminal device 10 displays information according to the user's input. The terminal device 10 may be any device as long as it can realize the processing in the embodiment. The terminal device 10 may be any device as long as it has a function of detecting the user's breathing information and transmitting it to the information processing device 100. For example, the terminal device 10 may be a smartphone, a smart speaker, a television, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or other device. The terminal device 10 may be a wearable device worn by a user. For example, the terminal device 10 may be a wristwatch terminal, a glasses terminal, or other device.

[0083] The devices 50 are various devices used by users. The devices 50 are various devices such as IoT (Internet of Things) devices. The devices 50 are IoT devices such as home appliances. For example, the devices 50 may be any devices that have a communication function, communicate with the information processing device 100 or the terminal device 10, and can perform processing in response to an operation request from the information processing device 100 or the terminal device 10. For example, the devices 50 may be so-called home appliances such as lighting equipment (lighting devices), music players, televisions, radios, air conditioners (air conditioners), washing machines, and refrigerators, or may be products installed in homes such as ventilation fans and floor heating. In the example of FIG. 5, the devices 50-1 may be lighting devices, the devices 50-2 may be music players, and the devices 50-3 may be televisions.

[0084] The device 50 may be, for example, an information processing device such as a smartphone, a tablet terminal, a notebook PC, a desktop PC, a mobile phone, or a PDA. For example, the device 50 may be any device that can realize the processing in the embodiment. The device 50 may include a terminal device 10. That is, the device to be operated using the operation amount based on the user's breathing may be the terminal device 10.

[0085] [1-3. Configuration of the information processing device according to the embodiment] Next, a configuration of the information processing device 100, which is an example of an information processing device that executes information processing according to an embodiment, will be described. Fig. 6 is a diagram showing an example of the configuration of the information processing device 100 according to an embodiment of the present disclosure.

[0086] 6, the information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The information processing device 100 may also include an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator of the information processing device 100, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

[0087] The communication unit 110 is realized by, for example, a network interface card (NIC) or the like. The communication unit 110 is connected to a network N (see FIG. 5) by wire or wirelessly, and transmits and receives information to and from other information processing devices such as the terminal device 10. The communication unit 110 may also transmit and receive information to and from a user terminal (not shown) used by a user.

[0088] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 6, the storage unit 120 according to the embodiment includes a respiratory information storage unit 121, a user information storage unit 122, a threshold information storage unit 123, and a function information storage unit 124.

[0089] The storage unit 120 stores various information other than the above. The storage unit 120 stores information on a voice recognition application (program) that realizes a voice recognition function. For example, the information processing device 100 can execute voice recognition by starting a voice recognition application (also simply called "voice recognition"). The storage unit 120 stores various information used for voice recognition. The storage unit 120 stores information on a dictionary (voice recognition dictionary) used in the voice recognition dictionary. The storage unit 120 stores information on a plurality of voice recognition dictionaries. The storage unit 120 stores information on a voice recognition dictionary for long sentences (dictionary for long sentences), a voice recognition dictionary for Chinese sentences (dictionary for Chinese sentences), a voice recognition dictionary for short sentences (dictionary for words and phrases), etc.

[0090] The respiratory information storage unit 121 according to the embodiment stores various information related to the user's breathing. The respiratory information storage unit 121 stores various information such as the respiratory information of each user in association with the identification information (user ID) of each user. The respiratory information storage unit 121 stores respiratory information indicating the user's breathing. The respiratory information storage unit 121 stores respiratory information including the user's intake volume. The respiratory information storage unit 121 stores respiratory information including the amount of change in the user's breathing. The respiratory information storage unit 121 stores speech information indicating the user's speech after the user's breathing. The respiratory information storage unit 121 stores speech information indicating the user's speech before the user's breathing.

[0091] The respiratory information storage unit 121 may store various information according to the purpose, not limited to the above. For example, the respiratory information storage unit 121 may store various information required for generating the graphs GR1 to GR6. For example, the respiratory information storage unit 121 may store various information shown in the graphs GR1 to GR6.

[0092] The user information storage unit 122 according to the embodiment stores various pieces of information related to users. For example, the user information storage unit 122 stores various pieces of information such as attribute information of each user.

[0093] The user information storage unit 122 stores information about the user, such as a user ID, age, sex, and place of residence. For example, the user information storage unit 122 stores information about the user U1, such as the age, sex, and place of residence of the user U1, in association with a user ID “U1” that identifies the user U1.

[0094] The user information storage unit 122 also stores information for identifying a device (such as a television or a smartphone) used by each user in association with the user. The user information storage unit 122 also stores information for identifying a terminal device 10 used by each user (such as a terminal ID) in association with the user.

[0095] The user information storage unit 122 may store various information according to the purpose, not limited to the above. For example, the user information storage unit 122 may store other demographic attribute information and psychographic attribute information in addition to age and gender. For example, the user information storage unit 122 may store information such as name, home, place of work, interests, family structure, income, and lifestyle.

[0096] The threshold information storage unit 123 according to the embodiment stores various information related to thresholds. The threshold information storage unit 123 stores various information related to thresholds used to determine the amount of operation. Fig. 7 is a diagram showing an example of the threshold information storage unit according to the embodiment. The threshold information storage unit 123 shown in Fig. 7 includes items such as "threshold ID", "purpose", "threshold name", and "value".

[0097] "Threshold ID" indicates identification information for identifying a threshold. "Use" indicates the use of the threshold. "Threshold name" indicates the name (character string) of the threshold (variable) used as the threshold identified by the corresponding threshold ID. "Value" indicates the specific value of the threshold identified by the corresponding threshold ID.

[0098] In the example of FIG. 7, the use of the threshold (threshold TH1) identified by the threshold ID "TH1" indicates that it is a threshold used for comparison with the amount of increase (inhalation). The threshold TH1 is used as the threshold name "B_increase_thres". The value of threshold TH1 is indicated as "VL1". Note that in FIG. 7, the value is indicated by an abstract code such as "VL1", but the value is assumed to be a concrete numerical value such as "0.5" or "1.8".

[0099] Also, in the example of FIG. 7, the use of the threshold (threshold TH2) identified by the threshold ID "TH2" is shown to be a threshold used for comparison with the increase in exhaust (exhaust increase amount). The threshold TH2 is shown to be used as the threshold name "B_decrease_thres". The value of threshold TH2 is shown to be "VL2". Note that in FIG. 7, the value is shown as an abstract code such as "VL2", but the value is assumed to be a concrete numerical value such as "-0.3" or "-2.5".

[0100] 7 illustrates only the thresholds TH1 and TH2 used to determine the amount of operation, but thresholds used to switch between displaying and not displaying the indicator may also be stored in the threshold information storage unit 123. Furthermore, the threshold information storage unit 123 may store various information according to the purpose, without being limited to the above.

[0101] The function information storage unit 124 according to the embodiment stores various information related to functions. The function information storage unit 124 stores information related to each function executed in response to a user's input. The function information storage unit 124 stores information related to an input required for executing a function. The function information storage unit 124 stores input items required for executing each function.

[0102] The function information storage unit 124 may also store various information related to the device. The function information storage unit 124 stores various information related to the device corresponding to each function. For example, the function information storage unit 124 can communicate with the information processing device 100 and stores various information of the device that can be an operation target. The function information storage unit 124 may store a device ID indicating identification information for identifying the device and device type information indicating the type of the corresponding device. The function information storage unit 124 stores functions and parameters possessed by each device in association with each device. The function information storage unit 124 stores information indicating the state of the device in association with each device. For example, the function information storage unit 124 stores various information such as parameter values ​​of each device at that time in association with each device. For example, the function information storage unit 124 stores various information such as parameter values ​​of each device at the current time (at the time when information was last acquired) in association with each device. When the device is a television, the function information storage unit 124 stores the current on / off state, volume, brightness, channel, etc. in association with the device ID. When the device is a lighting fixture, the function information storage unit 124 stores the current on / off state, brightness, color, etc. in association with the device ID.

[0103] The function information storage unit 124 is not limited to the above, and may store various kinds of information according to the purpose.

[0104] Returning to Fig. 6, the description will be continued. The control unit 130 is realized, for example, by a central processing unit (CPU), a micro processing unit (MPU), or the like executing a program (for example, an information processing program according to the present disclosure) stored inside the information processing device 100 using a random access memory (RAM) or the like as a working area. The control unit 130 is also realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0105] As shown in Fig. 6, the control unit 130 has an acquisition unit 131, a prediction unit 132, a determination unit 133, an execution unit 134, and a transmission unit 135, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Fig. 6, and may be other configurations as long as they perform the information processing described below. Also, the connection relationship between the processing units in the control unit 130 is not limited to the connection relationship shown in Fig. 6, and may be other connection relationships.

[0106] The acquisition unit 131 acquires various information. The acquisition unit 131 acquires various information from an external information processing device. The acquisition unit 131 acquires various information from the terminal device 10. The acquisition unit 131 acquires various information from the terminal device 10, that is, information detected by the sensor unit 17 of the terminal device 10. The acquisition unit 131 acquires various information from the terminal device 10, that is, information detected by the respiratory sensor 171 of the sensor unit 17.

[0107] The acquisition unit 131 acquires various information from the storage unit 120. The acquisition unit 131 acquires various information from the respiratory information storage unit 121, the user information storage unit 122, the threshold information storage unit 123, and the functional information storage unit 124. The acquisition unit 131 acquires various information predicted by the prediction unit 132. The acquisition unit 131 acquires various information determined by the determination unit 133.

[0108] The acquisition unit 131 acquires respiratory information indicating the breathing of the user. The acquisition unit 131 acquires respiratory information including the amount of intake of the user. The acquisition unit 131 acquires respiratory information including the amount of change in the user's breathing. The acquisition unit 131 acquires speech information indicating an utterance by the user after the user breathes. The acquisition unit 131 acquires speech information indicating an utterance by the user before the user breathes. The acquisition unit 131 acquires respiratory information indicating the breathing of the user U1 from the terminal device 10 used by the user U1.

[0109] The prediction unit 132 predicts various information. The prediction unit 132 classifies various information. The prediction unit 132 calculates various information. The prediction unit 132 determines various information. The prediction unit 132 makes various judgments. The prediction unit 132 judges various information. For example, the prediction unit 132 predicts various information based on information from an external information processing device or information stored in the storage unit 120. The prediction unit 132 predicts various information based on information from another information processing device such as the terminal device 10. The prediction unit 132 predicts various information based on information stored in the respiratory information storage unit 121, the user information storage unit 122, the threshold information storage unit 123, or the function information storage unit 124. The prediction unit 132 classifies various information. For example, the prediction unit 132 classifies various information based on information from an external information processing device or information stored in the storage unit 120.

[0110] The prediction unit 132 predicts various pieces of information based on various pieces of information acquired by the acquisition unit 131. The prediction unit 132 predicts various pieces of information based on various pieces of information determined by the determination unit 133. The prediction unit 132 makes various decisions based on the predictions. The prediction unit 132 makes various decisions based on the information acquired by the acquisition unit 131. The prediction unit 132 calculates a score based on the breathing information. The prediction unit 132 calculates a score used for predicting a user's speech based on the breathing information.

[0111] The prediction unit 132 predicts an amount of operation based on the breathing information acquired by the acquisition unit 131. The prediction unit 132 predicts an amount of intake of the user based on the breathing information acquired by the acquisition unit 131. The prediction unit 132 predicts an amount of operation (predicted amount of operation) based on the predicted amount of intake (predicted amount of intake). The prediction unit 132 calculates a predicted amount of operation based on the predicted amount of intake.

[0112] The determination unit 133 determines various information. The determination unit 133 judges various information. The determination unit 133 calculates various information. The determination unit 133 extracts various information. The determination unit 133 identifies various information. The determination unit 133 selects various information. The determination unit 133 determines various information based on information from an external information processing device or information stored in the storage unit 120. The determination unit 133 determines various information based on information from another information processing device such as the terminal device 10. The determination unit 133 determines various information based on information stored in the respiratory information storage unit 121, the user information storage unit 122, the threshold information storage unit 123, or the function information storage unit 124.

[0113] The determination unit 133 determines various pieces of information based on the various pieces of information acquired by the acquisition unit 131. The determination unit 133 determines various pieces of information based on the various pieces of information predicted by the prediction unit 132. The determination unit 133 determines various pieces of information based on the processing executed by the execution unit 134.

[0114] The determination unit 133 determines the amount of operation related to the operation by the user based on the respiration of the user indicated by the respiration information acquired by the acquisition unit 131. The determination unit 133 determines the amount of operation according to the amount related to the respiration of the user. The determination unit 133 determines the amount of operation based on the amount of inhalation. The determination unit 133 determines the amount of operation based on the amount of change. For example, the determination unit 133 calculates an indicator parameter "P_indicator" indicating the amount of operation by using a current amount of inhalation "B_current" and the following formula (1) until the maximum inhalation. For example, the determination unit 133 calculates an indicator parameter "P_indicator" indicating the amount of operation by using a maximum amount of inhalation "B_max", a time variable "Time", and the following formula (2) after the maximum inhalation. For example, the determination unit 133 determines the amount of operation related to the operation by the user U1 using the calculated amount of operation (indicator parameter). For example, the determination unit 133 determines the calculated amount of operation (indicator parameter) as the amount of operation related to the operation by the user U1.

[0115] The determination unit 133 increases the amount of operation in response to either one of the user's inhalation or exhalation. The determination unit 133 decreases the amount of operation in response to the other of the user's inhalation or exhalation, which is different from one of the two. The determination unit 133 increases the amount of operation in response to the user's inhalation. The determination unit 133 increases the amount of operation in response to an increase in the amount of inhalation when the user inhales. When a comparison result between the increase in the amount of inhalation of the user and a threshold value satisfies a predetermined condition, the determination unit 133 increases the amount of operation in response to the increase in the amount of inhalation. The determination unit 133 decreases the amount of operation constantly while the user is exhaling.

[0116] The determination unit 133 reduces the amount of operation in response to the user's exhaust. The determination unit 133 reduces the amount of operation in response to a decrease in the amount of intake air when the user exhausts. When a comparison result between the increase in the user's exhaust and a threshold value satisfies a predetermined condition, the determination unit 133 reduces the amount of operation in response to the increase in the exhaust. The determination unit 133 determines a target for operation by the user based on utterance information.

[0117] The determination unit 133 determines an object to be operated by the user based on the speech information. The determination unit 133 determines an amount of operation corresponding to a parameter of the device. The determination unit 133 determines an amount of operation corresponding to a position change of the object displayed on the screen.

[0118] The execution unit 134 executes various processes. The execution unit 134 decides to execute various processes. The execution unit 134 executes various processes based on information from an external information processing device. The execution unit 134 executes various processes based on information stored in the storage unit 120. The execution unit 134 executes various processes based on information stored in the respiratory information storage unit 121, the user information storage unit 122, the threshold information storage unit 123, and the function information storage unit 124.

[0119] The execution unit 134 executes various processes based on the various information acquired by the acquisition unit 131. The execution unit 134 executes various processes based on the various information predicted by the prediction unit 132. The execution unit 134 executes various processes based on the various information determined by the determination unit 133.

[0120] The execution unit 134 generates various information. The execution unit 134 generates various information based on information from an external information processing device or information stored in the storage unit 120. The execution unit 134 generates various information based on information from another information processing device such as the terminal device 10. The execution unit 134 generates various information based on information stored in the respiratory information storage unit 121, the user information storage unit 122, the threshold information storage unit 123, and the function information storage unit 124.

[0121] The execution unit 134 executes a process according to the amount of operation determined by the determination unit 133. The execution unit 134 executes a process according to the result of the prediction by the prediction unit 132.

[0122] The transmitting unit 135 transmits various information. The transmitting unit 135 transmits various information to an external information processing device. The transmitting unit 135 provides various information to an external information processing device. For example, the transmitting unit 135 transmits various information to another information processing device such as the terminal device 10. The transmitting unit 135 provides information stored in the memory unit 120. The transmitting unit 135 transmits information stored in the memory unit 120.

[0123] The transmission unit 135 provides various information based on information from other information processing devices such as the terminal device 10. The transmission unit 135 provides various information based on information stored in the storage unit 120. The transmission unit 135 provides various information based on information stored in the respiratory information storage unit 121, the user information storage unit 122, the threshold information storage unit 123, and the function information storage unit 124.

[0124] The transmission unit 135 transmits, to the terminal device 10, information indicating a function to be executed by the terminal device 10. The transmission unit 135 transmits, to the terminal device 10, information indicating a function determined to be executed by the execution unit 134. The transmission unit 135 transmits various information to the terminal device 10 in response to an instruction from the execution unit 134. The transmission unit 135 transmits, to the terminal device 10, information instructing the terminal device 10 to start a voice recognition application.

[0125] The transmission unit 135 transmits information indicating the amount of operation determined by the determination unit 133 to the terminal device 10 as a sound. The transmission unit 135 transmits the predicted amount of operation predicted by the prediction unit 132 to the terminal device 10 as a sound.

[0126] [1-4. Configuration of terminal device according to embodiment] Next, a configuration of the terminal device 10, which is an example of an information processing device that executes information processing according to an embodiment, will be described. Fig. 8 is a diagram illustrating an example of the configuration of the terminal device according to an embodiment of the present disclosure.

[0127] As shown in FIG. 8, the terminal device 10 includes a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, a control unit 15, a display unit 16, and a sensor unit 17.

[0128] The communication unit 11 is realized by, for example, a NIC, a communication circuit, etc. The communication unit 11 is connected to a network N (such as the Internet) by wire or wirelessly, and transmits and receives information to and from other devices such as the information processing device 100 via the network N.

[0129] The input unit 12 accepts various inputs. The input unit 12 accepts detection by the sensor unit 17 as input. The input unit 12 accepts input of respiratory information indicating the user's breathing. The input unit 12 accepts input of respiratory information detected by the sensor unit 17. The input unit 12 accepts input of respiratory information detected by the respiratory sensor 171. The input unit 12 accepts input of respiratory information based on point cloud data detected by the respiratory sensor 171. The input unit 12 accepts input of user's speech information. The input unit 12 accepts input of user's respiratory information that inputs by body movement. The input unit 12 accepts the user's gestures and gaze as input.

[0130] Various operations are input to the input unit 12 by the user. The input unit 12 receives sound as input by a sensor unit 17 having a function of detecting sound. The input unit 12 receives voice information detected by a microphone (sound sensor) that detects sound as input information. The input unit 12 receives voice uttered by the user as input information.

[0131] The input unit 12 may also receive an operation (user operation) on the terminal device 10 used by the user as an operation input by the user. The input unit 12 may receive information on a user operation using a remote controller via the communication unit 11. The input unit 12 may also have a button provided on the terminal device 10, or a keyboard or a mouse connected to the terminal device 10.

[0132] For example, the input unit 12 may have a touch panel that can realize the same functions as a remote control, a keyboard, or a mouse. In this case, various information is input to the input unit 12 via the display unit 16. The input unit 12 accepts various operations from the user via the display screen by the function of the touch panel realized by various sensors. That is, the input unit 12 accepts various operations from the user via the display unit 16 of the terminal device 10. For example, the input unit 12 accepts operations such as a user's designated operation via the display unit 16 of the terminal device 10. For example, the input unit 12 functions as a reception unit that accepts user operations by the function of the touch panel. In this case, the input unit 12 and the reception unit 153 may be integrated. Note that, in tablet terminals, the electrostatic capacitance method is mainly used as a method for detecting user operations by the input unit 12, but any method may be used as long as it can detect user operations and realize the function of a touch panel, such as a resistive film method, a surface acoustic wave method, an infrared method, or an electromagnetic induction method.

[0133] For example, the input unit 12 accepts an utterance of the user U1 as an input. The input unit 12 accepts an utterance of the user U1 detected by the sensor unit 17 as an input. The input unit 12 accepts an utterance of the user U1 detected by the sound sensor of the sensor unit 17 as an input.

[0134] The output unit 13 outputs various information. The output unit 13 has a function of outputting sound. For example, the output unit 13 has a speaker that outputs sound. The output unit 13 outputs various information by sound in response to control by the execution unit 152. The output unit 13 outputs information by sound to the user. The output unit 13 outputs information displayed on the display unit 16 by sound.

[0135] The memory unit 14 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The memory unit 14 stores information on a voice recognition application (program) that realizes a voice recognition function. For example, the terminal device 10 can execute voice recognition by starting a voice recognition application. The memory unit 14 stores various information used to display information. The memory unit 14 stores various information used for voice recognition. The memory unit 14 stores information on a dictionary used in a voice recognition dictionary (voice recognition dictionary).

[0136] Returning to Fig. 8, the description will be continued. The control unit 15 is realized, for example, by a CPU, an MPU, or the like executing a program (for example, an information processing program according to the present disclosure) stored inside the terminal device 10 using a RAM or the like as a working area. The control unit 15 may also be realized, for example, by an integrated circuit such as an ASIC or an FPGA.

[0137] 8, control unit 15 has receiving unit 151, executing unit 152, accepting unit 153, and transmitting unit 154, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of control unit 15 is not limited to the configuration shown in FIG. 8, and may be other configurations as long as they perform information processing described below.

[0138] The receiving unit 151 receives various information. The receiving unit 151 receives various information from an external information processing device. The receiving unit 151 receives various information from another information processing device such as the information processing device 100.

[0139] The receiving unit 151 receives information instructing the start of voice recognition from the information processing device 100. The receiving unit 151 receives information instructing the start of a voice recognition application from the information processing device 100.

[0140] The receiving unit 151 receives execution instructions for various functions from the information processing device 100. For example, the receiving unit 151 receives information specifying a function from the information processing device 100 as an execution instruction for the function. The receiving unit 151 receives content. The receiving unit 151 receives content to be displayed from the information processing device 100. The receiving unit 151 receives information indicating an amount of operation determined by the information processing device 100. The receiving unit 151 displays content including an indicator corresponding to the amount of operation.

[0141] The execution unit 152 executes various processes. The execution unit 152 decides to execute various processes. The execution unit 152 executes various processes based on information from an external information processing device. The execution unit 152 executes various processes based on information from the information processing device 100. The execution unit 152 executes various processes in response to instructions from the information processing device 100. The execution unit 152 executes various processes based on information stored in the storage unit 14. The execution unit 152 starts voice recognition.

[0142] The execution unit 152 controls various outputs. The execution unit 152 controls audio output by the output unit 13. The execution unit 152 controls various displays. The execution unit 152 controls the display of the display unit 16. The execution unit 152 controls the display of the display unit 16 in response to reception by the receiving unit 151. The execution unit 152 controls the display of the display unit 16 based on the information received by the receiving unit 151. The execution unit 152 controls the display of the display unit 16 based on the information accepted by the accepting unit 153. The execution unit 152 controls the display of the display unit 16 in response to acceptance by the accepting unit 153.

[0143] The reception unit 153 receives various types of information. The reception unit 153 receives input by the user via the input unit 12. The reception unit 153 receives speech by the user as input. The reception unit 153 receives operations by the user. The reception unit 153 receives operations by the user on information displayed by the display unit 16. The reception unit 153 receives character input by the user.

[0144] The transmission unit 154 transmits various information to an external information processing device. For example, the transmission unit 154 transmits various information to another information processing device such as the terminal device 10. The transmission unit 154 transmits information stored in the storage unit 14.

[0145] The transmission unit 154 transmits various types of information based on information from other information processing devices such as the information processing device 100. The transmission unit 154 transmits various types of information based on information stored in the storage unit .

[0146] The transmission unit 154 transmits sensor information detected by the sensor unit 17 to the information processing device 100. The transmission unit 154 transmits respiratory information of the user U1 detected by the respiratory sensor 171 of the sensor unit 17 to the information processing device 100.

[0147] The transmission unit 154 transmits input information input by a user to the information processing device 100. The transmission unit 154 transmits input information input by voice by the user to the information processing device 100. The transmission unit 154 transmits input information input by a user operation to the information processing device 100.

[0148] The transmitting unit 154 transmits respiratory information indicating the user's breathing to the information processing device 100. The transmitting unit 154 transmits respiratory information including the user's inhalation volume to the information processing device 100. The transmitting unit 154 transmits respiratory information including the amount of change in the user's breathing to the information processing device 100. The transmitting unit 154 transmits speech information indicating an utterance by the user after the user's breathing to the information processing device 100. The transmitting unit 154 transmits speech information indicating an utterance by the user before the user's breathing to the information processing device 100.

[0149] The display unit 16 is provided in the terminal device 10 and displays various information. The display unit 16 is realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 16 may be realized by any means as long as it is capable of displaying information provided by the information processing device 100. The display unit 16 displays various information in response to control by the execution unit 152.

[0150] The display unit 16 displays various information based on information from the information processing device 100. The display unit 16 displays information received from the information processing device 100. The display unit 16 displays information indicating an amount of operation received from the information processing device 100. The display unit 16 displays information indicating an amount of operation determined by the information processing device 100. The display unit 16 displays an indicator corresponding to the amount of operation. The display unit 16 displays content received from the information processing device 100. The display unit 16 displays content including an indicator corresponding to the amount of operation.

[0151] The display unit 16 displays the content. The display unit 16 displays the content received by the receiving unit 151.

[0152] The sensor unit 17 detects predetermined information. The sensor unit 17 detects breathing information of the user. The sensor unit 17 has a breathing sensor 171 as a means for detecting breathing information indicating the user's breathing. The sensor unit 17 detects the breathing information by the breathing sensor 171.

[0153] The sensor unit 17 detects respiratory information by the respiratory sensor 171. The sensor unit 17 detects respiratory information by the respiratory sensor 171 using a millimeter wave radar. The sensor unit 17 is not limited to a millimeter wave radar, and may have a respiratory sensor 171 of any configuration as long as it can detect the user's respiratory information. The respiratory sensor 171 may be an image sensor. The respiratory sensor 171 may be a wearable sensor. The respiratory sensor 171 may be either a contact type or a non-contact type sensor.

[0154] The sensor unit 17 may have various sensors other than those described above. The sensor unit 17 may have a sensor (position sensor) that detects position information such as a GPS (Global Positioning System) sensor. The sensor unit 17 may have various sensors other than those described above. The terminal device 10 may have a light source (light source unit) such as an LED (Light Emitting Diode) for notifying the user by light. For example, the light source unit blinks in response to control by the execution unit 152.

[0155] [1-5. Information processing procedure according to the embodiment] Next, the procedure of various information processes according to the embodiment will be described with reference to FIGS.

[0156] [1-5-1. Processing procedure related to information processing device] First, a flow of processing related to the information processing device according to the embodiment of the present disclosure will be described with reference to Fig. 9. Fig. 9 is a flowchart showing a processing procedure of the information processing device according to the embodiment of the present disclosure. Specifically, Fig. 9 is a flowchart showing the procedure of information processing by the information processing device 100.

[0157] 9, the information processing device 100 acquires respiration information indicating the respiration of the user (step S101). Then, the information processing device 100 determines an operation amount related to an operation by the user based on the respiration of the user indicated by the respiration information (step S102).

[0158] [1-5-2. Processing procedures related to information processing systems] Next, a process flow related to the information processing system according to the embodiment of the present disclosure will be described with reference to Fig. 10. Fig. 10 is a sequence diagram showing a process procedure of the information processing system according to the embodiment of the present disclosure.

[0159] 10, the terminal device 10 detects respiratory information indicating the user's breathing (step S201). For example, the terminal device 10 acquires the user's respiratory information detected by the respiratory sensor 171. Then, the terminal device 10 transmits the respiratory information indicating the user's breathing to the information processing device 100 (step S202).

[0160] The information processing device 100 determines the amount of operation related to the operation by the user based on the respiration of the user indicated by the respiration information acquired from the terminal device 10 (step S203). In the example of Fig. 10, the information processing device 100 determines the amount of operation to be a value "10" based on the respiration information acquired from the terminal device 10.

[0161] Then, the information processing device 100 transmits the determined amount of operation to the terminal device 10 (step S204). For example, the information processing device 100 transmits the determined value of the amount of operation, "10", to the terminal device 10. Then, the terminal device 10 displays an indicator corresponding to the value of the amount of operation, "10", received from the information processing device 100 (step S205). For example, the terminal device 10 displays an indicator of a size corresponding to the value of the amount of operation, "10", on a display (display unit 16, etc.).

[0162] [1-5-3. Specific examples of processing related to information processing systems] Next, an example of a specific process related to the information processing system will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the process of the information processing system according to the embodiment of the present disclosure. Note that, although the case where the information processing system 1 performs the process will be described below as an example, the process shown in Fig. 11 may be performed by either the information processing device 100 or the terminal device 10 included in the information processing system 1.

[0163] 11, the information processing system 1 acquires a Wuw (Wake up Word) or a trigger (step S301). For example, the information processing system 1 acquires an utterance by a user for activating voice recognition or a predetermined input serving as a trigger.

[0164] Then, the information processing system 1 acquires inhalation and exhalation data (step S302) For example, the information processing system 1 acquires data (breathing information) indicating the inhalation and exhalation of the user per unit time.

[0165] Then, the information processing system 1 calculates parameters such as an increase amount, a decrease amount, etc. (Step S303) For example, the information processing system 1 calculates a change amount of respiration, which is an increase amount or decrease amount of inhalation per unit time.

[0166] Then, the information processing system 1 determines a processing condition such as a sudden intake or exhaust (step S304). For example, when there is a sudden intake or exhaust, the information processing system 1 determines the amount of operation by a calculation method corresponding to the sudden intake or exhaust. For example, the information processing system 1 determines the amount of operation by a calculation method corresponding to the user's respiratory state, among the calculation methods shown in graphs GR1 to GR5.

[0167] Then, the information processing system 1 reflects the parameter in a GUI (Graphical User Interface) (step S305). For example, the information processing system 1 displays an indicator corresponding to the determined amount of operation (parameter).

[0168] If speech has not started (step S306: No), the information processing system 1 returns to step S302 and repeats the process.

[0169] Furthermore, when speech is started (step S306: Yes), the information processing system 1 acquires the user's speech (step S307). For example, the information processing system 1 detects the user's speech through the terminal device 10 used by the user. In the case of input area selection, the information processing system 1 inputs information into the selected area and ends the process. For example, in the case of input area selection, the information processing system 1 inputs information corresponding to the speech into the selected area and ends the process.

[0170] Then, the information processing system 1 determines whether or not the action requires a parameter (step S308). For example, the information processing system 1 determines whether or not the process (action) corresponding to the user's utterance is an action that requires an operation amount (parameter).

[0171] If the action does not require a parameter (step S308: No), the information processing system 1 executes the action (step S309). For example, if the process (action) corresponding to the user's utterance does not require an operation amount (parameter), the information processing system 1 executes the process (action) corresponding to the user's utterance.

[0172] If the action requires a parameter (step S308: Yes), the information processing system 1 determines whether there is linguistic parameter information (step S310). For example, the information processing system 1 determines whether the user's utterance includes information indicating an operation amount (parameter).

[0173] When there is no linguistic parameter information (step S310: No), the information processing system 1 uses the respiration data as an execution parameter (step S311). For example, when the user's utterance does not include information indicating an operation amount (parameter), the information processing system 1 executes a process (action) using an operation amount (parameter) determined based on the user's respiration.

[0174] On the other hand, if there is linguistic parameter information (step S310: Yes), the information processing system 1 acquires parameters from the linguistic information and executes the parameters (step S312). For example, if the user's utterance includes information indicating an operation amount (parameter), the information processing system 1 executes a process (action) using the operation amount (parameter) based on the user's utterance.

[0175] [1-6. Example of reflection on respiratory control volume] Reflection of the respiration in the manipulated variable is not limited to the above-mentioned manner, and various manners may be used. This point will be described below. In the following description, the same points as those in FIG. 2 will be omitted as appropriate.

[0176] [1-6-1. Example of reflection during sudden inspiration] An example of how rapid inhalation is reflected will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of how the user's breathing is reflected in the amount of operation.

[0177] Graph GR2 in FIG. 12 is a graph showing the relationship between the user's breathing and the operation amount. Specifically, graph GR2 in FIG. 12 is a graph showing the relationship between the user's inhalation amount and the size of the indicator based on the operation amount (indicator size). Graph GR2 in FIG. 12 is a graph in which the vertical axis indicates the inhalation amount or the size of the indicator, and the horizontal axis indicates time. In addition, in graph GR2 in FIG. 12, a line indicating the breathing amount (line LN21) is shown by a solid line, and a line indicating the indicator size (line LN22) is shown by a dashed line. In addition, in graph GR2 in FIG. 12, line LN1 in FIG. 2 is shown by a dashed line LN23 as an example of a comparison target for change in breathing amount.

[0178] The indicator parameter "P_indicator" increases until the maximum inspiration is reached. In the example of Fig. 12, the indicator parameter "P_indicator" increases taking into account the amount of inspiration increase until time t21 when the maximum inspiration amount "B_max" is reached. The amount of inspiration increase IL21 in Fig. 12 indicates the amount of inspiration increase at a certain point in time until the maximum inspiration is reached.

[0179] For example, when the inhalation increase amount "B_increase" becomes larger than the increase amount threshold "B_increase_thres" until the maximum inhalation, the information processing device 100 calculates an indicator parameter "P_indicator" indicating the amount of operation using the current inhalation amount "B_current", the inhalation increase amount "B_increase", and the following formula (3). For example, when the inhalation increase amount "B_increase" becomes larger than the increase amount threshold "B_increase_thres" while the user is inhaling, the information processing device 100 calculates an indicator parameter "P_indicator" indicating the amount of operation using formula (3).

[0180]

number

[0181] In the above formula (3), "a", "b", and "c" indicate predetermined constants. The values ​​of "a", "b", and "c" in formula (3) may be different from the values ​​of "a", "b", and "c" in formula (1) and formula (2). In this way, the information processing device 100 increases the operation amount in proportion to the intake amount and the intake increase amount during sudden intake. The information processing system 1 increases the operation amount in proportion to the intake amount and the intake increase amount during sudden intake. Thereby, the information processing system 1 enlarges the indicator in proportion to the intake amount and the intake increase amount during sudden intake. In this way, the information processing system 1 increases the operation amount taking into account the intake increase amount during sudden intake and enlarges the indicator, thereby reducing the burden of increasing the operation amount and enlarging the indicator.

[0182] The above formula (3) is an example of calculation of the indicator parameter "P_indicator" when the intake increase amount "B_increase" becomes larger than the increase amount threshold "B_increase_thres" until the maximum intake, and various formulas may be used to calculate the indicator parameter "P_indicator". For example, when the intake increase amount "B_increase" becomes equal to or smaller than the increase amount threshold "B_increase_thres", a different calculation formula (formula (1)) is used.

[0183] In addition, which of the following formulas includes the case where the intake increase amount "B_increase" is equal to the threshold value "B_increase_thres" may be set as appropriate. When the intake increase amount "B_increase" is equal to or greater than the threshold value "B_increase_thres", the information processing device 100 may use the above formula (3), and when the intake increase amount "B_increase" is less than the threshold value "B_increase_thres", the information processing device 100 may use the following formula (1).

[0184] Moreover, the indicator parameter "P_indicator" changes at a constant rate after the maximum intake. In the example of Fig. 12, the indicator parameter "P_indicator" decreases at a constant rate over time until the time t21 when the maximum intake amount "B_max" is reached. The intake decrease amount DL21 in Fig. 12 indicates the intake decrease amount (exhaust increase amount) at a certain point in time after the maximum intake amount is reached.

[0185] For example, after the maximum intake, the information processing device 100 calculates an indicator parameter "P_indicator" indicating the amount of operation using the maximum intake amount "B_max", a time variable "Time", and the following formula (4). For example, while the user is performing exhaust after the intake increase amount "B_increase" becomes larger than the increase amount threshold "B_increase_thres", the information processing device 100 calculates the indicator parameter "P_indicator" indicating the amount of operation using formula (4).

[0186]

number

[0187] "c" in the above formula (4) indicates a predetermined constant. The value of "c" in formula (4) may be different from the value of "c" in formula (2) or formula (3). The value of "c" in formula (4) may be the same as the value of "c" in formula (2). In this way, the information processing device 100 reduces the manipulated variable at a constant rate during exhaust and increases the manipulated variable in proportion to the intake volume during intake. The information processing system 1 reduces the indicator size at a constant rate during exhaust.

[0188] The above formula (4) is an example of calculation of the indicator parameter "P_indicator" when the exhaust increase amount "B_decrease" after the maximum intake is greater than the exhaust amount threshold "B_decrease_thres", and various formulas may be used to calculate the indicator parameter "P_indicator". For example, when the exhaust increase amount "B_decrease" becomes equal to or less than the exhaust amount threshold "B_decrease_thres", a different calculation formula (the following formula (6)) is used, which will be described in detail later.

[0189] In addition, which of the following includes the case where the exhaust gas increase amount "B_decrease" is equal to the threshold value "B_decrease_thres" may be appropriately set. When the exhaust gas increase amount "B_decrease" is equal to or greater than the exhaust gas increase amount threshold value "B_decrease_thres", the information processing device 100 may use the above formula (4), and when the exhaust gas increase amount "B_decrease" is less than the exhaust gas increase amount threshold value "B_decrease_thres", the information processing device 100 may use the following formula (6).

[0190] The information processing device 100 determines the amount of operation related to the user's operation by reflecting the user's sudden inhalation by calculating the amount of operation (indicator parameter) through the above-mentioned process. In this way, the information processing device 100 can appropriately calculate the amount of operation even during sudden inhalation by determining the amount of operation using information on the user's breathing.

[0191] [1-6-2. Example of reflecting exhaust operation amount] An example of reflecting the amount of operation of exhaust will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of reflecting the amount of operation of the user's breathing. Note that the description of the same points as in Fig. 12 will be omitted as appropriate.

[0192] Graph GR3 in FIG. 13 is a graph showing the relationship between the user's breathing and the operation amount. Specifically, graph GR3 in FIG. 13 is a graph showing the relationship between the user's inhalation amount and the size of the indicator based on the operation amount (indicator size). Graph GR3 in FIG. 13 is a graph in which the vertical axis shows the inhalation amount or the size of the indicator, and the horizontal axis shows time. In addition, in graph GR3 in FIG. 13, a line showing the breathing amount (line LN31) is shown by a solid line, and a line showing the indicator size (line LN32) is shown by a dashed line. In addition, in graph GR3 in FIG. 13, line LN1 in FIG. 2 is shown by a dashed line LN33 as an example of a comparison target for change in breathing amount.

[0193] The indicator parameter "P_indicator" increases until the maximum inspiration is reached. In the example of Fig. 13, the indicator parameter "P_indicator" increases according to the amount of inspiration until time t31 when the maximum amount of inspiration "B_max" is reached.

[0194] For example, when abrupt inhalation is performed until the maximum inhalation, the information processing device 100 calculates the indicator parameter "P_indicator" indicating the amount of operation using the following formula (5). For example, when the inhalation increase amount "B_increase" becomes larger than the increase amount threshold "B_increase_thres" while the user is inhaling, the information processing device 100 calculates the indicator parameter "P_indicator" indicating the amount of operation using formula (5).

[0195]

number

[0196] Since formula (5) is similar to formula (3), the explanation is omitted. Furthermore, when normal intake is performed until maximum intake, the information processing device 100 calculates the indicator parameter "P_indicator" indicating the manipulated variable using formula (1). Note that Fig. 13 is intended to mainly explain the determination of the manipulated variable during exhaust and the change in indicator size, and the process during intake is not limited to the above.

[0197] Also, the indicator parameter "P_indicator" decreases after the maximum intake. In the example of Fig. 13, the indicator parameter "P_indicator" decreases in proportion to the exhaust volume for a period after time t31 when the maximum intake volume "B_max" is reached. The intake volume decrease DL31 in Fig. 13 indicates the intake volume decrease (exhaust volume increase) at a certain point after the maximum intake volume is reached.

[0198] For example, when the exhaust increase amount "B_decrease" after the maximum intake is less than the exhaust amount threshold "B_decrease_thres", the information processing device 100 calculates an indicator parameter "P_indicator" indicating the operation amount using the maximum intake amount "B_max", the exhaust increase amount "B_decrease", and the following formula (6). For example, when the exhaust increase amount "B_decrease" is less than the exhaust amount threshold "B_decrease_thres" while the user is performing exhaust, the information processing device 100 calculates the indicator parameter "P_indicator" indicating the operation amount using formula (6).

[0199]

number

[0200] "b" and "c" in the above formula (6) indicate predetermined constants. The values ​​of "b" and "c" in formula (6) may be different from the values ​​of "b" and "c" in formula (1) to formula (5). In this way, the information processing device 100 reduces the operation amount in proportion to the exhaust volume when the exhaust is larger than normal. The information processing system 1 reduces the operation amount in proportion to the exhaust volume when the exhaust is larger than normal. This allows the information processing system 1 to accelerate the reduction of the operation amount with a larger exhaust increase amount (intake decrease amount) than normal. The information processing system 1 can accelerate the reduction of the indicator with a larger exhaust increase amount (reduction amount) than normal. In this way, the information processing system 1 reduces the operation amount and reduces the indicator by taking into account the exhaust increase amount (intake decrease amount) when the exhaust is larger than normal, thereby reducing the burden of reducing the operation amount and reducing the indicator. This allows the information processing system 1 to correct the overshoot in a short time.

[0201] The above formula (6) is an example of calculation of the indicator parameter "P_indicator" when the exhaust increase amount "B_decrease" after the maximum intake is less than the exhaust amount threshold "B_decrease_thres", and various formulas may be used to calculate the indicator parameter "P_indicator". For example, when the exhaust increase amount "B_decrease" is equal to or greater than the exhaust amount threshold "B_decrease_thres", a different calculation formula (formula (2) or formula (4)) is used.

[0202] In addition, which of the following includes the case where the exhaust gas increase amount "B_decrease" is equal to the threshold value "B_decrease_thres" may be appropriately set. When the exhaust gas increase amount "B_decrease" is equal to or less than the exhaust gas increase amount threshold value "B_decrease_thres", the information processing device 100 may use the above formula (6), and when the exhaust gas increase amount "B_decrease" becomes larger than the exhaust gas increase amount threshold value "B_decrease_thres", the information processing device 100 may use the formula (2) or formula (4).

[0203] The information processing device 100 determines the amount of operation related to the user's operation by reflecting the user's greater than normal exhalation by calculating the amount of operation (indicator parameter) through the above-mentioned process. In this way, the information processing device 100 can appropriately calculate the amount of operation even when the user's exhalation is greater than normal by determining the amount of operation using information on the user's breathing.

[0204] [1-6-3. Example of not reflecting exhaust during intake] For example, when attempting to greatly increase or decrease an operation amount or an indicator, one intake or exhaust may not be sufficient, and when intake or exhaust is to be performed again, intake or exhaust is always performed before that. Therefore, in the information processing system 1, these intakes and exhausts may be ignored so as not to affect the operation amount or indicator. For example, the information processing system 1 may ignore exhaust after a sudden intake. For example, the information processing system 1 may ignore intake after a sudden exhaust. These points will be described below.

[0205] First, an example in which the exhaust during the inhalation is not reflected will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example in which the user's breathing is reflected in the amount of operation. Note that the description of the same points as Fig. 12 and Fig. 13 will be omitted as appropriate.

[0206] Graph GR4 in Fig. 14 is a graph showing the relationship between the user's breathing and the amount of operation. Specifically, graph GR4 in Fig. 14 is a graph showing the relationship between the user's inhalation amount and the size of the indicator based on the amount of operation (indicator size). In graph GR4 in Fig. 14, the vertical axis shows the amount of inhalation or the size of the indicator, and the horizontal axis shows time. In addition, in graph GR4 in Fig. 14, the line showing the breathing amount (line LN41) is shown by a solid line, and the line showing the indicator size (line LN42) is shown by a dashed dotted line.

[0207] The example of Fig. 14 shows a case where the user suddenly inhales until time t41, and then resumes inhalation at time t42 shortly after time t41. Inhalation increase amount IL41 in Fig. 14 indicates the inhalation increase amount at a certain point in time until time t41. Inhalation increase amount IL42 in Fig. 14 indicates the inhalation increase amount at a certain point in time after time t41 until time t42.

[0208] For example, after an intake increase of equal to or greater than the increase threshold "B_increase_thres" is observed, the information processing device 100 does not use the exhaust for a certain time "T_ignore" (also called the "exhaust ignore period") from the point of switching from intake to exhaust (intake-exhaust conversion point) to determine the manipulated variable. For example, after an intake increase of equal to or greater than the increase threshold "B_increase_thres" is observed, the information processing device 100 does not use the exhaust for a certain time "T_ignore" (exhaust ignore period) from the intake-exhaust conversion point to adjust the indicator size.

[0209] In the example of FIG. 14, the information processing device 100 does not reduce the operation amount according to the user's exhaust during the exhaust ignore period from time t41, which is the intake / exhaust conversion point after an intake increase equal to or greater than the increase amount threshold "B_increase_thres", is observed. For example, the information processing device 100 does not reduce the operation amount according to the user's exhaust during the exhaust ignore period from time t41 to time t42, at which the exhaust ignore period has elapsed. Note that the information processing device 100 is not limited to cases in which the user's exhaust is not used at all during the exhaust ignore period, and may reduce the influence of the user's exhaust during the exhaust ignore period. For example, the information processing device 100 may reduce the decrease in the operation amount due to the user's exhaust during the exhaust ignore period compared to normal times.

[0210] Furthermore, the information processing device 100 uses the intake during a certain period of time "T_ignore" (exhaust ignore period) from the intake-exhaust transition point after an intake increase equal to or greater than the increase threshold "B_increase_thres" is observed, in determining the manipulated variable. In the example of FIG. 14, the information processing device 100 increases the manipulated variable according to the intake during the exhaust ignore period from time t41, which is the intake-exhaust transition point after an intake increase equal to or greater than the increase threshold "B_increase_thres", is observed. As shown in graph GR4 in FIG. 14, the information processing device 100 increases the indicator size according to the user's intake between time t41 and time t42 (exhaust ignore period).

[0211] For example, when trying to greatly increase or decrease the indicator, one intake may not be enough, and when taking in air again, exhaust is always performed before that, but the information processing device 100 ignores this exhaust so as not to affect the indicator. In this way, the information processing device 100 can appropriately calculate the operation amount (indicator parameter) and adjust the indicator size by ignoring the exhaust after the sudden intake by the above-mentioned process.

[0212] [1-6-4. Example of not reflecting intake during exhaust] Next, an example in which the intake during the exhaust is not reflected will be described with reference to Fig. 15. Fig. 15 is a diagram showing an example in which the user's breathing is reflected in the amount of operation. Note that the description of the same points as Figs. 12 to 14 will be omitted as appropriate.

[0213] Graph GR5 in Fig. 15 is a graph showing the relationship between the user's breathing and the amount of operation. Specifically, graph GR5 in Fig. 15 is a graph showing the relationship between the user's inhalation amount and the size of the indicator based on the amount of operation (indicator size). Graph GR5 in Fig. 15 is a graph in which the vertical axis indicates the amount of inhalation or the size of the indicator, and the horizontal axis indicates time. In addition, in graph GR5 in Fig. 15, a line indicating the amount of breathing (line LN51) is shown by a solid line, and a line indicating the indicator size (line LN52) is shown by a dashed dotted line.

[0214] The example in Fig. 15 shows a case where the user performs rapid exhaust up to time t51, and then resumes exhaust at time t52 shortly after time t51. The intake reduction amount IL51 in Fig. 15 indicates the intake reduction amount (exhaust increase amount) at a certain point in time up to time t51. The intake reduction amount IL52 in Fig. 15 indicates the intake reduction amount (exhaust increase amount) at a certain point in time after time t51 and up to time t52.

[0215] For example, after an intake decrease amount equal to or less than the exhaust volume threshold "B_decrease_thres" is observed, the information processing device 100 does not use the intake for a certain period of time "T_ignore" (also called the "intake ignore period") from the point of switching from exhaust to intake (exhaust / intake conversion point) to determine the manipulated variable. Note that the value of the certain period of time "T_ignore" in FIG. 15 may be different from the value of the certain period of time "T_ignore" in FIG. 14. For example, after an intake decrease amount equal to or less than the exhaust volume threshold "B_decrease_thres" is observed, the information processing device 100 does not use the intake for a certain period of time "T_ignore" (intake ignore period) from the exhaust / intake conversion point to adjust the indicator size.

[0216] In the example of FIG. 15, the information processing device 100 does not increase the operation amount according to the user's inhalation during the inhalation neglect period from time t51, which is the exhaust-inhalation conversion point after the intake decrease amount equal to or less than the exhaust amount threshold "B_decrease_thres", is observed. For example, the information processing device 100 does not increase the operation amount according to the user's inhalation during the inhalation neglect period from time t51 to time t52, at which the intake neglect period elapses. Note that the information processing device 100 is not limited to the case where the user's inhalation is not utilized at all during the inhalation neglect period, and may reduce the influence of the user's inhalation during the inhalation neglect period. For example, the information processing device 100 may reduce the increase in the operation amount due to the user's inhalation during the inhalation neglect period compared to normal times.

[0217] In addition, the information processing device 100 uses the exhaust during a certain time "T_ignore" (intake ignore period) from the exhaust-intake conversion point after the intake decrease amount equal to or less than the exhaust amount threshold "B_decrease_thres" is observed, in determining the manipulated variable. In the example of FIG. 15, the information processing device 100 reduces the manipulated variable according to the exhaust during the intake ignore period from time t51, which is the exhaust-intake conversion point after the intake decrease amount equal to or less than the exhaust amount threshold "B_decrease_thres", is observed. As shown in the graph GR5 in FIG. 15, the information processing device 100 reduces the indicator size according to the exhaust of the user during the time t51 to time t52 (intake ignore period). For example, when trying to increase or decrease the indicator greatly, one exhaust may not be sufficient, and when exhausting again, intake is always performed before that, but the information processing device 100 ignores this intake so as not to affect the indicator. In this way, the information processing device 100 can appropriately calculate the manipulated variable (indicator parameter) and adjust the indicator size by ignoring the intake after the sudden exhaust through the above-mentioned process.

[0218] [1-7. Prediction processing] The information processing system 1 may perform a prediction process for predicting the user's breathing or an operation amount based on the breathing. This point will be described with reference to Figs. 16 and 17.

[0219] [1-7-1. Example of operation volume prediction] First, an example of prediction of an operation amount will be described with reference to Fig. 16. Fig. 16 is a diagram showing an example of prediction of an operation amount. In the example shown below, the information processing device 100 predicts an operation amount (predicted operation amount) and an indicator size (predicted size) using an intake amount at the time of prediction (current intake amount) and an increase amount.

[0220] Graph GR6 in FIG. 16 is a graph showing the relationship between the user's breathing, the operation amount, and the predicted operation amount. Specifically, graph GR6 in FIG. 16 is a graph showing the relationship between the user's inhalation amount, and the size and predicted size of the indicator based on the operation amount. Graph GR6 in FIG. 16 is a graph in which the vertical axis shows the inhalation amount, the size and predicted size of the indicator, and the horizontal axis shows time. Also, in graph GR6 in FIG. 16, a line (line LN61) showing the breathing amount is shown by a solid line. Also, in graph GR6 in FIG. 16, a line (line LN62) showing the size of the indicator is shown by a dashed line portion, and the predicted size portion is shown by a dotted line. Specifically, up to time t61, the size of the indicator based on the operation amount determined based on the breathing information up to that point in time is shown, and from time t61 to time t62, the predicted size based on the predicted operation amount predicted based on the breathing information up to time t61 is shown.

[0221] Thus, in the example of Fig. 16, the information processing device 100 predicts a future predicted amount of operation and a predicted size based on the inhalation performed by the user up to time t61. The inhalation increase amount IL61 in Fig. 16 indicates the inhalation increase amount at a certain point in time up to time t61. For example, the inhalation increase amount IL61 indicates the inhalation increase amount at time t61. A point CR61 in Fig. 16 indicates the inhalation amount (current inhalation amount) at the current point in time (time t61).

[0222] The information processing device 100 calculates the indicator predicted arrival point "Point_pr" indicating the predicted operation amount by using the current intake amount "B_current", the increase amount "B_increase", and the following formula (7). As a result, the indicator predicted arrival point "Point_pr" becomes a value taking into account the current intake amount "B_current" and the increase amount "B_increase".

[0223]

number

[0224] In the above formula (7), "a" and "b" indicate predetermined constants. The values ​​of "a" and "b" in formula (7) may be different from the values ​​of "a" and "b" in formulas (1) to (6). The above formula (7) is an example of calculation of the indicator predicted arrival point "Point_pr", and various formulas may be used to calculate the indicator predicted arrival point "Point_pr". For example, in the case of a sudden inhalation, the information processing device 100 may predict the indicator predicted arrival point "Point_pr" by taking into account a sudden reflection as in formula (3). In addition, the information processing device 100 may predict the indicator predicted arrival point "Point_pr" based on the past breathing history of the user. The information processing device 100 may predict the indicator predicted arrival point "Point_pr" using a model that outputs the indicator predicted arrival point "Point_pr" when the current inhalation amount "B_current" and the increase amount "B_increase" are input. The information processing device 100 may predict the indicator predicted arrival point "Point_pr" using a model that is trained using a combination of the user's past intake volume and intake increase volume at a certain point in time and the maximum intake volume thereafter.

[0225] The information processing device 100 may also predict the time (predicted required time) required to reach the indicator predicted arrival point "Point_pr" using the current intake volume "B_current" and the increase amount "B_increase". The information processing device 100 may predict the time (predicted required time) required to reach the indicator predicted arrival point "Point_pr" based on the user's past breathing history. The information processing device 100 may predict the time from the current time to reach the maximum intake volume using a model that outputs the predicted required time to reach the maximum intake volume when the current intake volume "B_current" and the increase amount "B_increase" are input. The information processing device 100 may predict the predicted required time from the current time to reach the maximum intake volume using a model that is learned using a combination of a certain time point of the user in the past, the intake volume and intake increase amount at that time, and the time when the maximum intake volume was reached thereafter. Then, the information processing device 100 determines the time obtained by adding the predicted required time to the current time as the time (time) at which the indicator predicted arrival point "Point_pr" is reached. In the example of FIG. 16, the information processing device 100 predicts the time at which the indicator predicted arrival point "Point_pr" is reached as time t62. Note that the above is only an example, and the information processing device 100 may predict the time at which the indicator predicted arrival point "Point_pr" is reached by appropriately using various information. For example, the information processing device 100 may predict the time required to reach the indicator predicted arrival point "Point_pr" by using the current intake amount "B_current", the increase amount "B_increase", and time information indicating the current time.

[0226] 16, the information processing device 100 predicts that the operation amount (indicator parameter) will increase to the value of the indicator predicted arrival point "Point_pr" at time t62. Point PR61 in FIG. 16 indicates the predicted operation amount or predicted size predicted to be reached at time t62.

[0227] This allows the information processing system 1 to predict the point at which maximum inspiration will occur from the current inspiration amount and the increased inspiration amount.

[0228] [1-7-2. Example of predicted operation amount display] Furthermore, the information processing system 1 may display predicted information. This point will be described with reference to Fig. 17. Fig. 17 is a diagram showing an example of a display of a predicted operation amount. For example, Fig. 17 shows an example of a UI display based on a prior prediction of an intake amount.

[0229] 17 is a figure in which three circles of different sizes are arranged with indicator ID11 at the center. The size of indicator ID11 changes according to the user's breathing. Also, a circular prediction line PP11 shown by a dotted line arranged to be superimposed on mark MK11 indicates a predicted size. Mark MK11, indicator ID11, and prediction line PP11 are displayed on the terminal device 10.

[0230] For example, indicator ID11 is an indicator that responds according to the actual breathing volume. For example, prediction line PP11 is an area showing information on a predicted destination point based on the current inhalation volume. This allows the information processing system 1 to know the point that the user can reach with the current inhalation, and can be used as a guide for deciding whether to inhale again if necessary.

[0231] In this way, the information processing system 1 can indicate in advance whether the user needs to take an additional breath by displaying the predicted maximum breath point as a GUI, thereby improving the usability of the information processing system 1.

[0232] [1-8. Other application examples] The information processing system 1 is not limited to the above and may be used for operating various objects. This point will be described with reference to FIGS.

[0233] [1-8-1. Selector indicator operation] First, an example of the use of the manipulated variable will be described. For example, the determined manipulated variable may be used to change (move) the position of a selector indicator. This will be described with reference to FIG. 18. FIG. 18 is a diagram showing an example of the use of the manipulated variable. FIG. 18 shows an example in which the reflecting of the manipulated variable (parameter) according to the intake volume is applied to a selector (indicator) for selecting a GUI. In this way, the manipulated variable may be used to switch GUI operation targets. Note that the description of the same points in FIG. 18 as those in FIG. 1 will be omitted as appropriate.

[0234] 18, the user U1 first utters WUW (step S21). For example, the user U1 utters a speech for activating the speech recognition of the information processing system 1.

[0235] In the example of FIG. 18, the terminal device 10 starts voice recognition in response to an utterance of WUW by the user U1. The terminal device 10 performs a predetermined voice output process when the voice recognition is started. In the example of FIG. 18, as shown in the terminal device 10-1, a voice output of "How can I help?" is output. Thus, in FIG. 18, depending on the change in the display of the terminal device 10, the terminal device 10 may be distinguished and described as terminal devices 10-1, 10-2, and 10-3. Note that the terminal devices 10-1, 10-2, and 10-3 are the same terminal device 10, and when describing without distinction, they will be described as terminal device 10.

[0236] A plurality of objects are arranged in a predetermined direction on the display unit 16 of the terminal device 10. In the example of FIG. 18, as shown in the terminal device 10-1, objects OB1 to OB3 are arranged in a vertical direction on the display unit 16 of the terminal device 10. For example, the objects OB1 to OB3 may be a plurality of input areas. Each of the objects OB1 to OB3 may be an input area for an email destination, a title, and a text. The object OB1 may be an input area for an email destination. The object OB2 may be an input area for an email title. The object OB3 may be an input area for an email text.

[0237] 18, the selector indicator ID21, which is an indicator (inhalation volume indicator) corresponding to the inhalation volume of the user U1, is not displayed on the display unit 16 of the terminal device 10 as shown in the terminal device 10-1. The selector indicator ID21 is a display object whose position changes according to the breathing of the user U1. In the terminal device 10, an object corresponding to the position of the selector indicator ID21 is selected according to the position of the selector indicator ID21. Note that the selector indicator ID21 may be displayed at an initial position in the terminal device 10-1.

[0238] The user U1 inhales before speaking (step S22). For example, the user U1 breathes before speaking, and the information processing device 100 acquires breathing information indicating the breathing of the user U1. For example, the information processing device 100 acquires breathing information indicating the breathing of the user U1 from the terminal device 10 used by the user U1. The information processing device 100 performs a determination process to determine an amount of operation related to an operation by the breathing of the user U1 using the acquired breathing information. This is the same as in FIG. 1, and therefore a description thereof will be omitted.

[0239] The information processing system 1 changes the position of the selector indicator ID21 displayed on the terminal device 10 by using the operation amount (indicator parameter) determined by the information processing device 100 (step S23). For example, the information processing device 100 transmits information on the determined operation amount (indicator parameter) to the terminal device 10. The terminal device 10, which receives the information on the operation amount (indicator parameter) from the information processing device 100, changes the position of the displayed selector indicator ID21.

[0240] In the example of FIG. 18, the terminal device 10 changes the position of the selector indicator ID21 upward as shown in the terminal device 10-2. That is, the terminal device 10 moves the selector indicator ID21 upward. The terminal device 10 changes the display by changing the position of the selector indicator ID21 upward from the initial position using the operation amount (indicator parameter) received from the information processing device 100. As a result, the terminal device 10 selects the object OB2 corresponding to the position of the selector indicator ID21. For example, the terminal device 10 selects the object OB2, which is an input area for the subject of an email, based on the position of the selector indicator ID21. In this way, the information processing system 1 changes the selector indicator according to the amount of intake air, and changes the input area (object) to be the target. The information processing system 1 can change the display or determine the target to be selected using the user's breathing, thereby switching the GUI operation target.

[0241] The information processing device 100 may transmit the content (corresponding to the display screen of the terminal device 10-2) in which the position of the selector indicator ID21 has been changed to the terminal device 10, and the terminal device 10 may display the received content.

[0242] Then, the user U1 speaks (step S24). For example, the user U1 performs voice input of information to be input to the object OB2, which is the input area selected based on the selector indicator ID 21, to the terminal device 10. In the example of FIG. 18, the user U1 speaks "Hello" to perform voice input of information to the terminal device 10.

[0243] Then, the information processing system 1 performs a process of adding the spoken input by the user U1 to the object OB2, which is the input area selected based on the selector indicator ID21 (step S25). The terminal device 10 adds the character string "Hello" to the subject of the email corresponding to the object OB2. As shown in the terminal device 10-3, the terminal device 10 displays the character string "Hello" in the object OB2, which is the input area. This enables the information processing system 1 to input information to a target selected according to the user's inhalation volume.

[0244] In this way, the information processing system 1 determines the amount of operation using the user's breathing and uses the amount of operation in processing corresponding to the user's operation, thereby eliminating the need for the user to input the amount of operation by voice, etc., thereby reducing the operational burden on the user.

[0245] [1-8-2. Example of turning indicators on and off] Also, the indicator may be hidden. For example, a selector indicator as shown in FIG. 18 may be switchable between display and non-display depending on the breathing volume. For example, in the voice recognition process, commands unrelated to GUI operations are also accepted, so that an area for displaying and non-displaying the indicator may be set, such as not displaying the indicator unless the inhalation volume is above a certain level. This point will be described with reference to FIG. 19. FIG. 19 is a diagram showing an example of switching between displaying and non-displaying the indicator. Note that the same points as those in FIG. 2 and FIG. 12 to FIG. 16 will not be described as appropriate.

[0246] Graph GR7 in Fig. 19 is a graph showing the relationship between the user's breathing and the amount of operation. Specifically, graph GR7 in Fig. 19 is a graph showing the relationship between the user's inhalation amount and the size of the indicator based on the amount of operation (indicator size). In graph GR7 in Fig. 19, the vertical axis shows the amount of inhalation or the size of the indicator, and the horizontal axis shows time. In graph GR7 in Fig. 19, a line showing the amount of breathing (line LN71) is shown by a solid line, and a line showing the indicator size (line LN72) is shown by a dashed dotted line.

[0247] The threshold EV71 in the graph GR7 in Fig. 19 is a value indicating the minimum inhalation volume for indicator display. For example, the information processing system 1 displays the indicator when the respiratory volume is equal to or greater than the threshold EV71, and hides the indicator when the respiratory volume is less than the threshold EV71. The information processing device 100 compares the current respiratory volume with the threshold EV71, and determines whether to display or hide the indicator based on the comparison result. When the current respiratory volume is equal to or greater than the threshold EV71, the information processing device 100 determines to display the indicator. When the current respiratory volume is less than the threshold EV71, the information processing device 100 determines to hide the indicator.

[0248] 19, the information processing device 100 determines to hide the indicator because the current respiratory volume is less than the threshold EV71 until time t71. The information processing system 1 hides the indicator until time t71. For example, the terminal device 10 hides the indicator until time t71.

[0249] 19, the information processing device 100 determines to display the indicator because the current respiratory volume is equal to or greater than the threshold EV71 from time t71 to time t72. The information processing system 1 displays the indicator from time t71 to time t72. For example, the terminal device 10 displays the indicator from time t71 to time t72. The terminal device 10 switches the indicator from hidden to displayed at time t71.

[0250] 19, the information processing device 100 determines to hide the indicator after time t72 because the current respiratory volume is less than the threshold EV71. The information processing system 1 hides the indicator after time t72. For example, the terminal device 10 hides the indicator after time t72. The terminal device 10 switches the indicator from visible to hidden at time t72.

[0251] For example, when the indicator is displayed, the information processing system 1 processes the user's voice utterance as an input or command for the selected GUI. Also, when the indicator is not displayed, the information processing system 1 processes the user's voice utterance as a general command unrelated to the GUI.

[0252] In this way, the information processing system 1 can switch processing using the user's voice utterance depending on whether the indicator is displayed or not, thereby enabling appropriate processing depending on whether the indicator is displayed or not.

[0253] [1-8-3. Example of switching GUI operation target] The amount of operation may also be used to switch the GUI operation target. This will be described with reference to Fig. 20. Fig. 20 is a diagram showing an example of switching the GUI operation target.

[0254] Fig. 20 shows an example of GUI operation target switching. In Fig. 20, content CT1 may be distinguished and described as contents CT1-1, CT1-2, CT1-3, and CT1-4 depending on the change in display. Note that contents CT1-1, CT1-2, CT1-3, and CT1-4 are the same content CT1, and will be described as content CT1 when no distinction is made. For example, content CT1 is displayed on the display unit 16 of the terminal device 10.

[0255] The content CT1 includes a mark MK31 and elements EL1, EL2, and EL3. The mark MK31 is a figure in which a plurality of circles of different sizes are arranged with the indicator ID31 at the center. The indicator ID31 changes in size according to the user's breathing. The indicator ID31 is a display object whose size changes according to the breathing of the user U1. That is, the indicator ID31 is a display object whose size indicates an operation amount determined based on the user's breathing. For example, the indicator ID31-1 shown in the content CT1-1 indicates the initial state of the indicator ID31. Thus, in FIG. 1, the indicator ID31 may be distinguished and described as indicator ID31-1, ID31-2, ID31-3, and ID31-4 according to the change in the display of the indicator ID31. Note that the indicators ID31-1, ID31-2, ID31-3, and ID31-4 are described as indicator ID31 when they are not distinguished from each other.

[0256] Element EL1 is an input area corresponding to the email destination (To). Element EL2 is an input area corresponding to the email subject (Title). Element EL3 is an input area corresponding to the email body (Message). As shown in FIG. 20, the areas occupied by elements EL1, EL2, and EL3 become larger in this order, and it is expected that the amount of information inputted will increase in the order of elements EL1, EL2, and EL3.

[0257] In the example of FIG. 20, the information processing system 1 changes the display of the indicator ID31 in response to the user's breathing, similar to FIG. 1. Also, in the example of FIG. 20, the information processing system 1 changes the input area (element) to be selected in response to the user's breathing, similar to FIG. 18. Content CT1-1 in FIG. 20 indicates a non-selected state in which no input area (element) is selected. In this case, the information processing system 1 recognizes the user's utterance as a general-purpose command. Also, contents CT1-1 to CT1-4 indicate changes in the indicator ID31 and the selected input area (element) in response to changes in the amount of inhalation before the user speaks.

[0258] For example, the information processing system 1 changes the display mode of the content CT1 from content CT1-1 to content CT1-2 in response to an increase in the user's breathing rate. The information processing system 1 changes the display size of the indicator ID31 from indicator ID31-1 to indicator ID31-2 in response to an increase in the user's breathing rate. Furthermore, the information processing system 1 selects element EL1, which is an input area corresponding to the email destination (To), in response to an increase in the user's breathing rate. In this way, the information processing system 1 changes the size of indicator ID31 and selects an input area (element) in response to the user's inhalation rate. For example, when the user speaks in the state of content CT1-2, input information by the user's speech is added to element EL1, which is an input area corresponding to the email destination (To).

[0259] Also, for example, the information processing system 1 changes the display mode of the content CT1 from content CT1-2 to content CT1-3 in response to an increase in the user's breathing rate. The information processing system 1 changes the display size of the indicator ID31 from indicator ID31-2 to indicator ID31-3 in response to an increase in the user's breathing rate. Also, the information processing system 1 changes the selected input area (element) from an input area (element EL1) corresponding to the email destination (To) to an input area (element EL2) corresponding to the email subject (Title) in response to an increase in the user's breathing rate. In this way, the information processing system 1 changes the size of the indicator ID31 in response to the user's inhalation rate and changes the selected input area (element). For example, when the user speaks in the state of the content CT1-3, the input information by the user's speech is added to element EL2, which is an input area corresponding to the email subject (Title).

[0260] Also, for example, the information processing system 1 changes the display mode of the content CT1 from content CT1-3 to content CT1-4 in response to an increase in the user's breathing rate. The information processing system 1 changes the display size of the indicator ID31 from indicator ID31-3 to indicator ID31-4 in response to an increase in the user's breathing rate. Also, the information processing system 1 changes the selected input area (element) from an input area (element EL2) corresponding to the subject (Title) of the email to an input area (element EL3) corresponding to the body (Message) of the email in response to an increase in the user's breathing rate. In this way, the information processing system 1 changes the size of the indicator ID31 in response to the user's inhalation rate and changes the selected input area (element). For example, when the user speaks in the state of the content CT1-4, the input information by the user's speech is added to element EL3, which is an input area corresponding to the body (Message) of the email.

[0261] In this way, the information processing system 1 displays an indicator whose size changes according to the user's breathing rate, and a plurality of elements (input areas) that are selected according to the user's breathing rate. This allows the user to intuitively grasp the relationship between changes in the user's breathing and changes in the selection of elements, making it easier for the user to select elements by breathing. Therefore, the information processing system 1 can save the user the trouble of inputting by voice or the like, thereby reducing the user's operational burden and improving usability.

[0262] [2. Other embodiments] The processing according to each of the above-described embodiments may be implemented in various different forms (variations) other than the above-described embodiments and variations.

[0263] [2-1. Variation (Example of decision processing etc. on the client side)] In the embodiment, as an example of the system configuration, the information processing device 100 determines and predicts the amount of operation, but the terminal device 10 may determine and predict the amount of operation. That is, the terminal device 10, which is a client-side device, may be an information processing device that determines and predicts the amount of operation described above. In this way, the system configuration of the information processing system 1 is not limited to a configuration in which the information processing device 100, which is a server-side device, determines and predicts the amount of operation, but may be a configuration in which the terminal device 10, which is a client-side device, determines and predicts the amount of operation described above.

[0264] When the terminal device 10 is an information processing device that determines and predicts the above-mentioned operation amount, in the information processing system 1, the operation amount is determined and predicted on the client side (terminal device 10). Then, the server side (information processing device 100) acquires information on the operation amount and the predicted operation amount from the terminal device 10 and performs various processes. In this case, the terminal device 10 may have a prediction unit that realizes a function similar to the prediction unit 132 described above, or a determination unit that realizes a function similar to the determination unit 133. In this case, the information processing device 100 may not have the prediction unit 132 or the determination unit 133.

[0265] Furthermore, the information processing system 1 may have a system configuration in which the client side (terminal device 10) determines the amount of operation, and the server side (information processing device 100) predicts the amount of operation. In this case, the terminal device 10, which is a client-side device, may be an information processing device that performs the above-mentioned process of determining the amount of operation, and the information processing device 100, which is a server-side device, may be an information processing device that performs the above-mentioned process of predicting and determining the amount of operation. In this case, a determination unit of the terminal device 10 performs the determination process, and a prediction unit 132 of the information processing device 100 performs the prediction process.

[0266] Note that the above is just an example, and each process may be performed by any device in the information processing system 1. In this way, the information processing system 1 may have a system configuration in which each process is performed by either the client side device (terminal device 10) or the server side device (information processing device 100).

[0267] [2-2. Configuration example of information processing device according to modified example] In addition, when the amount of operation is determined on the client side (terminal device) as described above, the information processing device that determines the amount of operation functions as the terminal device. In this case, the terminal device that is the client side device has a function of determining the amount of operation and a function of performing display according to the amount of operation. This point will be described with reference to FIG. 21. FIG. 21 is a diagram showing an example of the configuration of an information processing device according to a modified example of the present disclosure. In this case, the information processing system 1 may include an information processing device 100A that is the client side terminal device, and a server device that communicates various information with the information processing device 100A.

[0268] The information processing device 100A used as a client-side device (terminal device) has a function of determining an amount of operation and a function of performing display according to the amount of operation. For example, the information processing device 100A acquires various information from a server device providing a voice recognition service, and executes various processes using the acquired information. For example, the information processing device 100A transmits user's utterance information to the server device, acquires information indicating the voice recognition result from the server device, and executes various processes using the acquired information. In the following description of the information processing device 100A, the same reference numerals are used to designate the same points as those of the information processing device 100 shown in FIG. 6 and the terminal device 10 shown in FIG. 8, and the description will be omitted as appropriate.

[0269] As shown in FIG. 21, the information processing device 100A includes a communication unit 110, an input unit 12, an output unit 13, a storage unit 120, a control unit 130, a display unit 16, and a sensor unit 17.

[0270] The communication unit 110 transmits and receives information to and from the server device. The input unit 12 receives various operations from a user. The output unit 13 outputs various information. The user information storage unit 122 stores various information of a user who uses the information processing device 100A.

[0271] The acquisition unit 131 of the information processing device 100A has the same functions as those described in Fig. 6 and the receiving unit 151 and the accepting unit 153. The acquisition unit 131 receives various information. The acquisition unit 131 receives various information from an external information processing device. The acquisition unit 131 receives various information from another information processing device such as a server device.

[0272] The acquisition unit 131 acquires information detected by the sensor unit 17. The acquisition unit 131 acquires information detected by the respiration sensor 171 of the sensor unit 17. The acquisition unit 131 receives information instructing the start of voice recognition from the server device. The acquisition unit 131 receives information instructing the start of a voice recognition application from the server device.

[0273] The acquisition unit 131 receives execution instructions for various functions from the server device. For example, the acquisition unit 131 receives information designating a function from the server device as an execution instruction for the function. The acquisition unit 131 receives content. The acquisition unit 131 receives content to be displayed from the server device.

[0274] The acquisition unit 131 accepts various types of information. The acquisition unit 131 accepts input by the user via the input unit 12. The acquisition unit 131 accepts speech by the user as input. The acquisition unit 131 accepts operations by the user. The acquisition unit 131 accepts operations by the user on information displayed by the display unit 16. The acquisition unit 131 accepts character input by the user.

[0275] 6 and the same functions as the execution unit 152. The execution unit 134 executes various processes based on information from the server device. The execution unit 134 executes various processes in response to instructions from the server device.

[0276] The transmission unit 135 of the information processing device 100A has the function described in Fig. 6 and the function of the transmission unit 154. The transmission unit 135 transmits various information based on information from other information processing devices such as a server device. The transmission unit 135 transmits the amount of operation determined by the determination unit 133 to the server device. The transmission unit 135 transmits the amount of operation predicted by the prediction unit 132 (predicted amount of operation) to the server device.

[0277] The transmitting unit 135 transmits respiratory information indicating the user's breathing to the server device. The transmitting unit 135 transmits respiratory information including the user's inhalation volume to the server device. The transmitting unit 135 transmits respiratory information including the amount of change in the user's breathing to the server device. The transmitting unit 135 transmits speech information indicating an utterance by the user after the user's breathing to the server device. The transmitting unit 135 transmits speech information indicating an utterance by the user before the user's breathing to the server device.

[0278] The transmission unit 135 transmits sensor information detected by the sensor unit 17 to the server device. The transmission unit 135 transmits respiratory information of the user U1 detected by the respiratory sensor 171 of the sensor unit 17 to the server device. The transmission unit 135 transmits input information input by the user to the server device. The transmission unit 135 transmits input information input by voice by the user to the server device. The transmission unit 135 transmits input information input by a user operation to the server device.

[0279] The display unit 16 has the functions described in Fig. 8. The display unit 16 displays information received from the server device. The display unit 16 displays information indicating the amount of operation determined by the determination unit 133. The display unit 16 displays an indicator corresponding to the amount of operation.

[0280] [2-3.Other] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0281] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0282] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory.

[0283] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0284] [3. Effects of this disclosure] As described above, the information processing device according to the present disclosure (information processing devices 100 and 100A in the embodiments) includes an acquisition unit (acquisition unit 131 in the embodiments) and a determination unit (decision unit 133 in the embodiments). The acquisition unit acquires respiratory information indicating the user's breathing. The determination unit determines an operation amount related to an operation by the user based on the user's breathing indicated by the respiratory information acquired by the acquisition unit.

[0285] In this way, the information processing device according to the present disclosure can determine the amount of operation related to the user's operation based on the user's breathing, thereby eliminating the need for the user to input the amount of operation, thereby reducing the operational burden on the user.

[0286] Furthermore, the determination unit determines the amount of operation depending on the amount of breathing of the user. In this manner, the information processing device determines the amount of operation depending on the amount of breathing of the user, thereby reducing the operation load on the user.

[0287] The acquisition unit acquires respiration information including the user's inhalation volume. The determination unit determines the amount of operation based on the amount of inhalation. In this manner, the information processing device can accurately determine the amount of operation by using the inhalation volume of the user's breath.

[0288] The acquisition unit acquires respiration information including a change in the user's respiration. The determination unit determines an amount of operation based on the change. In this manner, the information processing device can accurately predict whether the user is speaking by using the change in the user's respiration.

[0289] Moreover, the determination unit increases the amount of operation in response to either the user's inhalation or exhalation. In this way, the information processing device increases the amount of operation in conjunction with the user's breathing, thereby eliminating the need for the user to input the amount of operation, and thus reducing the operation load on the user.

[0290] In addition, the determination unit reduces the amount of operation in accordance with the other of the user's intake and exhaust, which is different from the other of the user's intake and exhaust. In this way, the information processing device uses the other of the user's intake and exhaust, which is different from the one used to increase the amount of operation, to decrease the amount of operation, thereby saving the user the trouble of inputting an increase or decrease in the amount of operation, thereby reducing the operational burden on the user.

[0291] Moreover, the determination unit increases the amount of operation in response to the user's inhalation. In this manner, the information processing device increases the amount of operation in response to the user's inhalation, thereby eliminating the need for the user to input the amount of operation, and thus reducing the operation load on the user.

[0292] Moreover, the determination unit increases the amount of operation in response to an increase in the amount of intake air when the user takes a breath. In this way, the information processing device increases the amount of operation in response to an increase in the amount of intake air when the user takes a breath, thereby saving the user the trouble of inputting the amount of operation, and thus reducing the operation load on the user.

[0293] Furthermore, when a comparison result between the increase in the user's inhalation and the threshold satisfies a predetermined condition, the determination unit increases the amount of operation according to the increase in the inhalation. In this way, when a comparison result between the increase in the user's inhalation and the threshold satisfies a predetermined condition, the information processing device increases the amount of operation according to the increase in the inhalation, thereby enabling the amount of operation to be increased more appropriately in conjunction with the user's inhalation.

[0294] In addition, the determination unit reduces the amount of operation at a constant rate while the user is exhausting. In this way, the information processing device reduces the amount of operation at a constant rate while the user is exhausting, and thus the time during which the user is exhausting can be used to reduce the amount of operation, thereby reducing the operational burden on the user.

[0295] In addition, the determination unit reduces the amount of operation in response to the user's exhaust. In this way, the information processing device can reduce the amount of operation in response to the user's exhaust, thereby saving the user the trouble of inputting the amount of operation, and thus reducing the operation load on the user.

[0296] In addition, the determination unit reduces the operation amount in response to a decrease in the intake amount when the user exhausts. In this way, the information processing device reduces the operation amount in response to a decrease in the intake amount when the user exhausts, thereby eliminating the need for the user to input the operation amount, and thus reducing the operation load on the user.

[0297] Furthermore, when a comparison result between the increase in the user's exhaust and the threshold satisfies a predetermined condition, the determination unit reduces the amount of operation according to the increase in the exhaust. In this way, when a comparison result between the increase in the user's exhaust and the threshold satisfies a predetermined condition, the information processing device reduces the amount of operation according to the increase in the exhaust, thereby making it possible to reduce the amount of operation more appropriately in conjunction with the user's exhaust.

[0298] Moreover, the acquisition unit acquires speech information indicating the user's speech after the user's breathing. The determination unit determines a target of the user's operation based on the speech information. In this way, the information processing device can appropriately determine a target to which the operation amount is applied by determining a target of the user's operation using the user's speech after the user's breathing. As a result, the information processing device determines the operation amount by the user's breathing and determines the operation target by the user's speech, so that the user can operate by only speaking the operation target, thereby reducing the operation load on the user.

[0299] Moreover, the acquisition unit acquires speech information indicating the user's speech before the user breathes. The determination unit determines a target of the user's operation based on the speech information. In this way, the information processing device can appropriately determine a target to which the operation amount is applied by determining a target of the user's operation using the user's speech before breathing. As a result, the information processing device determines the operation amount by the user's breathing and determines the operation target by the user's speech, so that the user can operate by only speaking the operation target, thereby reducing the operation load on the user.

[0300] In addition, the determination unit determines the amount of operation corresponding to the parameter of the device. In this way, the information processing device determines the amount of operation corresponding to the parameter of the device, thereby saving the user the trouble of inputting the parameter of the device, and thus reducing the operation load of the user.

[0301] In addition, the determination unit determines an amount of operation corresponding to a change in the position of the target displayed on the screen. In this way, the information processing device determines an amount of operation corresponding to a change in the position of the target displayed on the screen, thereby eliminating the need for the user to perform an operation to change the position of the target, thereby reducing the user's operational burden.

[0302] Moreover, the information processing device according to the present disclosure includes a display unit (display unit 16 in the embodiment). The display unit displays information indicating the amount of operation determined by the determination unit. In this manner, the information processing device displays the information indicating the amount of operation, thereby allowing the user to visually recognize the state of the amount of operation due to breathing, thereby improving usability.

[0303] Moreover, the display unit displays an indicator corresponding to the amount of operation. In this manner, the information processing device displays an indicator corresponding to the amount of operation, thereby enabling the user to intuitively understand the amount of operation, thereby improving usability.

[0304] [4. Hardware configuration] Information devices such as the information processing devices 100, 100A and the terminal device 10 according to the above-described embodiments are realized by a computer 1000 having a configuration as shown in FIG. 22, for example. FIG. 22 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device. The information processing device 100 according to the embodiment will be described below as an example. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

[0305] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.

[0306] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.

[0307] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.

[0308] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0309] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0310] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize functions of the control unit 130, etc. Also, the information processing program according to the present disclosure and data in the storage unit 120 are stored in the HDD 1400. Note that the CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0311] The present technology can also be configured as follows. (1) An acquisition unit that acquires respiration information indicating the respiration of a user; a determination unit that determines an amount of operation related to an operation by the user based on the respiration of the user indicated by the respiration information acquired by the acquisition unit; An information processing device comprising: (2) The determination unit is determining the operation amount in response to an amount related to the breathing of the user; An information processing device as described in (1). (3) The acquisition unit is acquiring the respiration information including an inhalation volume of the user; The determination unit is determining the manipulated variable based on the intake air volume; An information processing device according to (1) or (2). (4) The acquisition unit is Acquiring the respiration information including a change in the respiration of the user; The determination unit is determining the manipulated variable based on the change amount; The information processing device according to any one of (1) to (3). (5) The determination unit is increasing the operation amount in response to either one of intake and exhaust of the user; The information processing device according to any one of (1) to (4). (6) The determination unit is reducing the operation amount in response to the other of the intake and exhaust of the user that is different from the one of the intake and exhaust of the user; (5) An information processing device according to the present invention. (7) The determination unit is increasing the operation amount in response to the inhalation of the user; An information processing device according to (5) or (6). (8) The determination unit is increasing the operation amount in response to an increase in an intake amount during inhalation by the user; An information processing device according to (5) or (7). (9) The determination unit is When a comparison result between the increased amount of the user's inhalation and a threshold value satisfies a predetermined condition, the operation amount is increased according to the increased amount of the inhalation. An information processing device according to (7) or (8). (10) The determination unit is Constantly decreasing the manipulated variable while the user is exhausting. The information processing device according to any one of (7) to (9). (11) The determination unit is Reducing the operation amount in response to the exhaust of the user. The information processing device according to any one of (7) to (9). (12) The determination unit is reducing the operation amount in response to a decrease in the intake amount when the user exhales; An information processing device according to (11). (13) The determination unit is When a comparison result between the increase in the exhaust gas by the user and a threshold value satisfies a predetermined condition, the operation amount is reduced according to the increase in the exhaust gas. The information processing device according to (11) or (12). (14) The acquisition unit is acquiring speech information indicative of an utterance by the user after the breath of the user; The determination unit is determining a target for an operation by the user based on the speech information; The information processing device according to any one of (1) to (13). (15) The acquisition unit is acquiring speech information indicative of an utterance by the user before the breath of the user; The determination unit is determining a target for an operation by the user based on the speech information; The information processing device according to any one of (1) to (13). (16) The determination unit is determining the manipulated variable corresponding to a parameter of the device; The information processing device according to any one of (1) to (15). (17) The determination unit is determining the amount of operation corresponding to a change in position of an object displayed on a screen; The information processing device according to any one of (1) to (15). (18) a display unit that displays information indicating the operation amount determined by the determination unit; The information processing device according to any one of (1) to (17), further comprising: (19) The display unit is displaying an indicator corresponding to the operation amount; An information processing device according to (18). (20) Acquire breathing information indicative of the user's breathing; determining an amount of operation related to an operation by the user based on the respiration of the user indicated by the acquired respiration information; An information processing method for performing processing. [Explanation of symbols]

[0312] 1. Information Processing Systems 100, 100A Information processing device 110 Communications Department 120 Storage section 121 Breathing information storage unit 122 User information storage unit 123 Threshold information storage unit 124 Functional information storage unit 130 Control section 131 Acquisition Department 132 Prediction Department 133 Decision Section 134 Executive Department 135 Transmitter 10 Terminal Equipment 11 Communications Department 12 Input section 13 Output section 14 Storage section 15 Control section 151 Receiving unit 152 Executive Department 153 Reception 154 Transmitter 16 Display 17 Sensor section 171 Respiration Sensor

Claims

1. an acquisition unit that receives sensor information detected by a breathing sensor that detects breathing of a user from a terminal device having the breathing sensor; a determination unit that determines an amount of operation by the user for an operation target selected from a plurality of devices that are candidates for operation targets by the breath of the user, based on the respiration of the user indicated by respiration information that is data indicating at least one of the intake and exhaust of the user per unit time generated using the sensor information received by the acquisition unit; a transmission unit that transmits information indicating the operation amount determined by the determination unit to the terminal device; An information processing device comprising:

2. The determination unit is determining the operation amount in response to an amount related to the breathing of the user; The information processing device according to claim 1 .

3. The acquisition unit is acquiring the respiration information including an inhalation volume of the user; The determination unit is determining the manipulated variable based on the intake air volume; The information processing device according to claim 1 .

4. The acquisition unit is Acquiring the respiration information including a change in the respiration of the user; The determination unit is determining the manipulated variable based on the change amount; The information processing device according to claim 1 .

5. The determination unit is increasing the operation amount in response to either one of intake and exhaust of the user; The information processing device according to claim 1 .

6. The determination unit is reducing the operation amount in response to the other of the intake and exhaust of the user that is different from the one of the intake and exhaust of the user; The information processing device according to claim 5 .

7. The determination unit is increasing the operation amount in response to the inhalation of the user; The information processing device according to claim 5 .

8. The determination unit is increasing the operation amount in response to an increase in an intake amount during inhalation by the user; The information processing device according to claim 5 .

9. The determination unit is When a comparison result between the increased amount of the user's inhalation and a threshold value satisfies a predetermined condition, the operation amount is increased according to the increased amount of the inhalation. The information processing device according to claim 7.

10. The determination unit is Constantly decreasing the manipulated variable while the user is exhausting. The information processing device according to claim 7.

11. The determination unit is Reducing the operation amount in response to the exhaust of the user. The information processing device according to claim 7.

12. The determination unit is reducing the operation amount in response to a decrease in the intake amount when the user exhales; The information processing device according to claim 11.

13. The determination unit is When a comparison result between the increase in the exhaust gas by the user and a threshold value satisfies a predetermined condition, the operation amount is reduced according to the increase in the exhaust gas. The information processing device according to claim 11.

14. The acquisition unit is acquiring speech information indicative of an utterance by the user after the breath of the user; The determination unit is determining a target for an operation to be performed by the user based on the speech information; The information processing device according to claim 1 .

15. The acquisition unit is acquiring speech information indicative of an utterance by the user before the breath of the user; The determination unit is determining a target for an operation by the user based on the speech information; The information processing device according to claim 1 .

16. The determination unit is determining the manipulated variable corresponding to a parameter of the device; The information processing device according to claim 1 .

17. The determination unit is determining the amount of operation corresponding to a change in position of an object displayed on a screen; The information processing device according to claim 1 .

18. a display unit that displays information indicating the operation amount determined by the determination unit; The information processing device according to claim 1 , further comprising:

19. The display unit is displaying an indicator corresponding to the operation amount; The information processing device according to claim 18.

20. receiving sensor information detected by a respiration sensor that detects the respiration of a user from a terminal device having the respiration sensor; determining an operation amount related to an operation by the user on a target of operation selected from a plurality of devices which are candidates for target of operation by the breath of the user, based on the respiration of the user indicated by respiration information which is data indicating at least one of inhalation and exhalation of the user per unit time generated using the received sensor information; transmitting information indicating the determined operation amount to the terminal device; An information processing method for performing a process.

Images