Information Processing Method, Program, and Information Processing Apparatus

Abstract

According to this information processing method, which is executed by a computer: first sound data are acquired from speech of a user for controlling an appliance (S11); a term represented by the acquired first sound data is recognized (S12); first control information for the appliance, corresponding to the recognized term, is determined (S13); the appliance is controlled in accordance with the determined first control information (S14); and if second sound data are acquired from the speech of the user after the point in time at which the first sound data were acquired, a proposal relating to a short cut for controlling the appliance in accordance with the first control information is made by means of speech (S15).

Description

Technical Field

[0001] The present disclosure relates to an information processing method, a program, and an information processing apparatus.

Background Art

[0002] There is a technique for generating a logic that automatically executes control of a device based on control of the device and changes in the situation around the device (see Patent Document 1).

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there is room for improvement in the registration of information for device control.

[0005] Therefore, the present disclosure provides an information processing method and the like for more appropriately executing registration of shortcuts for device control.

Means for Solving the Problems

[0006] The information processing method in the present disclosure is an information processing method executed by a computer, which acquires first sound data from a user's voice for device control, recognizes a phrase indicated by the acquired first sound data, determines first control information of the device corresponding to the recognized phrase, controls the device according to the determined first control information, and when second sound data is acquired from the user's voice after the time when the first sound data is acquired, makes a proposal regarding a shortcut for controlling the device according to the first control information by voice.

[0007] According to the above aspect, the information processing method makes a proposal regarding a shortcut of a voice operation performed by a user when a subsequent voice operation of the user is performed, that is, when the user is present at a place where the voice can be listened to. Therefore, the user can be made to listen to the voice related to the above proposal. In other words, according to this information processing method, it is possible to avoid making the above proposal when the user U cannot listen to the voice related to the proposal. Even if the user U has an intention to accept the proposal, if the proposal is made when the user is not present at a place where the voice can be listened to, information indicating that the user accepts the proposal cannot be obtained, and the information processing and power consumption required for the proposal are wasted. According to the above information processing method, it is possible to avoid the above waste of information processing and power consumption while increasing the probability of obtaining information indicating that the user accepts the proposal. Therefore, according to the above information processing method, the registration of shortcuts for device control can be executed more appropriately.

[0008] Further, the proposal includes a registration proposal for newly registering the shortcut, and when information indicating that the user accepts the registration proposal is obtained, the registration related to the registration proposal may be performed.

[0009] According to the above aspect, the information processing method can avoid making a registration proposal when the user cannot listen to the voice related to the shortcut registration proposal. Therefore, according to the above information processing method, the new registration of shortcuts for device control can be executed more appropriately.

[0010] Also, when the first sound data is acquired in a state where the shortcut is registered, further recognize the phrase indicated by the acquired second sound data, determine second control information of the device corresponding to the recognized phrase, control the device according to the determined second control information, and the proposal includes a modification proposal to modify the registered shortcut to be registered as a shortcut for controlling the device according to the second control information when a condition indicating that the first control information and the second control information are similar is satisfied, and when information indicating that the user approves the modification proposal is obtained, the modification according to the modification proposal may be made.

[0011] According to the above aspect, the information processing method can avoid making a modification proposal when the user cannot listen to the voice related to the modification proposal of the shortcut. Therefore, according to the above information processing method, the modification of the registration of the shortcut for controlling the device can be executed more appropriately.

[0012] Also, when the second sound data is acquired in a state where the shortcut is registered and a predetermined time has elapsed since the shortcut was registered, the proposal includes an invalidation proposal to invalidate the registered shortcut, and when information indicating that the user approves the invalidation proposal is obtained, the invalidation according to the invalidation proposal may be made.

[0013] According to the above aspect, the information processing method can avoid making an invalidation proposal when the user cannot listen to the voice related to the invalidation of the shortcut. Therefore, according to the above information processing method, the invalidation of the registration of the shortcut for controlling the device can be executed more appropriately.

[0014] In addition, the device includes a plurality of devices, the first sound data includes a plurality of first sound data included in the voice collectively uttered by the user for controlling each of the plurality of devices, and the first control information includes a plurality of first control information that is the first control information corresponding to each of the phrases recognized as being indicated by each of the plurality of first sound data. When second sound data is acquired from the user's voice after the time when the plurality of first sound data is acquired, a proposal regarding the shortcut for collectively controlling the plurality of devices according to the plurality of first control information may be made.

[0015] According to the above aspect, the information processing method proposes a shortcut related to an instruction for collectively controlling device control by each voice uttered by the user in a relatively short time. By using the shortcut, the user has the advantage of not having to perform voice operations for each device. In addition, since the information processing method executes a plurality of instructions for a device based on one shortcut phrase uttered by the user, there is an advantage that the information processing and power consumption required for phrase recognition processing can be reduced. Therefore, according to the above information processing method, it is possible to more appropriately execute the registration of shortcuts for controlling a plurality of devices while suppressing information processing and power consumption.

[0016] In addition, the plurality of devices are grouped for each device belonging to the same type, and the proposal may include a message for collectively changing instructions for all devices belonging to the same group.

[0017] According to the above aspect, the information processing method proposes a shortcut for collectively controlling a plurality of devices belonging to the same type (in other words, device category), so the voice required for the proposal can be shortened. Therefore, there is an advantage that the information processing and power consumption for outputting voice can be reduced. Therefore, according to the above information processing method, it is possible to more appropriately execute the registration of shortcuts for controlling a plurality of devices while suppressing information processing and power consumption.

[0018] Further, the proposal may include information indicating the time when the control according to the first control information was performed.

[0019] According to the above aspect, the information processing method proposes a proposal including information indicating the time when a plurality of controls included in the shortcut were performed. Therefore, when the user listens to the above proposal, the user can know at what time the shortcut proposal for the voice operation made by the user is made. Therefore, there is an advantage that it becomes easy for the user to decide whether to accept the proposal. And since it becomes easier to obtain the user's acceptance, the information processing system can more appropriately execute the registration of more shortcuts for controlling the device.

[0020] Further, the proposal may not include the content of the first control information.

[0021] According to the above aspect, the information processing method makes a proposal that does not include the content of a plurality of controls included in the shortcut, so that the voice related to the proposal can be shortened. Therefore, there is an advantage that information processing and power consumption for outputting voice can be reduced. Therefore, according to the above information processing method, while reducing information processing and power consumption, the registration of shortcuts for controlling the device can be more appropriately executed.

[0022] Further, the acquisition of the second sound data is performed by a microphone arranged at a position where the voice uttered by the user reaches, and the microphone and the speaker are provided in the same housing, and the proposal may be made by the voice uttered by the speaker.

[0023] According to the above aspect, the information processing method can make a proposal to the user using a device provided with a microphone and a speaker in the same housing. Therefore, according to the above information processing method, the registration of shortcuts for controlling the device can be more easily executed.

[0024] Also, when making the above proposal, if it is determined that the location where the device that emits the voice related to the proposal is placed or the time when the proposal is made does not satisfy a predetermined condition indicating that it is suitable for the proposal, the proposal may be prohibited.

[0025] According to the above aspect, in a location or time not suitable for a proposal (for example, a location or time where the probability of obtaining consent is assumed to be low), the information processing method does not make a shortcut proposal, so that information processing and power consumption related to the proposal processing can be reduced. Therefore, according to the above information processing method, while reducing information processing and power consumption, the registration of shortcuts for device control can be executed more appropriately.

[0026] Also, the program in the present disclosure is a program for causing a computer to execute the above information processing method.

[0027] According to the above aspect, the same effect as the above information processing method is achieved.

[0028] Also, the information processing apparatus in the present disclosure includes an acquisition unit that acquires first sound data from a user's voice for controlling a device, a recognition unit that recognizes a phrase indicated by the first sound data acquired by the acquisition unit, a determination unit that determines first control information of the device corresponding to the phrase recognized by the recognition unit and controls the device according to the determined first control information, and a proposal unit that makes a proposal regarding a shortcut for controlling the device according to the first control information by voice when second sound data is acquired from a user's voice after the time when the first sound data is acquired.

[0029] According to the above aspect, the same effect as the above information processing method is achieved.

[0030] Note that these general or specific aspects may be implemented in a system, apparatus, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or may be implemented in any combination of a system, apparatus, integrated circuit, computer program, and recording medium.

Advantages of the Invention

[0031] The information processing method of the present disclosure can more appropriately execute the registration of shortcuts for controlling devices.

Brief Description of the Drawings

[0032]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

MODE FOR CARRYING OUT THE INVENTION

[0033] Hereinafter, the embodiments will be described in detail with reference to the drawings as appropriate. However, a more detailed description may be omitted as necessary. For example, a detailed description of well-known matters and a redundant description of substantially the same configuration may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate the understanding of those skilled in the art.

[0034] Note that the inventor(s) provide the accompanying drawings and the following description so that those skilled in the art can fully understand the present disclosure, and do not intend to limit the subject matter described in the claims thereby.

[0035] (Embodiment) Hereinafter, after explaining the background leading to the present invention and the problems to be solved by the present invention in detail, the embodiments will be described.

[0036] In the present embodiment, an information processing method and the like for more appropriately executing the registration of a shortcut for controlling a device will be described.

[0037] FIG. 1 is an explanatory diagram schematically showing the configuration of an information processing system 1 according to the present embodiment.

[0038] The information processing system 1 shown in FIG. 1 is a system for controlling devices D11, etc. in the user U's house.

[0039] As shown in FIG. 1, the information processing system 1 includes a server 10, and smart speakers S1, S2, S3, and S4 (also referred to as S1, etc.). The server 10 and the smart speakers S1, etc. are communicably connected via a network N.

[0040] The network N includes a carrier network of a mobile phone, a telephone line or a telephone line network using an optical fiber, a LAN (including a wired or wireless LAN), etc., and also includes a network in which a plurality of these are connected.

[0041] The server 10 is a computer device including a CPU (Central Processing Unit), a memory, a communication interface, etc. The server 10 is a device that acquires sound data indicating the user's voice from the smart speakers S1, etc. through the network N, and controls the devices D11, etc. through the network N based on the acquired sound data.

[0042] The smart speaker S1 is a speaker device provided in the same housing as a microphone and a speaker. The smart speaker S1 can be realized by information processing performed by a CPU using a memory, a microphone, a speaker, a communication interface, etc. The smart speaker S1 is arranged at a position where the voice uttered by the user reaches. In other words, a smart speaker arranged at a position where the voice uttered by the user reaches functions as the smart speaker S1.

[0043] The smart speaker S1 picks up the voice uttered by the user through the microphone to generate voice data, and transmits the generated voice data to the server 10. The voice uttered by the user includes voices for controlling devices such as the device D11. Also, the smart speaker S1 receives voice data related to the voice to be uttered to the user from the server 10, and outputs the voice related to the received voice data through the speaker for the user to listen to. The smart speaker S1 is arranged in the room R1. Note that instead of the smart speaker S1, an information terminal having the same functions as the smart speaker S1 according to the above description, for example, a smartphone or a tablet, can also be used.

[0044] The smart speakers S2, S3, and S4 each have the same functions as the smart speaker S1 and are speaker devices that operate independently. The smart speakers S2, S3, and S4 are arranged in the rooms R2, R3, and R4, respectively.

[0045] An example is shown in which the user U's house has four rooms, but it is not limited thereto. The four rooms R1, R2, R3, and R4 (also referred to as R1, etc.) of the user U's house are, for example, a bedroom, a kitchen, a living room, and an entrance hall.

[0046] In the rooms R1, R2, R3, and R4, there are the devices D11 and D12, the devices D21 and D22, the devices D31 and D32, and the device D41, respectively. The above devices (also referred to as devices D11, etc.) are, for example, electrical devices such as lighting devices, air conditioners, vacuum cleaners, washing machines, refrigerators, and dishwashers, and are devices that accept voice operations by the user U via the smart speaker S1, etc. and the server 10. The devices D11, etc. are each communicably connected to the server 10 via the network N.

[0047] FIG. 2 is a block diagram showing the functional configuration of the information processing system 1 according to the present embodiment.

[0048] As shown in FIG. 2, the information processing system 1 includes an acquisition unit 11, a recognition unit 12, a control unit 13, and a management unit 14. Each functional unit included in the information processing system 1 can be realized by a CPU included in the server 10 or the smart speaker S1 or the like executing a predetermined program using a memory.

[0049] The acquisition unit 11 is a functional unit that acquires sound data from the voice of the user U. The acquisition unit 11 acquires first sound data from the voice of the user U for controlling the device D11 or the like. Specifically, the acquisition unit 11 acquires sound data (corresponding to the first sound data) indicating the voice of the user generated by the smart speaker S1 or the like. Since the above-mentioned voice of the user includes the voice uttered by the user for controlling the device D11 or the like, the first sound data acquired by the acquisition unit 11 includes sound data related to the voice of the user U for controlling the device D11 or the like. For example, an example of the voice of the user U acquired by the acquisition unit 11 is the voice of "Turn on the air conditioner" uttered by the user.

[0050] The recognition unit 12 is a processing unit that recognizes the words indicated by the first sound data acquired by the acquisition unit 11. The recognition unit 12 acquires the first sound data acquired by the acquisition unit 11 and performs voice recognition processing on the acquired first sound data to recognize the words indicated by the first sound data. The words recognized by the recognition unit 12 include at least the name of the device D11 or the like, words related to commands, and shortcut words. Known techniques can be adopted for the voice recognition processing. For example, when the acquisition unit 11 acquires the voice of the user "Turn on the air conditioner", the recognition unit 12 recognizes the word "air conditioner" which is the name of the device and the word "turn on" which indicates the command to change to the ON state.

[0051] The control unit 13 is a functional unit that determines control information (also referred to as first control information) of devices D11, etc., corresponding to the words and phrases recognized by the recognition unit 12, and controls the devices D11, etc. according to the determined first control information. When the control unit 13 controls the devices D11, etc., the control unit 13 transmits a communication packet including control information via the network N using a communication interface. For example, when the acquisition unit 11 acquires the user's voice of "Turn on the air conditioner", the control unit 13 transmits a communication packet including control information for changing the air conditioner as the device D11 to the ON state based on the recognition result of the words and phrases by the recognition unit 12, and changes the air conditioner to the ON state. Also, when the acquisition unit 11 acquires the user U's voice including a shortcut phrase (for example, "I'm home"), the control unit 13 refers to the content of the shortcut command managed by the management unit 14 and transmits a communication packet including control information including commands for each device associated with the shortcut.

[0052] The management unit 14 is a functional unit that manages shortcuts, which are commands for controlling devices D11, etc. Specifically, the management of shortcuts includes the generation, new registration, modification, and invalidation of shortcuts.

[0053] The management unit 14 has a proposal unit 15 that is a functional unit for making proposals about shortcuts by voice. A shortcut is a command different from the commands predetermined as commands for controlling devices D11, etc. Note that a shortcut is, for example, a function for controlling devices D11, etc. by a shortened phrase of a phrase related to a predetermined command for controlling devices D11, etc. A shortcut is generally sometimes called an alias.

[0054] When the acquisition unit 11 acquires second sound data from the user's voice after the acquisition unit 11 acquires the first sound data, the proposal unit 15 makes a proposal to the user by voice regarding a shortened phrase for controlling the devices D11, etc. according to the first control information. The proposal by the proposal unit 15 is made by outputting the voice related to the proposal by the smart speaker S1, etc.

[0055] The abbreviated phrase may be a phrase for controlling one device D11 or the like, or may be a phrase for controlling a plurality of devices D11 or the like.

[0056] That is, when the voice uttered by the user includes, in the voice uttered collectively by the user for controlling each of the plurality of devices D11 or the like, the information processing system 1 operates as follows. The acquisition unit 11 acquires a plurality of first sound data included in the voice uttered collectively by the user for controlling each of the plurality of devices D11 or the like. The recognition unit 12 recognizes each of the phrases indicated by the plurality of first sound data acquired by the acquisition unit 11. The control unit 13 determines control information (also referred to as first control information) of the devices D11 or the like corresponding to each of the plurality of phrases recognized by the recognition unit 12, and controls the devices D11 or the like according to the determined first control information. When the proposal unit 15 acquires second sound data from the user's voice after the time when the plurality of first sound data are acquired, the proposal unit 15 makes a proposal regarding one abbreviated phrase for collectively controlling the plurality of devices D11 or the like according to the plurality of first control information.

[0057] Specifically, the proposal made by the proposal unit 15 includes at least a registration proposal, a modification proposal, and an invalidation proposal.

[0058] The registration proposal is a proposal to newly register an abbreviated phrase. When the proposal unit 15 makes a registration proposal, the management unit 14 executes the registration process related to the registration proposal when obtaining information indicating that the user approves the registration proposal. The proposal unit 15 may make an inquiry for receiving from the user a designation of a phrase (also referred to as a shortcut phrase) for executing a shortcut related to the registration at the time of the registration proposal. In that case, the management unit 14 registers and manages the shortcut phrase designated by the user in response to this inquiry as a phrase for executing a shortcut related to the registration.

[0059] Note that the registration proposal may be made only when a condition indicating that the first control information and the second control information in the process related to the following modification proposal are similar is not satisfied.

[0060] A correction proposal is a proposal to correct an already registered abbreviation so that it is registered as an abbreviation for other controls. When the acquisition unit 11 acquires the first sound data in a state where the abbreviation is registered, the information processing system 1 operates as follows. The recognition unit 12 recognizes the phrase indicated by the second sound data acquired by the acquisition unit 11. The control unit 13 determines the second control information such as the device D11 corresponding to the phrase recognized by the recognition unit 12, and controls the device D11 etc. according to the determined second control information. When a condition indicating that the first control information and the second control information are similar is satisfied, the proposal unit 15 makes a correction proposal to correct the registered abbreviation so that it is registered as an abbreviation for controlling the device D11 etc. according to the second control information. When the proposal unit 15 makes a correction proposal, when information indicating that the user approves the correction proposal is obtained, the correction process related to the correction proposal is executed.

[0061] An invalidation proposal is a proposal to invalidate an already registered abbreviation. When the acquisition unit 11 acquires the second sound data in a state where the abbreviation is registered and a predetermined time has elapsed since the abbreviation was registered, the information processing system 1 operates as follows. The proposal unit 15 makes an invalidation proposal to invalidate the registered abbreviation. When the proposal unit 15 makes an invalidation proposal, when information indicating that the user approves the invalidation proposal is obtained, the invalidation process related to the invalidation proposal is executed.

[0062] In addition, when the proposal unit 15 intends to make a proposal, if it determines that the location where the device that emits the voice related to the proposal (that is, the smart speaker S1 or the like) is placed, or the time for making the proposal does not meet the predetermined conditions indicating that it is suitable for the proposal, the proposal may be prohibited. An example of the above-mentioned predetermined conditions can be that the smart speaker, which is the device that emits the voice related to the proposal, is placed in a room other than the entrance hall. In general life, people only stay at the entrance hall for about one minute. In that case, even if a proposal is made by the smart speaker placed at the entrance hall, the user may already not be at the entrance hall, and in that case, consent cannot be obtained, and the power consumption required for the processing for the proposal is wasted. This is also because it is inconvenient for the user to have to stay at the entrance hall until consent is given for the proposal.

[0063] Also, another example of the above-mentioned predetermined conditions can be that the time for making the proposal is a time other than the user's bedtime. It is assumed that the user performs a voice operation to turn off the lighting at bedtime, but even if a proposal regarding a shortcut is made at that time, the user may already be asleep, and in that case, consent cannot be obtained, and the power consumption required for the processing for the proposal is wasted. This is also because it is inconvenient for the user to be unable to sleep until consent is given for the proposal, or to have their sleep disturbed when a proposal is made while they are asleep.

[0064] Hereinafter, the processes executed by each functional unit will be specifically described while referring to various types of information.

[0065] FIG. 3 is an explanatory diagram showing a log of state changes by voice operation of the device D11 or the like according to the present embodiment. The log shown in FIG. 3 is an extraction of those in the operation logs of the device D11 or the like in which the operation state has changed by voice operation.

[0066] As shown in FIG. 3, entries 31, 32, 33, 34, 35, 36, 37, ··· are included as logs of state changes. One entry indicates a state change due to one voice operation and includes information on the time, device ID, state before change, and state after change. It can be said that one entry corresponds to one voice operation. Also, entries 31 to 36 belong to operation group 41. Operation group 41 will be described later.

[0067] The time indicates the time when the voice operation related to the entry was performed. More specifically, it indicates the time when the control related to the voice operation related to the entry was executed.

[0068] The device indicates the device D11 etc. on which the voice operation related to the entry was performed. Note that an identifier that can uniquely identify the device D11 etc. on which the voice operation related to the entry was performed can also be used.

[0069] The state before change indicates the state of the device D11 etc. before the voice operation related to the entry is performed.

[0070] The state after change indicates the state of the device D11 etc. after the voice operation related to the entry is performed.

[0071] For example, entry 31 shown in FIG. 3 indicates that at the time "June 28, 2019, 8:00", a voice operation was performed on the device "bedroom air conditioner", and it changed from the ON state (i.e., the cooling operation state) at a setting of 23°C in cooling to the OFF state (i.e., the stopped state).

[0072] Also, entry 32 shown in FIG. 3 indicates that at the time "June 28, 2019, 8:01", a voice operation was performed on the device "bedroom lighting", and it changed from the ON state (i.e., the lit state) to the OFF state (i.e., the unlit state).

[0073] Entries 33 to 37 are as shown in FIG. 3.

[0074] Note that the voice operations related to the logs shown in Entries 31 to 37 are an example of the voice operations executed by the acquisition unit 11, the recognition unit 12, and the control unit 13.

[0075] FIG. 4 is an explanatory diagram showing state change data according to the present embodiment.

[0076] The state change data shown in FIG. 4 is an example of data generated by the management unit 14 from the log of the state change shown in FIG. 3.

[0077] The management unit 14 acquires the log of the state change shown in FIG. 3, refers to the time column included in each entry of the log of the state change, and groups one or more voice operations corresponding to each entry of the log of the state change shown in FIG. 3 into one or more operation groups. At this time, for each entry of the log of the state change shown in FIG. 3, the management unit 14 generates an operation group by regarding an entry that is separated from the previous entry by a predetermined time or more as the first voice operation included in the operation group. In other words, the management unit 14 divides the log of the state change shown in FIG. 3 at intervals of a period having a time length of a predetermined time or more during which no voice operation is performed, and groups a plurality of operations into one or more operation groups. The predetermined time is, for example, about 10 minutes. In this case, Entries 31 to 36 in FIG. 3 are grouped into Operation Group 41.

[0078] As shown in FIG. 4, the state change data is data indicating state changes of one or more devices D11 and the like. One state change data is generated corresponding to one operation group.

[0079] More specifically, the state change data includes the state of each device immediately before the first operation included in the operation group corresponding to the state change data, and the state of each device immediately after the last operation included in the operation group. Further, it has the time when the last operation included in the operation group was performed as the "completion time".

[0080] One state change data is data indicating how the state of the devices D11 and the like has changed before and after the operation group corresponding to the state change data.

[0081] For example, in the state change data shown in FIG. 4, the state before the change for the living room air conditioner is "ON / cooling / 23 degrees C (that is, cooling operation with a temperature setting of 23 degrees C)", and the state after the change is "OFF". This state change corresponds to the log of entry 33 shown in FIG. 3.

[0082] Also, the state before the change for the living room lighting is "ON", and the state after the change is "OFF". This state change corresponds to the log of entry 34 shown in FIG. 3.

[0083] Also, the state before the change for the living room vacuum cleaner is the stopped state, and the state after the change is the cleaning state. This state change corresponds to the log of entry 35 shown in FIG. 3.

[0084] The same applies to other state changes as shown in FIG. 4.

[0085] FIG. 5 is an explanatory diagram showing the content of an instruction by a generated shortcut according to the present embodiment. FIG. 6 is an explanatory diagram showing an example of an instruction table according to the present embodiment. FIG. 7 is an explanatory diagram showing the registration of a shortcut according to the present embodiment. The generation process and registration process of the shortcut will be described with reference to FIGS. 5, 6, and 7.

[0086] The shortcut 61 shown in FIG. 5 is an example of a shortcut generated from the state change data shown in FIG. 4. The shortcut is a collection of instructions for devices D11, etc. for causing the state changes shown in FIG. 4. Here, the devices D11, etc. that are the control targets are shown grouped by room. Also, the state, phrase, and creation date and time are associated with the shortcut 61. The state is information indicating the state of the shortcut 61 and is selected from unproposed, valid, invalid, etc. The phrase is a phrase for executing the shortcut and is also referred to as a shortcut phrase. The creation date and time indicates the date and time when the shortcut was created.

[0087] FIG. 6 shows an instruction table that defines control instructions necessary to change from one state to another for a plurality of states that the device D11 and the like can assume.

[0088] In the instruction table shown in FIG. 6, it is defined that the control instruction necessary to transition the living room air conditioner from the ON state to the OFF state is the OFF instruction. It is defined that the control instruction necessary to transition the living room lighting from the ON state to the OFF state is the OFF instruction. It is defined that the control instruction necessary to transition the living room cleaner from the stopped state to the cleaning state is the START instruction.

[0089] The management unit 14 generates the shortcut shown in FIG. 5 by extracting from the instruction table an instruction for causing the state change described in the state change data shown in FIG. 4 to occur in the device D11 and the like.

[0090] For example, the management unit 14 extracts the OFF instruction from the instruction table as an instruction for causing a state change 51, that is, a state change for changing the living room air conditioner from the state “ON / cooling / 23° C.” to the state “OFF”.

[0091] Also, the management unit 14 extracts the OFF instruction from the instruction table as an instruction for causing a state change 52, that is, a state change for changing the living room lighting from the state “ON” to the state “OFF”.

[0092] Also, the management unit 14 extracts the START instruction from the instruction table as an instruction for causing a state change 53, that is, a state change for changing the living room cleaner from the state “stopped” to the state “cleaning”.

[0093] In this way, the management unit 14 generates the shortcut 61 in FIG. 5 by extracting from the instruction table an instruction for causing the state change of each device D11 and the like shown in FIG. 4 to occur.

[0094] In addition, when there is no change in the state of each device D11 etc. shown in FIG. 4 (that is, the state before the change and the state after the change are the same), there is no instruction to cause a state change. In this case, the management unit 14 includes information indicating that there is no corresponding instruction (described as "no instruction" in FIG. 5) in the shortcut.

[0095] When the management unit 14 generates a shortcut, it sets the state of the generated shortcut to "not proposed".

[0096] After a shortcut in the "not proposed" state is generated in FIG. 6, the proposal unit 15 proposes to the user to register the generated shortcut as a valid (that is, usable) shortcut (that is, a registration proposal).

[0097] When information indicating that the user approves the registration proposal is obtained, the management unit 14 registers this shortcut as a valid shortcut. Note that when registering, the management unit 14 associates the shortcut phrase specified by the user with the shortcut and registers it (see FIG. 7).

[0098] The processing executed by the information processing system 1 configured as described above will be described. Hereinafter, (1) processing for newly registering a shortcut, (2) processing for selectively performing new registration or modification according to the similarity of the shortcut, and (3) processing for invalidating the shortcut will be described in order.

[0099] (1) Processing for newly registering a shortcut FIG. 8 is a flowchart showing the processing for registering a shortcut according to the present embodiment.

[0100] In step S101, the acquisition unit 11 determines whether it has acquired audio data from the voice of user U for controlling device D11 or the like. If it is determined that the audio data has been acquired (Yes in step S101), the process proceeds to step S102; otherwise (No in step S101), step S101 is executed again. That is, the acquisition unit 11 waits in step S101 until the audio data is acquired. This audio data corresponds to the first audio data.

[0101] In step S102, the recognition unit 12 recognizes the phrase indicated by the audio data acquired in step S101.

[0102] In step S103, the control unit 13 determines the first control information for device D11 or the like corresponding to the phrase recognized by the recognition unit 12 in step S102, and controls device D11 or the like according to the determined first control information.

[0103] In step S104, the management unit 14 acquires a log of the state change of device D11 or the like by voice operation.

[0104] In step S105, the management unit 14 generates state change data based on the log of the state change acquired in step S104.

[0105] In step S106, the management unit 14 generates a shortcut based on the state change data generated in step S105. At this time, the state of the generated shortcut is "not proposed". The generated shortcut cannot be used in this state.

[0106] In step S107, the proposal unit 15 determines whether it has acquired audio data from the voice of user U. If it is determined that the audio data has been acquired (Yes in step S107), the process proceeds to step S108. Otherwise (No in step S107), step S107 is executed again. That is, the proposal unit 15 waits in step S107 until the audio data is acquired. This audio data corresponds to the second audio data. Note that in this step, the voice of user U may or may not be a voice for controlling devices such as device D11.

[0107] In step S108, the recognition unit 12 recognizes the phrase indicated by the second audio data acquired by the acquisition unit 11 in step S107.

[0108] In step S109, the control unit 13 determines the second control information of devices such as device D11 corresponding to the phrase recognized by the recognition unit 12 in step S108, and controls devices such as device D11 according to the determined second control information.

[0109] In step S110, the proposal unit 15 proposes the registration of the shortcut generated in step S106.

[0110] In step S111, the management unit 14 determines whether it has acquired information indicating that user U has agreed to the registration proposal in step S110. If it has acquired information indicating that user U has agreed (Yes in step S111), the process proceeds to step S112. Otherwise (No in step S111), the series of processes shown in FIG. 8 is terminated.

[0111] In step S112, the management unit 14 registers the shortcut generated in step S106 as a valid shortcut. After that, if the voice uttered by the user includes the shortcut phrase, the management unit 14 issues an instruction related to the shortcut phrase to devices such as device D11.

[0112] When proposing to register a shortcut, the proposal unit 15 proposes to the user by a message such as, for example, "Do you want to register a voice operation at 8:00 am as a shortcut?" In this way, the proposal unit 15 may include information indicating the time when the control according to the first control information was performed in the proposal message. Also, the above message may not include the specific content of the voice operation. If, for example, a message such as "Do you want to register a shortcut for the operation of turning off the air conditioner, turning off the lighting, and starting cleaning?" is used, it is complicated and the voice becomes long, so the processing amount and power consumption of the information processing system 1 increase, and it is also difficult for the user to understand.

[0113] Regarding the time expression "at 8:00 am" in the above message, if there is no other voice operation that can be the target of shortcut registration, a simpler expression may be used. For example, expressions such as "this morning" or "when going out this morning" may be used.

[0114] Through a series of processes shown in FIG. 8, the information processing system 1 proposes to newly register a shortcut and, if the commitment is obtained, performs the registration of the shortcut.

[0115] (2) Processing for selectively performing new registration or modification according to the similarity of the shortcut First, the modification of the shortcut will be described. Modifying the shortcut means modifying the content of the command of the shortcut that has already been registered.

[0116] FIG. 9 is a first explanatory diagram showing the modification of the shortcut according to the embodiment.

[0117] The shortcuts 61 and 62 shown in FIG. 9 are those generated by the above (1) and registered as valid shortcuts.

[0118] The shortcut 63 shown in FIG. 9 is a shortcut that is generated by the above (1) and is in an unproposed state (the state after step S106).

[0119] Here, the creation time of the shortcut 62 is 19:10, and the creation time of the shortcut 63 is 19:02. These only differ by a range of several minutes at most. Therefore, for the user, it may be more convenient to register the content of the shortcut 63 as a modification of the shortcut 62 that has already been registered as a valid shortcut, that is, as a shortcut using the phrases of the shortcut 62.

[0120] An example of the result of the modification when registering as a modification is shown as columns 62A and 63A in FIG. 9.

[0121] Since the shortcut 62 is modified to the shortcut 63, its state becomes invalid and the shortcut phrase is deleted (see column 62A).

[0122] Also, the shortcut 63 is registered as a valid shortcut, and the phrase "now" that was registered as the phrase of the shortcut 62 is registered as the phrase of the shortcut 63 (see column 63A).

[0123] Thus, a method will be described in which, instead of newly registering a new shortcut, a proposal for modifying a shortcut can be made.

[0124] FIG. 10 is a flowchart showing a process of selectively performing new registration and modification of shortcuts according to the present embodiment. FIG. 10 is assumed to be executed in a state where at least one shortcut has already been registered. The at least one shortcut already registered corresponds to the shortcuts 61 and 62 shown in FIG. 9.

[0125] The processing from step S101 to step S109 shown in FIG. 10 is the same as the processing with the same reference numerals shown in FIG. 8. The "unproposed" state shortcut generated in step S106 corresponds to the shortcut 63 shown in FIG. 9.

[0126] In step S121, the proposal unit 15 determines whether a condition indicating that the first control information and the second control information are similar is satisfied. If it is determined that the above condition is satisfied (Yes in step S121), the process proceeds to step S131, and if not (No in step S121), the process proceeds to step S122. When there is a plurality of first control information, it is determined whether the above condition is satisfied between each of the plurality of first control information and the second control information.

[0127] In steps S122 to S124, the management unit 14 makes a registration proposal for the shortcut generated in step S106 in the same manner as steps S110 to S112 shown in FIG. 8, and registers it as a valid shortcut when approval for the registration proposal is obtained.

[0128] In step S131, the proposal unit 15 makes a modification proposal for the shortcut generated in step S106.

[0129] In step S132, the management unit 14 determines whether information indicating that the user U approves the modification proposal made in step S131 has been obtained. If information indicating that the user U approves has been obtained (Yes in step S132), the process proceeds to step S133, and if not (No in step S132), the series of processes shown in FIG. 10 ends.

[0130] Note that when information indicating that the user U approves cannot be obtained in step S132 (No in step S132), a new shortcut proposal may be made (step S122). Thereby, similar shortcuts can be registered separately.

[0131] In step S133, the management unit 14 corrects the first control information determined to be similar to the second control information in step S121 into the shortcut generated in step S106. At this time, the shortcut phrase uses the phrase of the shortcut before correction as it is. After that, when the voice uttered by the user includes the shortcut phrase, the management unit 14 executes an instruction related to the shortcut phrase on the device D11 or the like.

[0132] Note that the information processing system 1 may output, as voice, a message indicating the correction content at the time of proposing a correction. For example, when correcting the shortcut 62 shown in FIG. 9 to the shortcut 63, a message such as "Of the shortcuts of 'right now', do you want to invalidate the instruction for the air conditioner in the living room and invalidate the instruction for the air conditioner in the bedroom?" may be output as voice. By doing so, the user can specifically know the correction content.

[0133] Note that by devising the shortcut generation process, it may be possible to shorten the length of the correction proposal message. In FIG. 5, the devices D11 and the like are shown grouped by room, but when grouped by the type of the devices D11 and the like, the shortcut is expressed as shown in FIG. 11.

[0134] FIG. 11 is a second explanatory diagram showing the correction of the shortcut according to the present embodiment.

[0135] The shortcuts 62B and 63B shown in FIG. 11 are the shortcuts 62 and 63 shown in FIG. 9 expressed by device type, respectively. This expression can also be said to be an expression in which a plurality of devices D11 and the like are grouped by devices belonging to the same type.

[0136] In this case, the proposal unit 15 may make a proposal including a message for collectively changing the instructions for all the devices belonging to the same group.

[0137] When expressing shortcuts for each type of machine, modifying from shortcut 62B to shortcut 63B is equivalent to invalidating commands for all air conditioners.

[0138] Therefore, when making a modification proposal, the proposal unit 15 can use a message saying "Do you want to invalidate commands for all air conditioners among the current shortcuts?" By doing so, the message related to the proposal can be shortened, reducing the information processing and power consumption of the information processing system 1, and enabling an easy-to-understand proposal for the user.

[0139] Note that the result of the modification when registering as a modification is the same as in the case of FIG. 9 (see columns 62C and 63C in FIG. 11).

[0140] Note that when making a modification proposal, the shortcut may be modified to include commands made at a time relatively close to the execution of the shortcut, together with the registered shortcut.

[0141] FIG. 12 is a third explanatory diagram showing the modification of the shortcut according to the present embodiment. FIG. 12 shows a log of the execution of the shortcut and the state changes by voice operations of devices D11, etc.

[0142] FIG. 12 includes an entry 71 which is a log of the execution of shortcut 62, and entries 72, 73,... which are logs of state changes.

[0143] The entry 71 shown in FIG. 12 indicates that the shortcut 62 was executed at the time "October 7, 2019, 19:00". Also, it is shown that the voice operations shown in entries 72 and 73 were made at the times "October 7, 2019, 19:00" and "October 7, 2019, 19:02", respectively.

[0144] Similar to the description in FIG. 3, the management unit 14 groups the execution of the shortcuts and voice operations shown in FIG. 12 into one or more operation groups. At this time, the management unit 14 generates an operation group by treating an entry that is separated from the immediately previous entry by a predetermined time or more as the first voice operation included in the operation group. As a result, entries 71 to 73 are grouped into operation group 75.

[0145] In this case, the management unit 14 proposes to modify the shortcut 62 to the shortcut 63 by adding the voice operations shown in the logs of entries 72 and 73 to the instructions included in the shortcut 62.

[0146] By doing so, the management unit 14 can modify the content of the shortcut 62.

[0147] (3) Process for disabling shortcuts First, the disabling of shortcuts will be explained. Disabling a shortcut means making a previously registered shortcut unusable (i.e., in an invalid state). Disabling a shortcut includes both deleting the registration of the shortcut and keeping it in an unusable state.

[0148] As an example, a shortcut is disabled when the usage frequency of the registered shortcut becomes equal to or less than a predetermined frequency. The predetermined frequency is, for example, about 1 to 2 times a week or 0 times.

[0149] FIG. 13 is a flowchart showing the process for disabling shortcuts according to the present embodiment.

[0150] Steps S101 to S103 are the same as the steps with the same reference numerals shown in FIG. 8.

[0151] In step S201, the proposal unit 15 determines whether or not it has acquired the voice of user U. If it is determined that the voice of user U has been acquired (Yes in step S201), the process proceeds to step S202. Otherwise (No in step S201), step S201 is executed again. That is, the proposal unit 15 waits in step S201 until the voice of user U is acquired. Note that in this step, the voice of user U may or may not be a voice for controlling devices such as device D11. Also, if a proposal for invalidation is to be executed regardless of whether the voice of user U is a voice for controlling devices such as device D11, it is not necessary to execute the subsequent steps S202 and S203.

[0152] In step S202, the recognition unit 12 recognizes the phrase indicated by the second audio data acquired by the acquisition unit 11 in step S201.

[0153] In step S203, the control unit 13 determines the second control information of devices such as device D11 corresponding to the phrase recognized by the recognition unit 12 in step S202, and controls devices such as device D11 according to the determined second control information.

[0154] In step S204, the management unit 14 extracts shortcuts among the already registered shortcuts whose usage frequency is equal to or lower than a predetermined frequency.

[0155] In step S205, the proposal unit 15 makes a proposal to invalidate the shortcuts extracted by the management unit 14 in step S204.

[0156] In step S206, the management unit 14 determines whether or not it has acquired information indicating that user U has approved the invalidation proposal made in step S205. If it has acquired information indicating that user U has approved (Yes in step S206), the process proceeds to step S207. Otherwise (No in step S206), the series of processes shown in FIG. 13 is terminated.

[0157] In step S207, the management unit 14 invalidates the shortcut that has acquired the information indicating approval in step S106.

[0158] Hereinafter, the flow of shortcut proposal by the information processing system 1 according to the present embodiment will be described while comparing it with the information processing system according to the related art.

[0159] FIG. 14 is an explanatory diagram showing a state of shortcut proposal by the information processing system according to the related art.

[0160] FIG. 14 shows a state where the user U controls devices using a smart speaker in the room R1. The smart speaker is connected to a server (not shown) via a network (not shown). This server has a function of proposing a shortcut that summarizes a plurality of voice operations performed in a relatively short time, and performing the proposal by the smart speaker 10 minutes after the plurality of voice operations are performed.

[0161] Assume that at 8:00, the user uttered the voice "Turn off the air conditioner" related to the voice operation of turning off the air conditioner, the voice "Turn off the lighting" related to the voice operation of turning off the lighting, and the voice "Start cleaning" related to the voice operation of starting cleaning with a vacuum cleaner in preparation for going out.

[0162] After uttering the above voices, the user goes out, and no one is left in the room R1. The information processing system executes control to turn off the air conditioner, turn off the lighting, and start cleaning with the vacuum cleaner based on the above voices.

[0163] At 8:10, the smart speaker makes a registration proposal for a shortcut that summarizes these voice operations. This corresponds to about 10 minutes after the plurality of voice operations are performed.

[0164] However, when a shortcut registration proposal is made, user U is not in room R1. Therefore, no response to the registration proposal can be obtained. As a result, since no approval response is obtained from the user, the shortcut is not registered.

[0165] FIG. 15 is an explanatory diagram showing how a shortcut is proposed by the information processing system 1 according to the present embodiment.

[0166] Similar to the case in FIG. 14, it is assumed that at 8:00, user U uttered voices related to the voice operations of turning off the air conditioner, turning off the lighting, and starting the cleaning of the vacuum cleaner. After that, user U goes out and no one is left in room R1. The information processing system 1 executes control of turning off the air conditioner, turning off the lighting, and starting the cleaning of the vacuum cleaner based on the above voices (steps S101 to S103 in FIG. 8).

[0167] At 19:00, user U returns home from outside.

[0168] It is assumed that after returning home, user U uttered the voice "Turn on the lighting" related to the voice operation for turning on the lighting. The information processing system 1 executes control of turning on the lighting based on the above voice (steps S108 to 109).

[0169] Also, a registration proposal for a shortcut summarizing the voice operations performed around 8:00 is made, and the shortcut is registered after obtaining approval from the user (steps S110 to S112).

[0170] As described above, according to the information processing system 1 of the present embodiment, since the proposal related to the shortcut of the voice operation performed by user U is made when the subsequent voice operation of user U is performed, user U can be made to listen to the voice related to the proposal. In other words, the information processing system 1 can avoid a situation where, when user U is not in room R1, a voice of a proposal related to a voice operation is emitted and no response to the registration proposal can be obtained. In this way, the information processing system 1 can effectively make a proposal regarding the shortcut.

[0171] (Modification of the Embodiment) FIG. 16 is a flowchart showing an information processing method according to this modification example.

[0172] In step S11, first sound data is acquired from the user's voice for controlling the device.

[0173] In step S12, the words indicated by the acquired first sound data are recognized.

[0174] In step S13, first control information of the device corresponding to the recognized words is determined.

[0175] In step S14, the device is controlled according to the determined first control information.

[0176] In step S15, when second sound data is acquired from the user's voice after the time when the first sound data was acquired, a proposal regarding a shortcut for controlling the device according to the first control information is made by voice.

[0177] Through the series of processes shown in FIG. 16, the registration of shortcuts for device control is executed more appropriately.

[0178] As described above, the embodiments have been described as examples of the technology in the present disclosure. For this purpose, the accompanying drawings and detailed description have been provided.

[0179] Therefore, among the components described in the accompanying drawings and detailed description, not only the components essential for solving the problems but also the components not essential for solving the problems may be included for exemplifying the above implementation. Therefore, just because those non-essential components are described in the accompanying drawings and detailed description, it should not be immediately determined that those non-essential components are essential.

[0180] In addition, since the above-described embodiments are for exemplifying the technology in the present disclosure, various changes, replacements, additions, omissions, etc. can be made within the scope of the claims or the equivalent scope thereof.

Industrial Applicability

[0181] The present disclosure is applicable to an information processing system that controls equipment and the like.

Explanation of Signs

[0182] 1 Information processing system 10 Server 11 Acquisition unit 12 Recognition unit 13 Control unit 14 Management unit 15 Proposal unit 31, 32, 33, 34, 35, 36, 37, 71, 72, 73 Entry 41, 75 Operation group 51, 52, 53 State change 61, 62, 62B, 63, 63B Shortcut 62A, 62C, 63A, 63C Column D11, D12, D21, D22, D31, D32, D41 Equipment N Network R1, R2, R3, R4 Room S1, S2, S3, S4 Smart speaker U User

Claims

1. An information processing method executed by a computer, comprising: obtaining first sound data from a user's voice for controlling a device, the voice including the name of the device and a command for controlling the device; recognizing a phrase indicated by the obtained first sound data; determining first control information of the device corresponding to the recognized phrase; controlling the device according to the determined first control information; obtaining sound data from a second user voice for controlling the device after a predetermined time has elapsed since the first sound data was obtained, the second voice including the name of the device and a command for controlling the device, and using the obtained sound data as second sound data; recognizing a phrase indicated by the obtained second sound data, determining second control information of the device corresponding to the recognized phrase, and controlling the device corresponding to the phrase according to the determined second control information; making a proposal regarding a shortcut for controlling the device according to the first control information by voice; wherein the predetermined time is a time having a predetermined time length during which no voice operation is performed, and is used as a delimiter when grouping the user's voice operations into one or more operation groups in the control of the device by the user's voice operation. An information processing method.

2. The proposal includes a registration proposal for proposing to newly register the shortcut, and when information indicating that the user approves the registration proposal is obtained, performing the registration according to the registration proposal. The information processing method according to claim 1.

3. When the first sound data is obtained in a state where an existing shortcut is already registered, the proposal includes a modification proposal for modifying the registered existing shortcut to be registered as a shortcut for controlling the device corresponding to the phrase according to the second control information when a condition indicating that the first control information and the second control information are similar is satisfied, and when information indicating that the user approves the modification proposal is obtained, performing the modification according to the modification proposal. The information processing method according to claim 1 or 2.

4. When the second sound data is obtained in a state where an existing shortcut is already registered and a second predetermined time has elapsed since the existing shortcut was registered, the proposal includes an invalidation proposal for proposing to invalidate the registered existing shortcut. ​ When information indicating that the user has accepted the invalidation proposal is obtained, perform the invalidation related to the invalidation proposal. The information processing method according to claim 1 or 2.

5. The device includes a plurality of devices. The first sound data includes a plurality of first sound data included in the voice collectively uttered by the user for controlling each of the plurality of devices. The first control information includes a plurality of first control information, which is the first control information corresponding to each of the phrases recognized as being indicated by each of the plurality of first sound data. When the second sound data is obtained from the user's voice after the time when the plurality of first sound data is obtained, propose a shortcut for collectively controlling the plurality of devices according to the plurality of first control information. The information processing method according to any one of claims 1 to 4.

6. The plurality of devices are grouped by devices belonging to the same type. The proposal is including a message for collectively changing commands for all devices belonging to the same group. The information processing method according to claim 5.

7. The proposal includes information indicating the time when the control according to the first control information was performed. The information processing method according to any one of claims 1 to 6.

8. The proposal does not include the content of the first control information. The information processing method according to any one of claims 1 to 7.

9. The acquisition of the second sound data is performed by a microphone arranged at the position where the voice uttered by the user reaches, and the microphone and the speaker are provided in the same housing. The proposal is made by the voice emitted by the speaker. The information processing method according to any one of claims 1 to 8.

10. During the proposal, if it is determined that the place where the device that emits the voice related to the proposal is arranged or the time when the proposal is made does not satisfy a predetermined condition indicating that it is suitable for the proposal, prohibit the proposal. The information processing method according to any one of claims 1 to 9.

11. A program for causing a computer to execute the information processing method according to any one of claims 1 to 10.

12. Voice of the user for controlling the device, including the name of the device and the command for controlling the device an acquisition unit that acquires first sound data from the voice; a recognition unit that recognizes the phrase indicated by the first sound data acquired by the acquisition unit; A control unit that determines first control information of the device corresponding to the phrase recognized by the recognition unit and controls the device according to the determined first control information; Second voice data of the user for controlling the device after a predetermined time has elapsed since the time when the first voice data was acquired, the voice data being acquired from second voice including the name of the device and an instruction for controlling the device, and the acquired voice data being used as the second voice data; Recognize the phrase indicated by the acquired second voice data, determine second control information of the device corresponding to the recognized phrase, and control the device corresponding to the phrase according to the determined second control information; And a proposal unit that makes a proposal regarding a shortcut for controlling the device according to the first control information by voice. The predetermined time is a time having a predetermined time length during which no voice operation is performed, and is used as a delimiter when summarizing the user's voice operation as one or more operation groups in the control of the device by the user's voice operation. An information processing apparatus.

Images