Information processing device

Abstract

The present disclosure relates to an information processing device that enables a cluster, in which a plurality of operations which are habitual to a user are carried out collectively, to be changed in a simple manner. A conversation cluster is registered, the cluster being constituted by: a plurality of operations in which a habituation score indicating user operation habituation is greater than a prescribed threshold; and an operation that is in accordance with a conversation with the user. The present disclosure is applicable to information processing devices.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, and more particularly to an information processing device that enables a user to easily change a cluster for collectively performing a plurality of operations that the user has become accustomed to. [Background technology]

[0002] A technique has been proposed for inferring a user's operation habits from the user's operation history and assisting the user in performing operations.

[0003] For example, a technique has been proposed in which an operation history is stored, the next operation is predicted from a transition probability, and the predicted next operation is executed (see Patent Document 1).

[0004] Also, a technology has been proposed that estimates viewing habits for different days of the week and times from viewing histories and assists with viewing-related operations (see Patent Document 2).

[0005] However, the technology in Patent Document 1 cannot estimate multiple simultaneous operations taking into account the time intervals between operations (cluster termination judgment as a group of operations), and further cannot detect operations in which the order of multiple operations may change.

[0006] Furthermore, the technology of Patent Document 2 cannot estimate habituation based on the continuity or simultaneity of multiple operations.

[0007] In view of this, it is conceivable to generate a cluster that allows easy collective operation by clustering a plurality of operations that have become the user's habit as a group of operations.

[0008] Even if a cluster is generated by clustering a plurality of operations as a group of operations in this way, it may be necessary to modify each operation as the user's behavior changes.

[0009] However, in order to change individual operations, it is necessary to individually specify the operation to which the change is to be made, which places a heavy burden on the user.

[0010] Therefore, a technology has been proposed in which a scenario-based dialogue is generated, some of the dialogue is left blank, and the blank items are filled in based on past behavioral history to complete the dialogue, thereby changing the cluster that performs multiple operations collectively (Patent Document 3). [Prior art documents] [Patent documents]

[0011] [Patent Document 1] Patent No. 5340774 [Patent Document 2] Patent No. 5175518 [Patent Document 3] JP 2019-105685 A Summary of the Invention [Problem to be solved by the invention]

[0012] However, when using the technology of Patent Document 3 to provide blank items in a scenario-based voice dialogue and fill in the blank items from behavioral history, it is not possible to change individual operations in a cluster consisting of a group of multiple operations based on the habitual nature of the composite operation.

[0013] For this reason, it was not possible to change the clusters appropriately in response to changes in compound operation habits.

[0014] The present disclosure has been made in consideration of such circumstances, and in particular makes it possible to easily change a cluster, which is a grouping of multiple operations that a user has become accustomed to, through appropriate dialogue based on an estimation of the user's habits. [Means for solving the problem]

[0015] An information processing device according to one embodiment of the present disclosure includes: a control unit that calculates a habit score indicating a habit of a user operation; and registers a cluster including a plurality of operations including an operation having the habit score higher than a predetermined threshold and an operation corresponding to a dialogue with the user. The operation corresponding to the dialogue with the user is an operation having the same operation type but different attributes, and the control unit sets whether or not to execute a dialogue related to selection of the attribute in the operation having the same operation type but different attributes by a selection dialogue flag. It is an information processing device.

[0016] In one aspect of the present disclosure, a habit score indicating the habit of a user operation is calculated, and a cluster including a plurality of operations including an operation having a habit score higher than a predetermined threshold and an operation corresponding to a dialogue with the user is registered. The operation corresponding to the dialogue with the user is an operation having the same operation type but different attributes, and a selection dialogue flag is set to determine whether or not to execute a dialogue related to the selection of the attribute in the operation having the same operation type but different attributes. do. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram for explaining browsing up to now. [Diagram 2] FIG. 11 is a diagram for explaining the macro registration process up to now. [Diagram 3] FIG. 1 is a diagram illustrating an overview of the present disclosure. [Figure 4] FIG. 1 is a diagram illustrating an example of the configuration of an information processing device to which the technology of the present disclosure is applied. [Diagram 5] 11 is a diagram illustrating user operation information and corresponding habit scores. FIG. [Figure 6] 13 is a flowchart illustrating an habitual operation clustering process. [Figure 7] FIG. 13 is a diagram illustrating the integration of addictive operation clusters. [Figure 8] FIG. 13 is a diagram illustrating the integration of addictive operation clusters. [Figure 9] 13 is a flowchart illustrating a dialogue clustering process. [Figure 10] FIG. 13 is a diagram for explaining operations in which the operation type (Intent) is the same but the attribute (Entity) is different. [Figure 11] 13 is a flowchart illustrating an operation determination process in which the operation type (Intent) is the same but the attribute (Entity) is different. [Figure 12]FIG. 13 is a diagram illustrating an example of dialogue generation for operations with the same Intent but different Entities. [Figure 13] 13 is a flowchart illustrating an interaction process for operations having the same Intent but different Entities. [Figure 14] FIG. 13 is a diagram illustrating an added operation. [Figure 15] FIG. 13 is a diagram illustrating a deleted operation. [Figure 16] 13 is a flowchart illustrating a process for determining whether an operation has been added or deleted. [Figure 17] FIG. 13 is a diagram illustrating an example of dialogue generation for added and deleted operations. [Figure 18] FIG. 13 is a diagram illustrating interaction for added and deleted operations. [Figure 19] FIG. 13 is a diagram illustrating an operation that depends on an observation context. [Figure 20] 13 is a flowchart illustrating a process of determining an operation dependent on an observation context. [Figure 21] 13 is a flowchart illustrating a process of updating context information of a dialogue cluster candidate. [Figure 22] FIG. 13 is a diagram illustrating an example of dialogue generation for an observation context-dependent operation. [Diagram 23] 11 is a flowchart illustrating an interaction process for an observation context-dependent operation. [Figure 24] FIG. 1 is a diagram illustrating an example of the configuration of a general-purpose personal computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0019] Hereinafter, an embodiment of the present technology will be described in the following order. 1. Overview of this Disclosure 2. Preferred Embodiments 3. Software implementation example

[0020] <<1. Overview of this Disclosure>> <Browsing> The present disclosure makes it possible to easily change a cluster, which is a grouping of a plurality of operations that a user has become accustomed to, through appropriate dialogue based on an estimation of the user's habits.

[0021] First, an overview of the present disclosure will be described.

[0022] The present disclosure will be outlined below by taking as an example a case where a user browses using a web browser on a PC (Personal Computer).

[0023] In normal browsing, as shown in the upper part of Figure 1, a user 11 operates an operation input device 12 consisting of a keyboard, mouse, etc. of a PC 13 to input keywords and various operational instructions, etc., thereby causing the browser to search for articles, and browsing is realized by presenting the search results.

[0024] However, as shown in the upper part of FIG. 1, the operation of the user 11 operating the operation input device 12 to input keywords and various operational instructions requires inputting multiple pieces of information in a predetermined order, and the operation becomes more complicated as the amount of information to be input increases.

[0025] Furthermore, if the operation is habitual, similar operations must be repeated multiple times.

[0026] Therefore, as shown in the lower part of Figure 1, a technology has been proposed in which, in addition to the operation input device 12, a voice input device 20 such as a microphone is added to accept voice input, allowing input to be made in an interactive format using a voice agent 21.

[0027] In this case, the user 11 can input information interactively to the voice agent 21 via the voice input device 20 by operating the operation input device 12 or by speaking.

[0028] As a result, as shown in the lower part of FIG. 1, when the voice agent 21 is used, in the input operation using the operation input device 12, the required operations can be made into macros and executed all at once by simply speaking the associated phrase.

[0029] That is, a plurality of operations are made into a macro and stored in association with one phrase, and the plurality of operations made into a macro are executed collectively by speaking the stored phrase.

[0030] In order to execute multiple operations that have been made into a macro by speaking a specified phrase in this way, it is first necessary to carry out a macro registration process in which the multiple operations are made into a macro and registered in association with the specified phrase.

[0031] <Macro registration process during browsing> There are two methods for this macro registration process, as shown in the upper part of Figure 2.

[0032] That is, the first method of macro registration processing, as shown in the upper left of Figure 2, involves, when a phrase that is not registered as a macro is spoken, associating multiple operations with the phrase and storing them as a macro.

[0033] In addition, the second method of macro registration processing, as shown in the upper right part of Figure 2, involves first converting a number of operations into macros, and then, when a phrase to be associated and stored is spoken, storing the multiple operations that have been converted into macros in association with that phrase.

[0034] (First method of macro registration process) For example, in the first method of the macro registration process, as shown in the upper left of Figure 2, in process SC1, when the voice agent 21 detects the phrase "XXX" uttered by the user 11 via the voice input device 20, it determines whether or not the phrase is a macro registered phrase.

[0035] If the phrase is not a registered macro, in process SC2, the voice agent 21 presents information such as "Shall I memorize XXX?" through images and sounds from a display or speaker (not shown) of the PC 13, thereby inquiring as to whether or not a plurality of operations should be registered as a macro in association with the phrase "XXX."

[0036] In response to this inquiry, when it is detected in process SC3 that the phrase "OK" has been uttered by the user 11 via the voice input device 20, the voice agent 21 recognizes that there has been a request to associate "XXX" with a macro of multiple operations and register it.

[0037] Next, in process SC4, the voice agent 21 notifies the user 11 that it is ready to sequentially store the operations of the operation input device 12 by the user 11 thereafter, by issuing an image or sound, for example, "Please operate," from a display or speaker (not shown) of the PC 13.

[0038] In response to this, in step SC5, the user 11 operates the operation input device 12 to input a plurality of operations to be registered as macros, and the voice agent 21 stores the inputted plurality of operations in sequence.

[0039] When the multiple operation inputs are completed, in step SC6, the user 11 utters a phrase, for example, "register," and the voice agent 21 recognizes that the multiple operation inputs to be registered as a macro have been completed.

[0040] Then, in process SC7, the voice agent 21 registers the phrase "XXX" as a macro in association with the multiple operations that have been memorized up to that point, and notifies the user 11 that the multiple operation inputs that have been input up to that point have been converted into a macro and registered in association with the phrase "XXX" by issuing an image or sound, for example, "I have memorized XXX," from a display or speaker (not shown) of the PC 13.

[0041] Through this series of processes, multiple operation inputs are registered as macros in association with the phrase "XXX."

[0042] As a result, thereafter, when the user 11 utters "XXX", as shown in the lower part of FIG. 2, the voice agent 21 assumes that the user has instructed to execute the macro registered in association with "XXX", and executes the multiple operation inputs registered in association with it all at once.

[0043] (Second method of macro registration process) For example, in the second technique of the macro registration process, as shown in the upper right part of FIG. 2, when the user 11 wants to register a plurality of operations as a macro for a given phrase, he or she utters "remember."

[0044] In process SC11, when the voice agent 21 detects the phrase "remember" uttered by the user 11 via the voice input device 20, it recognizes that it is desired to register a plurality of operations for a specific phrase as a macro.

[0045] Therefore, in process SC12, the voice agent 21 notifies the user 11 that the operation contents of the operation input device 12 by the user 11 from now on are ready to be sequentially stored by uttering, for example, "Please operate", through images and sounds from a display or speaker (not shown) of the PC 13.

[0046] In response to this, the user 11 operates the operation input device 12 to input a plurality of operations to be registered as macros, and in step SC13 the voice agent stores the inputted plurality of operation inputs.

[0047] When the user 11 has completed the multiple operation inputs to be registered as a macro, the user 11 notifies the voice agent 21 that the multiple operation inputs have been completed by uttering the phrase "register."

[0048] As a result, in process SC14, the voice agent 21 detects the phrase "register" uttered by the user 11 via the voice input device 20, and thereby recognizes the completion of the multiple operational inputs to be registered as a macro.

[0049] Furthermore, in process SC15, the voice agent 21 inquires about a phrase to be stored in association with the multiple operations to be registered as macros input by the user 11 operating the operation input device 12, for example, by uttering, through images and voice from a display and speaker (not shown) of the PC 13, "What would make you perform this operation?"

[0050] In response to this, the user 11 instructs to associate the plurality of operations stored up to that point with the phrase "XXX", for example, and to register them as macros.

[0051] In process SC16, when the voice agent 21 detects the phrase "XXX" uttered by the user 11 via the voice input device 20, it recognizes that there has been an instruction to make a macro of the multiple operations that have been stored up to that point and register it in association with the phrase "XXX."

[0052] In process SC17, the voice agent 21 registers the multiple operations stored up to that point as macros, associating them with the phrase "XXX." The voice agent 21 then notifies the user 11 that the multiple operation inputs input up to that point have been registered as macros, associating them with the phrase "XXX," by uttering, for example, "XXX has been memorized" as an image or sound from a display or speaker (not shown) of the PC 13.

[0053] Through this series of processes, multiple operation inputs are registered as macros in association with the phrase "XXX."

[0054] Even after registration in this manner, as shown in the lower part of FIG. 2, when user 11 utters the phrase "XXX," the corresponding and registered macro operations are executed all at once.

[0055] However, in both the first and second methods of the macro registration process described above, the premise is that the user has memorized multiple operations, and the user can only register the multiple memorized operations in association with a phrase.

[0056] In other words, unless the user is aware and aware of what he or she habitually does during normal browsing, the user will not be able to register a macro.

[0057] Furthermore, even if a user is able to recognize and understand their own habitual operations and register them as macros, if they forget the phrase that was registered as a macro, they may not be able to utter the registered phrase to execute multiple macro operations all at once.

[0058] In view of this, it is conceivable that a plurality of habitual operations of a user can be estimated as a cluster from the operation history and made known to the user, thereby allowing the user to execute the plurality of operations collectively while reducing the burden on the user.

[0059] <Clustering of user operation history> In other words, it is possible to generate habitual operation clusters by clustering multiple highly habitual operations based on the user's operation history, execute voice macro processing on a unit basis of the habitual operation cluster, suggest macro creation through a situation suggestion notification, and execute habitual operation clusters that include keywords that match the user's spoken keywords more closely than a predetermined degree of match all at once.

[0060] For example, a user operation history is recorded and presented as shown in the upper left part of FIG. 3, and for a number of operations that are highly habitual among the history, habitual operation clusters are set as shown in the upper right part of FIG. 3.

[0061] In addition, the user operation history in the upper left of Figure 3 shows, from the top, that operation A was performed at 20:40:13 on 3 / 18, operation B was performed at 20:40:14 on 3 / 18, operation C was performed at 20:40:16 on 3 / 18, and operation D was performed at 20:40:27 on 3 / 18.

[0062] It also shows that operation E was performed at 20:40:32 on 3 / 18, operation F was performed at 20:40:35 on 3 / 18, operation G was performed at 20:40:39 on 3 / 18, operation H was performed at 20:40:39 on 3 / 18, and operation I was performed at 20:40:41 on 3 / 18.

[0063] It further shows that operation J was performed at 20:40:44 on 3 / 18, operation K was performed at 20:40:46 on 3 / 18, and operation L was performed at 20:40:49 on 3 / 18.

[0064] In the user operation history shown in the upper left of Figure 3, which is a part of the user operation history, a habitual score indicating the degree of habituality of each operation is calculated for each user operation history, and two or more operations whose habitual score is higher than a predetermined value are clustered and set as a habitual operation cluster as shown in the upper right of Figure 3.

[0065] The upper right part of FIG. 3 is an example of an addictive operation cluster consisting of two or more operations whose addictive scores are higher than a predetermined value.

[0066] In the upper right part of FIG. 3, operations F to I are two or more operations whose habitual scores are higher than a predetermined value, and therefore are clustered as an habitual operation cluster.

[0067] In addition, in the addictive operation cluster, an operation E immediately before two or more operations whose addictive scores are higher than a predetermined value is also included in the addictive operation cluster.

[0068] For this reason, in the upper right part of FIG. 3, operations E to I are included as an addictive operation cluster.

[0069] For the addictive operation cluster in the upper right part of FIG. 3, score display fields LV1 to LV5 indicating the level of the selected addictive score are displayed to the left of each of operations E to L.

[0070] In each of the score display fields LV1 to LV5, the higher the habitual score, the further to the right in the figure the bar graph extends, and the habitual score is displayed in a different color.

[0071] That is, in the upper right part of FIG. 3, it is shown that the addictive scores in the score display columns LV3 and LV4 for operations G and H, respectively, are the highest, and the addictive scores in the score display columns LV2 and LV5 for operations F and I, respectively, are next highest.

[0072] In the upper right portion of FIG. 3, score display fields LV2 to LV5 for operations having habit scores higher than a predetermined threshold are displayed between brackets BL1 and BL2.

[0073] The display of brackets BL1 and BL2 makes it possible to visually confirm that there are operations F to I consisting of two or more operations whose habitual scores are higher than a predetermined value, and that together with the operation E immediately preceding them, an habitual operation cluster is formed.

[0074] Furthermore, in operations E to I, switches SSE to SSI are provided to the left of the score display fields LV1 to LV5, respectively, and by operating each of them, the operation to be registered when making it into a macro can be selected.

[0075] That is, as shown in the upper left and upper right parts of Figure 3, habitual operation clusters are set based on the user operation history, and then multiple highly habitual operations are converted into macros, and then phrases are associated with them and stored, thereby performing a macro registration process and realizing voice macroization (voice macroization in the lower part of Figure 3).

[0076] This allows the user to execute multiple operations that are highly addictive to the user all at once by speaking the registered phrase.

[0077] In addition, as shown in the lower part of Figure 3, by presenting an addictive operation cluster consisting of multiple operations whose addictive score is higher than a predetermined value, the user is presented with an addictive situation consisting of multiple consecutive operations, and a situation suggestion notification is presented to determine whether or not to create a macro (situation suggestion notification at the bottom of Figure 3).

[0078] This enables the user to be aware of his / her own habitual successive operations, and when the user realizes that the habitual score has reached a certain level, the user can turn the consecutive operations with a high habitual score into a macro and register the macro.

[0079] Furthermore, as shown in the lower part of Figure 3, addictive operation clusters that include the keywords spoken by the user are searched for and presented (keyword search in the lower part of Figure 3).

[0080] As a result, even if a phrase associated with a habitual operation cluster and registered as a macro is forgotten, habitual operation clusters searched for based on related phrases are presented and a suggestion for executing them all at once is given, so that the habitual operation cluster registered as a macro can be utilized.

[0081] However, even after the addictive operation cluster is registered as a macro, the addictive operation may change.

[0082] In such cases, when using the above-described method, even if there is a change in the habitual operation, it takes time for it to be recognized as a new habitual operation, so there is a risk that it will take a long time for a habitual operation cluster corresponding to the new habitual operation to be generated.

[0083] Therefore, in the present disclosure, habitual operations are estimated from the user's past operation history, and a dialogue with the user is realized at the timing when an operation that needs to be changed among multiple macronized operations is performed, making it possible to easily change the operation in accordance with the dialogue.

[0084] <<2. Preferred embodiment>> Next, preferred embodiments of the present disclosure will be described.

[0085] FIG. 4 shows a preferred embodiment of an information processing device to which the technology of the present disclosure is applied.

[0086] The information processing device 41 in Figure 4 includes an input operation unit 51, a functional application program execution unit 52, an operation memory execution control unit 53, a user operation history memory unit 54, a habitual operation / dialogue cluster memory unit 55, a voice input unit 56, a voice recognition processing unit 57, a speech meaning understanding processing unit 58, an image input unit 59, an image recognition processing unit 60, a sensor input unit 61, a sensor recognition processing unit 62, a response generation unit 63, a display image processing unit 64, a voice synthesis processing unit 65, an image output unit 66, and a voice output unit 67.

[0087] The input operation unit 51 is composed of a keyboard, mouse, touch panel (input operation device), etc., and accepts user operation input, generates an operation signal corresponding to the accepted operation input, and outputs it to the function application program execution unit 52.

[0088] The functional application program execution unit 52 executes application programs that realize various functions, such as presenting content supplied from a web browser or an accessed site, based on the operation signal supplied from the input operation unit 51 and the system operation instructions supplied from the operation memory execution control unit 53.

[0089] More specifically, the functional application program execution unit 52 realizes functions as a general device equipped with multiple operating functions, such as games (controllers), televisions (remote controllers), music / video players, smartphones / tablets, PCs (Personal Computers), AR (Augmented Reality) / VR (Virtual Reality) devices, home appliances, cameras / camcorders, commercial systems, and other devices.

[0090] In the remainder of this specification, we will explain an embodiment in which the functional application program execution unit 52 functions as a web browser, but it may naturally be configured to realize the functions of other devices.

[0091] In addition, the function application program execution unit 52 outputs to the image output unit 66 images consisting of moving images and still images that constitute the execution results based on the operation signal supplied from the input operation unit 51 and the system operation instructions supplied from the operation memory execution control unit 53.

[0092] Furthermore, the function application program execution unit 52 outputs to the audio output unit 67 a voice constituting the execution result based on the operation signal supplied from the input operation unit 51 and the system operation instruction supplied from the operation memory execution control unit 53 .

[0093] The voice input unit 56 is composed of a microphone and the like, and receives input of a speech phrase consisting of a voice uttered by the user, generates a corresponding voice signal, and outputs it to the voice recognition processing unit 57 .

[0094] The voice recognition processing unit 57 performs voice recognition processing (ASR: Automatic Speech Recognition) based on the voice signal supplied from the voice input unit 56, and outputs the voice recognition result consisting of text to the utterance meaning understanding processing unit 58 and the operation memory execution control unit 53.

[0095] The speech semantic understanding processing unit 58 performs speech semantic understanding processing (NLU: Natural Language Understanding) on ​​the speech recognition result consisting of text supplied from the speech recognition processing unit 57, and estimates the speech intention (operation type) (Intent) of the user's spoken phrase and the attribute information (Entity) that is the subject of the speech.

[0096] The utterance meaning understanding processor 58 outputs the utterance intention (operation type) (Intent) of the user's utterance phrase, which is the estimation result, and attribute information (Entity) of the target of the utterance to the operation memory execution controller 53.

[0097] The image input unit 59 is composed of a camera or the like, and receives input of facial expressions representing the user's situation and images of the surroundings, generates corresponding image signals, and outputs them to the image recognition processing unit 60 .

[0098] The image recognition processing unit 60 performs image recognition processing on the image signal supplied from the image input unit 59, and outputs the processing result to the operation memory execution control unit 53 as observation context information representing the user's situation.

[0099] The sensor input unit 61 is composed of a GPS (Global Positioning System) sensor, a pulse sensor, a blood pressure sensor, and the like, and receives input of the sensing results of each sensor, generates a corresponding sensor signal, and outputs it to a sensor recognition processing unit 62.

[0100] The sensor recognition processing unit 62 performs sensor recognition processing on the sensor signal supplied from the sensor input unit 61, and outputs the processing result to the operation memory execution control unit 53 as observation context information representing the user's situation.

[0101] The operation memory execution control unit 53 assigns a timestamp to the user operation information supplied from the function application program execution unit 52 and stores it in the user operation history memory unit 54 as a user operation history, so that it is accumulated as the user operation history.

[0102] The operation memory execution control unit 53 may generate a specific history management screen or the like using the display image processing unit 64 via the response generation unit 63 to display the user operation history stored in the user operation history memory unit 54 in chronological order by the image output unit 66 as the past user operation history described with reference to the upper left corner of Figure 3, so that the user can view it.

[0103] The user operation information is information corresponding to the operation content of the input operation unit 51. For example, if the function realized by the function application program execution unit 52 is a web browser, the information corresponds to operations such as clicking, form input, form submit, and site transition.

[0104] Furthermore, the user operation information also includes information on UI (User Interface) parts such as links and forms that are the targets of the operations, and destination URLs (Uniform Resource Locators) (+site title).

[0105] The user operation information also includes information on the URL (+site title) of the site at the time when the operation was performed by the input operation unit 51.

[0106] The operation memory execution control unit 53 estimates the user's habitual multiple operations (composite operations) as habitual operation clusters or dialogue clusters based on the past user operation history accumulated in the user operation history memory unit 54, and stores them in the habitual operation / dialogue cluster memory unit 55.

[0107] The operation memory execution control unit 53 reads out a corresponding habitual operation cluster or dialogue cluster from the habitual operation / dialogue cluster memory unit 55 based on the user's spoken phrase, its speech intention (operation type) (Intent), the attribute information (Entity) of the subject of the speech, the observation context information, and the user operation information.

[0108] Then, the operation memory execution control unit 53 generates system operation instructions by converting the read habitual operation cluster or dialogue cluster into a voice macro, or by executing a plurality of operations at once or by executing a macro based on the habitual operation cluster or dialogue cluster, and outputs the instructions to the function application program execution unit 52 for execution.

[0109] The functional application program executing section 52 executes a plurality of operations constituting the habitual operation cluster or the interaction cluster collectively in time series for each of them, based on a system operation instruction corresponding to the habitual operation cluster or the interaction cluster.

[0110] In addition, the operation memory execution control unit 53 controls the response generation unit 63 to generate and output display images and sounds for suggestion notifications and confirmation dialogues regarding the execution of voice macros, habitual operation clusters, or dialogue clusters to the user, in order to provide the user with a suggestion notification and confirmation dialogue regarding the execution of voice macros, habitual operation clusters, or dialogue clusters in bulk.

[0111] The response generation unit 63 causes the display image processing unit 64 to generate a display image for a suggestion notification or a confirmation dialogue related to voice macro generation or the execution of a habitual operation cluster or a dialogue cluster, and also causes the voice synthesis processing unit 65 to generate a voice for similar suggestion notification or a confirmation dialogue.

[0112] The display image processing unit 64 outputs a display image for a suggestion notice or a confirmation dialogue related to the generated voice macro or execution of the habitual operation cluster or dialogue cluster to the image output unit 66 for display.

[0113] The voice synthesis processing unit 65 causes the voice output unit 67 to output voice for suggestion notification or confirmation dialogue relating to voice macro generation or execution of habitual operation clusters or dialogue clusters.

[0114] The image output unit 66 is composed of a display, a projector, etc., and is controlled by the function application program execution unit 52 to display a display image that is the processing result of the function application program execution unit 52 .

[0115] Moreover, the image output unit 66 displays the display image supplied from the display image processing unit 64 .

[0116] The audio output unit 67 is composed of a speaker, headphones, etc., and is controlled by the function application program executing unit 52 to output audio that is the processing result of the function application program executing unit 52 .

[0117] Moreover, the voice output unit 67 outputs the voice supplied from the voice synthesis processing unit 65 .

[0118] <User operation information and corresponding habit score> Next, with reference to FIG. 5, the user operation information and the corresponding habit score will be described.

[0119] The user operation information is stored as information such as user operation information Pr1 to Pr3 in FIG.

[0120] That is, the user operation information Pr1 in Figure 5 is composed of the descriptions in lines 1 to 12, and lines 2 to 6 indicate that an operation has been performed to display the transfer guide homepage at the URL indicated by "url": "https: / / www.JJJ.co.jp / norikae / " and "title": "Transfer Guide|JJJ".

[0121] Furthermore, lines 7 to 10 of the user operation information Pr1 in FIG. 5 indicate that information stored at the location on the network indicated by "url": "https: / / www.tootle.co.jp / search?q=%E4%B9%97%E3%82%8A%E6%8F%9B%E3%81%88" and "title": "Transfer - Tootle Search" is being read out and displayed.

[0122] Furthermore, the 11th line of the user operation information Pr1 in Figure 5 indicates that the operation of the user operation information ST1 was performed at 05:04:38.896 on April 5, 2019, as indicated by "date": "2019-04-05T05:04:38.896Z".

[0123] The user operation information Pr2 describes information on the operation of inputting the name of the boarding station into the displayed transfer guide homepage by operating this user operation information Pr1.

[0124] That is, the user operation information Pr2 in FIG. 5 is composed of the descriptions in the 1st to 15th lines, and the 2nd to 9th lines indicate that an operation has been performed to input Shinjuku as the boarding station.

[0125] In addition, the 10th to 13th lines of the user operation information Pr2 in Figure 5 indicate that the information stored at the location on the network indicated by "url": "https: / / www.JJJ.co.jp / norikae / ", "title": "Transfer Guide|JJJ" is being read out and displayed.

[0126] Furthermore, the 14th line of user operation information Pr2 in Figure 5 indicates that the operation of user operation information ST2 was performed on April 5, 2019 at 05:04:46.341, as indicated by "date": "2019-04-05T05:04:46.341Z".

[0127] Furthermore, information on the operation of inputting the name of the station where the user gets off into the displayed transfer guide homepage by operating this user operation information Pr1 is described in user operation information Pr3.

[0128] That is, the user operation information Pr3 in FIG. 5 is composed of the descriptions in the 1st to 15th lines, and the 2nd to 9th lines indicate that an operation to input Osaki as the disembarking station has been performed.

[0129] In addition, the 10th to 13th lines of the user operation information Pr3 in Figure 5 indicate that the information stored at the location on the network indicated by "url": "https: / / www.JJJ.co.jp / norikae / ", "title": "Transfer Guide|JJJ" is being read out and displayed.

[0130] Furthermore, the 14th line of user operation information Pr3 in Figure 5 indicates that the operation of user operation information ST3 was performed at 05:04:57.721 on April 5, 2019, as indicated by "date": "2019-04-05T05:04:57.721Z".

[0131] That is, the user operation information Pr1 to Pr3 in FIG. 5 is sequentially stored in the user operation history storage unit 54 in chronological order.

[0132] When the operation storage execution control unit 53 receives the user operation information, the operation storage execution control unit 53 sequentially stores the information in the user operation history storage unit 54 to form a user operation history.

[0133] In addition, when user operation information is supplied to the operation memory execution control unit 53, it calculates a habitual score using the user operation history stored in the user operation history memory unit 54, stores the habitual score in correspondence with the user operation history, and updates it sequentially.

[0134] Here, the habit score is a value indicating the habit tendency of an operation n corresponding to the nth operation in the user operation history, for example, in the past user operation history, and is a score defined by the following formula (1).

[0135] S n =3 / (1 / T n +1 / F n +1 / P n ) (1)

[0136] Here, S n is the addictive score of operation n, and T n is the time interval from the previous operation n-1 to operation n, and F n is the number of transitions from operation n-1 to operation n, and P n is the transition probability from operation n-1 to operation n.

[0137] That is, the habit score of operation n is calculated by the time interval T n , the number of transitions from operation n-1 to operation n, F n , and the transition probability P from operation n-1 to operation n n is the harmonic mean of

[0138] In addition, the time interval T from the previous operation n-1 to operation n n is defined by the following equation (2).

[0139] T n =Ht / ((t n -t (n-1) )+H t ) (2)

[0140] Here, T n is the time interval from the previous operation n-1 to operation n, and t n is the time when operation n was performed, and t (n-1) is the time when operation n-1 was performed, and H t is a normalization constant. The normalization constant H t may be, for example, 30 seconds.

[0141] In addition, the normalization constant H t may be made variable by the user. n is the normalization constant H c For this reason, for example, people who are familiar with the operation and those who are not can be judged from the average time of use of the device and the operation time interval, and those who are familiar with the device can be judged from the average time of use of the device and the operation time interval. t For those who are not familiar with it, the normalization constant H t may be increased.

[0142] Furthermore, the number of transitions from operation n-1 to operation n, F n is defined by the following equation (3).

[0143] F n =1-H c / (C tn +H c ) (3)

[0144] Here, C tn is the number of times that the operation n-1 has transitioned to the operation n in the user operation history, and H c is a normalization constant. The normalization constant H c For example, three times.

[0145] In addition, the transition probability P nis defined by the following equation (4).

[0146] P n =C tn / C (n-1) (4)

[0147] Here, C (n-1) is the number of times the same operation as operation n-1 has been performed in the past in the user operation history.

[0148] That is, the addictive score S of operation n n is the time when operation n is performed within a short time period from the previous operation n-1, and the number of transitions is F n and the transition probability P n The higher the score, the more likely it is that the activity has become habitual.

[0149] When the operation memory execution control unit 53 receives user operation information and stores it in the user operation history in the user operation history storage unit 54, the operation memory execution control unit 53 sequentially stores the habit score S n is calculated and stored for each operation accumulated in the user operation history, and is updated as necessary.

[0150] More specifically, the operation memory execution control unit 53 calculates the habit score S on the time series of the operation history. n is the habit threshold S th When two or more operations are performed that are larger than (for example, 0.5), the habit score S n is the habit threshold S th The last habit score S from the operation before the operation that is larger n is the habit threshold S th The operations up to and including the larger one are set as candidates for the habitual operation cluster.

[0151] Next, if there is no habitual operation cluster with the same sequence (matching the order of operations in the habitual operation cluster) as the habitual operation cluster candidate among the habitual operation clusters stored in the habitual operation / interaction cluster memory unit 55, and there is also no habitual operation cluster with the same operations but a different order, the operation memory execution control unit 53 stores the habitual operation cluster candidate in the habitual operation / interaction cluster memory unit 55 as a new habitual operation cluster.

[0152] Furthermore, when an addictive operation cluster having the same operations as an addictive operation cluster candidate but a different order exists within the addictive operation cluster stored in the addictive operation / interaction cluster storage unit 55, the operation memory execution control unit 53 compares the two addictive operation cluster candidates with the addictive operation cluster, detects order-variable operations within the addictive operation cluster, and integrates them to update and store them in the addictive operation / interaction cluster storage unit 55 as an addictive operation cluster.

[0153] In addition, the operation memory execution control unit 53 calculates the habit score S on the time series of the operation history. n is the habit threshold S th When two or more operations are performed that are larger than (for example, 0.5), the habit score S n is the habit threshold S th The last habit score S from the operation before the operation that is larger n is the habit threshold S th The operations up to and including the larger one are set as candidates for the dialogue cluster.

[0154] In addition, if there are no clusters contained in the sequence of the dialogue cluster candidate among the habitual operation clusters and dialogue clusters (hereinafter, the habitual operation clusters and dialogue clusters will also be simply referred to as clusters) stored in the habitual operation / dialogue cluster memory unit 55, and there are no clusters that can be replaced by dialogue, the operation memory execution control unit 53 stores the dialogue cluster candidate in the habitual operation / dialogue cluster memory unit 55 as a new dialogue cluster.

[0155] <Addiction operation clustering processing> Next, the addictive operation clustering process will be described with reference to the flowchart of FIG.

[0156] In step S11, the operation memory execution control unit 53 initializes a counter n, which counts operations, to 1.

[0157] In step S12, the operation memory execution control unit 53 determines whether or not the user operation information of the operation n has been supplied, and repeats the same process until it is supplied.

[0158] Then, in step S12, if user operation information of operation n is supplied, the process proceeds to step S13.

[0159] In step S13, the operation storage execution control unit 53 adds a time stamp to the user operation information of the operation n, and stores the information in the user operation history storage unit 54 as a user operation history.

[0160] In step S14, the operation memory execution control unit 53 calculates the habit score S of the operation n by the calculation described with reference to the above-mentioned formulas (1) to (4) based on the user operation history stored in the user operation history storage unit 54. n Calculate.

[0161] In step S15, the operation memory execution control unit 53 calculates the habit score S corresponding to the operation n. n However, the habit threshold S th Determine whether it is greater than

[0162] In step S15, the habit score S corresponding to the operation n is calculated. n However, the habit threshold S th If it is determined that the operation n is greater than n, the operation n is deemed to be a highly habitual operation, and the process proceeds to step S16.

[0163] In step S16, the operation memory execution control unit 53 calculates the habit score S of the operation n-1 immediately before the operation n. n-1 However, the habit threshold S thDetermine whether it is smaller than

[0164] In step S16, the habit score S of the operation n-1 immediately before the operation n is calculated. n-1 However, the habit threshold S th If it is smaller, the process proceeds to step S17.

[0165] In step S17, the operation memory execution control unit 53 adds the operation n-1 to the addictive operation cluster candidates.

[0166] In step S18, the operation memory execution control unit 53 adds the operation n to the addictive operation cluster candidates.

[0167] In step S19, the operation memory execution control unit 53 determines whether or not an instruction to end the process has been given, and if an instruction to end the process has been given, the process ends.

[0168] Moreover, in step S19, if an instruction to end the process has not been given, the process proceeds to step S20.

[0169] In step S20, the operation memory execution control unit 53 increments the counter n by 1, and the process returns to step S12.

[0170] That is, until the end of the habitual operation clustering process is instructed, the habitual score S n is the habit threshold S th When there are a plurality of operations n higher than the number of operations (repeated), the processes of steps S11 to S20 are repeated, the counter n is sequentially incremented by 1, and operations of the habitual operation cluster candidate are continuously added.

[0171] In addition, the first operation in the addictive operation cluster has an addictive score S n is the habit threshold S th Since the habit score S of the operation n-1 immediately before the operation n is higher than the habit score S n-1 is the habit threshold S thIf it is lower, the process in step S17 causes the operation n-1 immediately preceding the operation n to be included as the leading operation of the habitual operation cluster candidate.

[0172] Therefore, among the operations that constitute the candidate addictive operation cluster, the operations other than the first one have an addictive score S n is the habit threshold S th Therefore, in step S16, the habit score S n is the habit threshold S th Since the value of the first block is not smaller than 1, the process of step S17 is skipped.

[0173] Furthermore, in step S15, the habit score S corresponding to the operation n is calculated. n However, the habit threshold S th If it is determined that the operation n is not larger than n, the operation n is not considered to be a highly habitual operation, and the process proceeds to step S21.

[0174] In step S21, the operation memory execution control unit 53 calculates the habit score S of the immediately previous operation n-1. n-1 is the habit threshold S th Determine whether it is greater than

[0175] In step S21, the habit score S of the operation n-1 is n-1 However, the habit threshold S th If it is determined that the difference is larger than the previous operation, the operation up to that point is determined to be part of the operation included in the habitual operation cluster candidate, and the process proceeds to step S22.

[0176] In step S21, the habit score S of the operation n-1 is n-1 However, the habit threshold S th If it is determined that the difference is smaller than 0, the immediately preceding operation is determined to be an operation that is not included in the habitual operation cluster candidates, and the process proceeds to step S19.

[0177] In step S22, the operation memory execution control unit 53 determines whether or not the number of operations included in the addictive operation cluster candidate formed by the operations up until that point is two or less.

[0178] Since an habitual operation cluster is assumed to be three or more, if the number of operations included in the habitual operation cluster candidate formed by the operations up to the immediately preceding operation is two or less in step S22, it is deemed that an habitual operation cluster has not been formed, and processing proceeds to step S23.

[0179] Then, in step S23, the operation memory execution control unit 53 clears the information on the operations stored as the habitual operation cluster candidates, and the process proceeds to step S19.

[0180] Also, in step S22, if the habitual operation cluster candidate formed by the immediately preceding operations does not include two or fewer operations, that is, it is deemed that the habitual operation cluster candidate includes three or more operations, and the process proceeds to step S24.

[0181] In step S24, the operation memory execution control unit 53 determines whether or not there are any habitual operation clusters already registered and stored in the habitual operation / dialogue cluster memory unit 55 that are completely identical in that the operations contained in the habitual operation cluster candidate currently being processed are the same and have the same sequence (the order is the same).

[0182] In other words, if the habitual operation cluster candidate is completely identical to an habitual operation cluster already registered in the habitual operation / dialogue cluster storage unit 55, there is no need to register it anew, so it is determined whether or not there is an identical candidate.

[0183] In step S24, if the already registered habitual operation cluster stored in the habitual operation / dialogue cluster storage unit 55 is completely identical to the habitual operation cluster candidate currently being processed, the process proceeds to step S23.

[0184] That is, in this case, since the habitual operation cluster already registered and stored in the habitual operation / interaction cluster memory unit 55 is completely identical to the habitual operation cluster candidate currently being processed, the process proceeds to step S23, and the information of the habitual operation cluster candidate currently being processed is cleared.

[0185] Also, in step S24, if there is no habitual operation cluster that is completely identical to the habitual operation cluster candidate currently being processed among the habitual operation / dialogue clusters already registered and stored in the habitual operation / dialogue cluster storage unit 55, the process proceeds to step S25.

[0186] In step 25, the operation memory execution control unit 53 determines whether or not there are any habitual operation clusters already registered and stored in the habitual operation / dialogue cluster memory unit 55 that contain the same operations but not the same order in the habitual operation cluster candidate currently being processed, i.e., whether or not there are any that contain the same operations but differ only in order.

[0187] In step S25, if there is no habitual operation cluster candidate currently being processed among the already registered habitual operation clusters stored in the habitual operation / dialogue cluster memory unit 55 that contains the same operations but differs only in order, the process proceeds to step S26.

[0188] In step S26, the operation memory execution control unit 53 registers the operations of the habitual operation cluster candidate currently being processed in the habitual operation / dialogue cluster storage unit 55 in the same order as they are, as a new habitual operation cluster.

[0189] In other words, among the already registered habitual operation clusters stored in the habitual operation / dialogue cluster storage unit 55, there are none that are identical to the operations included in the habitual operation cluster candidate currently being processed and differ only in order, so the habitual operation cluster candidate currently being processed is regarded as a new habitual operation cluster consisting of a plurality of operations, and is therefore registered in the habitual operation / dialogue cluster storage unit 55.

[0190] In step S25, if there is an habitual operation cluster already registered and stored in the habitual operation / dialogue cluster memory unit 55 that has the same operations as those included in the habitual operation cluster candidate currently being processed but whose order is not the same, i.e., only the order is different, the process proceeds to step S27.

[0191] In step S27, the operation memory execution control unit 53 detects an order-variable operation by comparing the currently processed habitual operation cluster candidate with a registered habitual operation cluster that is the same as the operation included in the currently processed habitual operation cluster candidate but is not the same in order. Hereinafter, this is also referred to as detecting an order-variable operation.

[0192] In step S28, the operation memory execution control unit 53 integrates the currently processed habitual operation cluster candidate and a registered habitual operation cluster that is identical to the operations included in the currently processed habitual operation cluster candidate but has a different order, together with information on the variable range of the order-variable operations, into one habitual operation cluster, and registers it in the habitual operation / dialogue cluster memory unit 55.

[0193] The comparison and integration of addictive operation clusters will be described in detail later with reference to FIGS.

[0194] By the above process, when a new operation n is accepted, the habit score S n is required, and the habit threshold S th By comparing with the above, addictive operation cluster candidates are set.

[0195] In addition, by comparing the constituent operations of the habitual operation cluster candidate with those of already registered habitual operation clusters, any habitual operation cluster candidate consisting of operations completely different from those of the already registered habitual operation cluster is registered as a new habitual operation cluster.

[0196] Furthermore, by comparing the operations that constitute the habitual operation cluster candidate with an already registered habitual operation cluster, if the habitual operation cluster candidate is composed of the same operations but differs only in the operation sequence, it is integrated with the registered habitual operation cluster into a single habitual operation cluster and registered in the habitual operation / interaction cluster memory unit 55.

[0197] This makes it possible to sequentially register a plurality of operations that have a high habitual score and are highly habitual as user operations as an habitual operation cluster.

[0198] Incidentally, since there is a possibility that the habitual nature of a registered habitual operation cluster may deviate over time, habitual operation clusters that remain registered for a predetermined period of time or longer may be deleted.

[0199] <Comparison and integration of addictive operation clusters (part 1)> Now, with reference to FIG. 7, the comparison and integration of addictive operation clusters will be described.

[0200] Consider the case where, for example, there exist an habitual operation cluster candidate CL1 as shown in FIG. 7 and an habitual operation cluster CL2 which differs only in the stored order as habitual operation cluster candidates and registered habitual operation clusters.

[0201] Here, it is assumed that the habitual operation cluster candidate CL1 in FIG. 7 is composed of four operations, namely, operations A to D, and the habitual operation cluster CL2 is also composed of four operations, namely, operations A to D.

[0202] Here, in the habitual operation cluster candidate CL1, operations are performed in the order of operation A, operation C, operation B, and operation D, whereas in the registered habitual operation cluster, operations are performed in the order of operation A, operation B, operation C, and operation D.

[0203] In addition, operation A is an operation to transition to a transfer information site, operation C is an operation to input "Osaki" as the disembarking station, operation B is an operation to input "Shinjuku" as the boarding station, and operation D is an operation to click the search execution button.

[0204] In addition, in both the addictive operation cluster candidate CL1 and the addictive operation cluster CL2, the addictive scores S n Both are at the habit threshold S th shall be greater than

[0205] That is, when comparing the operations in the habitual operation cluster candidate CL1 and the habitual operation cluster CL2, the order of operation B, which is an input operation for the boarding station, and operation C, which is an input operation for the disembarking station, is different.

[0206] However, even if the order of operation B, which is the input operation of the boarding station, and operation C, which is the input operation of the disembarking station, is different, the same transfer guide will be displayed when the final operation D is performed, so habitual operation cluster candidate CL1 and habitual operation cluster CL2 are essentially identical, and it is desirable to treat them together as the same habitual operation cluster.

[0207] Therefore, the operation memory execution control unit 53 compares the operations before and after each of the operations constituting the habitual operation cluster candidate CL1 and the habitual operation cluster CL2, as shown by the comparison result CL3 in FIG.

[0208] That is, in the case of operation A of the habitual operation cluster candidate CL1, since operation A is the leading operation, there is no previous operation (denoted as leading in the figure), and operation C is the following operation.

[0209] In operation C of the habitual operation cluster candidate CL1, the previous operation is operation A and the subsequent operation is operation B.

[0210] In operation B of the habitual operation cluster candidate CL1, the previous operation is operation C and the subsequent operation is operation D.

[0211] In addition, in the case of operation D of the habitual operation cluster candidate CL1, the previous operation is operation B, operation D itself is a terminal operation, and there is no subsequent operation (the subsequent operation is written as a terminal operation in the figure).

[0212] In addition, in the case of operation A in the habitual operation cluster CL2, since operation A is the leading operation, there is no previous operation (denoted as leading in the figure), and operation B is the following operation.

[0213] In operation C of addictive operation cluster CL2, the previous operation is operation B and the subsequent operation is operation D.

[0214] In operation B of addictive operation cluster CL2, the previous operation is operation A and the subsequent operation is operation C.

[0215] In addition, in the case of operation D in the habitual operation cluster CL2, the previous operation is operation C, operation D itself is a terminal operation, and there is no subsequent operation (the subsequent operation is written as a terminal operation in the figure).

[0216] Here, the operation memory execution control unit 53 detects, based on the comparison result CL3 of FIG. 7, an operation in which both the preceding operation and the succeeding operation are different between the habitual operation cluster candidate CL1 and the habitual operation cluster CL2 as a sequence variable operation.

[0217] That is, in the comparison result CL3 in FIG. 7, among the operations, operation B and operation C have different previous operations and subsequent operations, and therefore are detected as order-variable operations.

[0218] In addition, the operation A is the leading operation in both the habitual operation cluster candidate CL1 and the habitual operation cluster CL2, and the operation D is the terminal operation in both the habitual operation cluster candidate CL1 and the habitual operation cluster CL2. Therefore, neither the operation A nor the operation D is a sequence-variable operation.

[0219] Therefore, the operation memory execution control unit 53 generates an integrated memory cluster CL4 as shown in Figure 7 by integrating the habitual operation cluster candidate CL1 and the habitual operation cluster CL2 based on the detected order-variable operation, and registers it in the habitual operation / dialogue cluster memory unit 55.

[0220] That is, in integrated memory cluster CL4 in Figure 7, it is recorded that, from the left, operation A is an operation of transitioning to a transfer information site, operation C is an operation of inputting "Osaki" as the disembarking station, operation B is an operation of inputting "Shinjuku" as the boarding station, and operation D is an operation of clicking the search execution button.

[0221] Furthermore, among these, operation C and operation B are detected as order-variable operations, and therefore a variable range is set for them.

[0222] The variable range has a pre-condition and a post-condition, and the earliest operation that can be a pre-operation of an operation is set as the pre-condition, and the latest operation that can be a post-operation of the operation is set as the post-condition.

[0223] In integrated memory cluster CL4 in Figure 7, operations C and B are detected as order-variable operations, so the pre-condition of both operations C and B is operation A and the post-condition is operation D, as indicated by the arrows in the figure.

[0224] In other words, the information in integrated storage cluster CL4 in Figure 7 records that the order of both operation C and operation B can be changed if they are in the range between operations A and D, i.e., the variable range.

[0225] Incidentally, in the integrated memory cluster CL4 in FIG. 7, an example is shown in which the most recently performed operation sequence is registered, but the operation sequence of the registered habitual operation cluster may be registered.

[0226] <Comparison and integration of addictive operation clusters (part 2)> In the above, we have explained the case where, in comparing each operation between a candidate habitual operation cluster and a registered habitual operation cluster, there are two operations whose previous and subsequent operations are different, and a variable range is set for each operation.

[0227] However, the number of reorderable operations may be a number other than two, for example, it may be only one.

[0228] For example, consider a case where, as habitual operation cluster candidates and registered habitual operation clusters, there exist an habitual operation cluster candidate CL11 as shown in FIG. 8 and an habitual operation cluster CL12 which differs only in the stored order.

[0229] Here, it is assumed that the habitual operation cluster candidate CL11 in FIG. 8 is composed of four operations, namely, operations A to D, and the habitual operation cluster CL12 is also composed of the same four operations, namely, operations A to D.

[0230] Here, in the habitual operation cluster candidate CL11, operations are performed in the order of operation A, operation D, operation C, and operation B, whereas in the registered habitual operation cluster, operations are performed in the order of operation A, operation B, operation C, and operation D.

[0231] In addition, operation A is an operation of transitioning to a video viewing site, operation D is an operation of setting the volume to 30, operation B is an operation of selecting video content Z, and operation C is an operation of clicking the play button.

[0232] In addition, in both the addictive operation cluster candidate CL11 and the addictive operation cluster CL12, the addictive scores S n Both are at the habit threshold S th Greater than.

[0233] That is, when comparing the operations in the habitual operation cluster candidate CL11 and the habitual operation cluster CL12, it is found that operation D, which is an operation for setting the volume to 30, is an order-variable operation between operations A and B.

[0234] However, for operation D, which is an operation that sets the volume to 30, even though the order is different, the volume will be set to 30, so habitual operation cluster candidate CL11 and habitual operation cluster CL12 are substantially identical, and it is desirable to treat them together as the same habitual operation cluster.

[0235] Therefore, the operation memory execution control unit 53 compares the operations before and after each of the operations constituting the habitual operation cluster candidate CL11 and the habitual operation cluster CL12, as shown by the comparison result CL13 in FIG.

[0236] That is, in the case of operation A of the habitual operation cluster candidate CL11, since operation A is the leading operation, there is no previous operation (denoted as leading in the figure), and operation D is the following operation.

[0237] In operation D of the habitual operation cluster candidate CL11, the previous operation is operation A and the subsequent operation is operation B.

[0238] In operation B of the habitual operation cluster candidate CL11, the previous operation is operation D and the subsequent operation is operation C.

[0239] In addition, in the case of operation C of the habitual operation cluster candidate CL11, the previous operation is operation B, operation C itself is a terminal operation, and there is no subsequent operation (the subsequent operation is written as a terminal operation in the figure).

[0240] In addition, in the case of operation A in the habitual operation cluster CL12, since operation A is the leading operation, there is no previous operation (indicated as leading in the figure), and operation B is the following operation.

[0241] In the case of operation D in the habitual operation cluster CL12, the previous operation is operation C, operation D itself is a terminal operation, and there is no subsequent operation (the subsequent operation is written as a terminal in the figure).

[0242] In operation B of addictive operation cluster CL12, the previous operation is operation A and the subsequent operation is operation C.

[0243] In addition, in the operation C of the addictive operation cluster CL12, the previous operation is the operation B and the subsequent operation is the operation D.

[0244] Here, the operation memory execution control unit 53 detects, based on the comparison result CL13 of FIG. 8, an operation in which both the preceding operation and the succeeding operation are different between the habitual operation cluster candidate CL11 and the habitual operation cluster CL12 as a sequence variable operation.

[0245] That is, in the comparison result CL13 of FIG. 8, operation D among the operations has different preceding and succeeding operations, and is therefore detected as an order-variable operation.

[0246] Note that operations A and C have different subsequent operations but the same previous operation. Also, operation B has a different previous operation but the same subsequent operation. Therefore, operations A, B, and C are not detected as order-variable operations.

[0247] Therefore, the operation memory execution control unit 53 integrates the habitual operation cluster candidate CL11 and the habitual operation cluster CL12 based on the detected order-variable operation to generate an integrated memory cluster CL14 as shown in Figure 8, and registers it in the habitual operation / dialogue cluster memory unit 55.

[0248] That is, in the integrated memory cluster CL14 of Figure 8, it is recorded that, from the left, operation A is an operation of transitioning to a video viewing site, operation D is an operation of setting the volume to 30, operation B is an operation of selecting video content Z, and operation C is an operation of clicking the play button.

[0249] Furthermore, since operation D is detected as an order-variable operation, a variable range is set for it.

[0250] In the integrated storage cluster CL14 in FIG. 8, since the operation D is detected as an order-variable operation, the pre-condition of the operation D is set to the operation A, and the post-condition is set to the termination, as indicated by the arrow in the figure.

[0251] That is, the information of the integrated storage cluster CL14 in FIG. 8 records that the order of operation D can be changed if it is within a variable range between operation A and the terminal operation.

[0252] Incidentally, in the integrated memory cluster CL14 of FIG. 8, an example is shown in which the most recently performed operation sequence is registered, but it is also possible to register the operation sequence of an already registered habitual operation cluster.

[0253] Through the above processing, among the registered habitual operation clusters, habitual operation cluster candidates in which the operations are identical and only the order is different are integrated with the registered habitual operation cluster and registered as a single habitual operation cluster (integrated memory cluster CL14).

[0254] At this time, an operation recognized as an order-variable operation can be set as a variable range in which the order of the operations can be changed.

[0255] Through the above processing, the habitual operation clusters are successively registered in the habitual operation / interaction cluster storage unit 55. The registered habitual operation clusters can be presented to the user so that the multiple operations that make up the habitual operation cluster can be executed all at once, or can be registered as a macro, thereby enabling the execution of multiple highly addictive operations.

[0256] <Dialogue Cluster> The above describes the habitual operation clustering process based on habitual operations. However, after an habitual operation cluster is generated, changes to the habitual operations may require various changes, such as deletions and additions to the habitual operation cluster, and changes to entities.

[0257] Therefore, a dialogue cluster is a cluster that is able to respond to changes in habitual operations by including operations based on dialogue with the user in at least some of the multiple operations that constitute a cluster and are executed collectively based on habitual operations.

[0258] More specifically, the operation memory execution control unit 53 stores a dialogue cluster with the following three types of operations in the habitual operation cluster as variables from the operation history, and generates a dialogue based on the dialogue information of the variable operations stored in the dialogue cluster, and executes the operations that make up the cluster while appropriately changing them.

[0259] The first operation is an operation having the same operation type (utterance intention) (Intent) but different attributes (Entity). For operations having the same operation type (Intent) but different attributes (Entity), the operation memory execution control unit 53 sets a selection dialogue flag f as a flag indicating whether or not a dialogue is to be performed when the corresponding operation is performed. s Set the multiple entities entered and select them. s The result is stored in the dialogue cluster together with the number of executions.

[0260] In addition, when executing a dialogue cluster related to an operation having the same operation type (Intent) but different attributes (Entity), the operation memory execution control unit 53 sets a selection dialogue flag f s Depending on the setting, the option information I s Based on the input probability, options are presented based on the input probability, and a query is made for the selected Entity. An operation is performed based on the user's response, and the option information I s Update.

[0261] The second operation is an operation of adding and deleting. For the operation of adding and deleting, the operation memory execution control unit 53 sets the confirmation dialogue flag f a Set and check the execution count information I a The result is stored in the dialogue cluster as

[0262] In addition, when executing a dialogue cluster related to an added or deleted operation, the operation memory execution control unit 53 sets a confirmation dialogue flag f a Depending on the setting, an inquiry is made as to whether the operation can be performed or not, and the operation is performed according to the user's response, and confirmation information I a Update.

[0263] The third operation is an operation that depends on the observation context. For an operation that depends on the observation context, the operation memory execution control unit 53 sets the context interaction flag f c and the entity that matches the observation context is stored in the context information I c The result is stored in the dialogue cluster as

[0264] In addition, when executing a dialogue cluster related to an operation that depends on an observation context, the operation memory execution control unit 53 sets a context dialogue flag f c Depending on the settings of the context information I c Based on the above, if the observed context does not match, an inquiry is made to the entity, and an operation is performed based on the user's response. c Update.

[0265] (Interactive clustering processing) Next, in order to explain the process for generating dialogue clusters, the dialogue clustering process will be described with reference to the flowchart of FIG.

[0266] In step S51, the operation memory execution control unit 53 initializes a counter n, which counts operations, to 1.

[0267] In step S52, the operation memory execution control unit 53 determines whether or not the user operation information of the operation n has been supplied, and repeats the same process until it is supplied.

[0268] Then, in step S52, if user operation information of operation n is supplied, the process proceeds to step S53.

[0269] In step S53, the operation storage execution control unit 53 adds a time stamp to the user operation information of the operation n, and stores the information in the user operation history storage unit 54 as a user operation history.

[0270] In step S54, the operation memory execution control unit 53 calculates the habitual score S of the operation n by the calculation described with reference to the above-mentioned formulas (1) to (4) based on the user operation history stored in the user operation history storage unit 54. n Calculate.

[0271] In step S55, the operation memory execution control unit 53 calculates the habit score S corresponding to the operation n. n However, the habit threshold S th Determine whether it is greater than

[0272] In step S55, the habit score S corresponding to the operation n is calculated. n However, the habit threshold S th If it is determined that the operation n is greater than n, the operation n is deemed to be a highly habitual operation, and the process proceeds to step S56.

[0273] In step S56, the operation memory execution control unit 53 executes an observation context dependent operation determination process to determine whether the operation n is an operation that depends on the observation context. If the operation n is an operation that depends on the observation context, the operation memory execution control unit 53 stores the context information I c and the context interaction flag f c Set the following.

[0274] The observation context dependent operation determination process will be described in detail later with reference to FIGS.

[0275] In step S57, the operation memory execution control unit 53 calculates the habit score S of the operation n-1 immediately before the operation n. n-1 However, the habit threshold S th Determine whether it is smaller than

[0276] In step S57, the habit score S of the operation n-1 immediately before the operation n is calculated. n-1 However, the habit threshold S th If it is smaller, the process proceeds to step S58.

[0277] In step S58, the operation memory execution control unit 53 adds the operation n-1 to the dialogue cluster candidates.

[0278] In step S59, the operation memory execution control unit 53 adds the operation n to the dialogue cluster candidates.

[0279] In step S60, the operation memory execution control unit 53 determines whether or not an instruction to end the process has been given, and if an instruction to end the process has been given, the process ends.

[0280] Also, in step S60, if an instruction to end has not been given, the process proceeds to step S61.

[0281] In step S61, the operation memory execution control unit 53 increments the counter n by 1, and the process returns to step S52.

[0282] That is, the habit score S n is the habit threshold S th When there are a plurality of operations n higher than n (repeated), the processes of steps S52 to S61 are repeated, the counter n is sequentially incremented by 1, and operations of the dialogue cluster candidates are continuously added.

[0283] In addition, the first operation in the interaction cluster has a habit score of S n is the habit threshold Sth Since the habit score S of the operation n-1 immediately before the operation n is higher than the habit score S n-1 is the habit threshold S th If it is lower than n, the process in step S58 causes the operation n-1 immediately preceding the operation n to be included as the leading operation of the dialogue cluster candidate.

[0284] Therefore, among the operations that make up the conversation cluster candidates, the operations other than the first one have a habit score S n is the habit threshold S th Therefore, in step S57, the habit score S n is the habit threshold S th Therefore, the process of step S58 is skipped.

[0285] Furthermore, in step S55, the habit score S corresponding to the operation n is calculated. n However, the habit threshold S th If it is determined that the operation n is not larger than n, the operation n is not considered to be a highly habitual operation, and the process proceeds to step S62.

[0286] In step S62, the operation memory execution control unit 53 executes an operation determination process for an operation with the same Intent but different Entities to determine whether the operation n is an operation with the same Intent but different Entities. If the operation is an operation with the same Intent but different Entities, the selection dialogue flag f s and option information I s Set the following.

[0287] The operation determination process for when the Intent is the same but the Entity is different will be described in detail later with reference to FIGS.

[0288] In step S63, the operation memory execution control unit 53 sets the selection dialogue flag f of the operation n. s Determine whether or not the property is set to true.

[0289] In step S63, the selection dialogue flag f of the operation n is s If it is determined that the setting is true, the process proceeds to step S59.

[0290] In step S63, the selection dialogue flag f of the operation n is s If it is determined that the setting is not set to true, the process proceeds to step S64.

[0291] In step S64, the operation memory execution control unit 53 executes an added / deleted operation determination process to determine whether the operation n is either an added operation or a deleted operation. If the operation n is either an added operation or a deleted operation, the operation memory execution control unit 53 sets a confirmation dialogue flag f a And, confirmation information I a and, if necessary, set the dummy dialogue flag f d Set.

[0292] The added and deleted operation determination processes will be described in detail later with reference to FIGS.

[0293] In step S65, the operation memory execution control unit 53 sets the confirmation dialogue flag f a Determine whether or not the property is set to true.

[0294] In step S65, the confirmation dialogue flag f of the operation n is a If it is determined that the setting is true, the process proceeds to step S59.

[0295] In step S65, the confirmation dialogue flag f of the operation n is set. a If it is determined that the setting is not set to true, the process proceeds to step S66.

[0296] In step S66, the operation memory execution control unit 53 sets the Tammy dialogue flag f d Determine whether or not the property is set to true.

[0297] In step S66, the dummy dialogue flag f d If it is determined that the setting is true, the process proceeds to step S59.

[0298] In step S66, the dummy dialogue flag f d If it is determined that the setting is not set to true, the process proceeds to step S67.

[0299] In step S67, the operation memory execution control unit 53 calculates the habit score S of the immediately previous operation n-1. n-1 is the habit threshold S th or selection dialogue flag f s , confirmation dialogue flag f a , context interaction flag f c , and a dummy dialogue flag f d Determine whether any of the following is true.

[0300] In step S67, the habit score S of the operation n-1 is calculated. n-1 However, the habit threshold S th greater than or equal to the selection dialogue flag f s , confirmation dialogue flag f a , context interaction flag f c , and a dummy dialogue flag f d If any of the above is deemed to be true, the operation up to that point is deemed to be part of the operation included in the dialogue cluster candidate, and the process proceeds to step S68.

[0301] In step S67, the habit score S of the operation n-1 is n-1 However, the habit threshold S th and the selection dialogue flag f s , confirmation dialogue flag f a , context interaction flag f c , and a dummy dialogue flag f dIf any of the above is not considered to be true, the immediately preceding operation is considered to be an operation that is not included in the dialogue cluster candidates, and the process proceeds to step S60.

[0302] In step S68, the operation memory execution control unit 53 determines whether or not the number of operations included in the dialogue cluster candidate formed by the operations up until that point is two or less.

[0303] Since a dialogue cluster is assumed to be three or more in number, if in step S68 the dialogue cluster candidate formed by the operations up to that point contains two or fewer operations, it is deemed that no dialogue cluster has been formed, and processing proceeds to step S69.

[0304] Then, in step S69, the operation storage execution control unit 53 clears the information on the operations stored as dialogue cluster candidates, and the process proceeds to step S60.

[0305] Also, in step S68, if the dialogue cluster candidate formed by the immediately preceding operations does not contain two or fewer operations, that is, it is deemed that the dialogue cluster candidate contains three or more operations, and the process proceeds to step S70.

[0306] In step S70, the operation memory execution control unit 53 sets the selected dialogue flag f s , confirmation dialogue flag f a , context interaction flag f c , and a dummy dialogue flag f d Determine whether any of the following operations is set to true:

[0307] In step S70, the dialogue cluster candidates stored in the habitual operation / dialogue cluster storage unit 55 are assigned a selection dialogue flag f s , confirmation dialogue flag f a , context interaction flag f c , and a dummy dialogue flag fd If the operation does not include any of the operations in which any of the items is set to true, the process proceeds to step S69.

[0308] That is, in this case, since there is no need to register it as a dialogue cluster, the stored information on the dialogue cluster candidate is cleared.

[0309] In step S70, the dialogue cluster candidates stored in the habitual operation / dialogue cluster storage unit 55 are assigned a selected dialogue flag f s , confirmation dialogue flag f a , context interaction flag f c , and a dummy dialogue flag f d If it is determined that the operation includes any one of the operations in which the first item is set to true, the process proceeds to step S71.

[0310] In step S71, the operation memory execution control unit 53 determines whether or not there are any completely identical clusters among the already registered habitual operation clusters and dialogue clusters (hereinafter simply referred to as registered clusters) stored in the habitual operation / dialogue cluster memory unit 55, in which the operations contained in the dialogue cluster candidate currently being processed are identical and have the same sequence (the order is the same).

[0311] That is, if the dialogue cluster candidate is completely identical to a cluster already registered in the habitual operation / dialogue cluster storage unit 55, the context information I c Since it is necessary to update the data, it is determined whether there is an exact match.

[0312] In step S71, if there is a cluster already registered and stored in the habitual operation / dialogue cluster storage unit 55 that is completely identical to the dialogue cluster candidate currently being processed, the process proceeds to step S72.

[0313] In step S72, the operation memory execution control unit 53 executes a process of updating the context information of the dialogue cluster candidate, and updates the context information I cUpdate.

[0314] The process of updating the context information of the dialogue cluster candidates will be described in detail later with reference to FIG.

[0315] Also, in step S71, if there is no cluster already registered and stored in the habitual operation / dialogue cluster storage unit 55 that is completely identical to the dialogue cluster candidate currently being processed, the process of step S72 is skipped and the process proceeds to step S73.

[0316] In step S73, the operation memory execution control unit 53 determines whether or not any of the already registered clusters stored in the habitual operation / dialogue cluster memory unit 55 is included in the sequence of the dialogue cluster candidate currently being processed.

[0317] If it is determined in step S73 that any of the already registered clusters stored in the habitual operation / dialogue cluster storage unit 55 is included in the sequence of the dialogue cluster candidate currently being processed, the process proceeds to step S74.

[0318] In step S74, the operation memory execution control unit 53 deletes, from among the already registered clusters stored in the habitual operation / dialogue cluster storage unit 55, any registered cluster that is included in the sequence of the dialogue cluster candidate currently being processed.

[0319] Also, in step S73, if it is determined that none of the already registered clusters stored in the habitual operation / dialogue cluster memory unit 55 is contained in the sequence of the dialogue cluster candidate currently being processed, the process of step S74 is skipped and the process proceeds to step S75.

[0320] In step S75, the operation memory execution control unit 53 determines whether or not there is any cluster that has already been registered and is stored in the habitual operation / dialogue cluster storage unit 55 that can be replaced with the dialogue cluster currently being processed.

[0321] If it is determined in step S75 that any of the already registered clusters stored in the habitual operation / dialogue cluster storage unit 55 can be replaced with the dialogue cluster currently being processed, the process proceeds to step S76.

[0322] In step S76, the operation memory execution control unit 53 deletes, from among the already registered dialogue clusters stored in the habitual operation / dialogue cluster memory unit 55, any registered cluster that can be replaced with the dialogue of the dialogue cluster candidate currently being processed.

[0323] Also, in step S75, if it is determined that there is no cluster already registered and stored in the habitual operation / dialogue cluster storage unit 55 that can be replaced with the dialogue cluster currently being processed, the process of step S76 is skipped and the process proceeds to step S77.

[0324] In step S77, the operation memory execution control unit 53 registers the operation of the dialogue cluster candidate currently being processed in the habitual operation / dialogue cluster memory unit 55 as a new dialogue cluster.

[0325] In other words, among the already registered clusters stored in the habitual operation / dialogue cluster storage unit 55, the context information of those having the same sequence of operations as those included in the dialogue cluster candidate currently being processed is updated, and those that include or can replace operations included in the dialogue cluster candidate currently being processed are deleted, and the other dialogue cluster candidates currently being processed are regarded as new dialogue clusters consisting of multiple operations, and are therefore registered in the habitual operation / dialogue cluster storage unit 55.

[0326] By the above process, when a new operation n is accepted, the habit score S n is required, and the habit threshold S th By comparing with the above, conversation cluster candidates are set.

[0327] Furthermore, by comparing the operations constituting each of the dialogue cluster candidates with those of already registered clusters, any dialogue cluster candidate consisting of operations completely different from those of the already registered clusters is registered as a new dialogue cluster.

[0328] This makes it possible to sequentially register, as a dialogue cluster, dialogue cluster candidates that include a plurality of operations that have a high habituation score, are highly habitual as user operations, and have the dialogue flag set to true.

[0329] Incidentally, since there is a possibility that the habitual nature of a registered dialogue cluster may deviate over time, a dialogue cluster that remains registered for a predetermined period of time or longer may be deleted.

[0330] <Operations with the same operation type (Intent) but different attributes (Entity)> Next, operations having the same operation type (Intent) but different attributes (Entity) will be described with reference to Fig. 10. Note that hereinafter, the operation type and attribute are also simply referred to as Intent and Entity, respectively.

[0331] Consider a case where an habitual operation cluster made up of operations 1 to 8 shown in state ST101 shown on the left side of FIG. 10 is generated by the habitual operation clustering process described above and executed.

[0332] In state ST101, operation 1 is an operation of transitioning to a route information site, operation 2 is an operation of transitioning to a transfer guide page, operation 3 is an operation of inputting "Osaki" as the departure station, operation 4 is an operation of inputting "Moto-Atsugi" as the arrival station, operation 5 is an operation of selecting "Departure" as the date and time condition, operation 6 is an operation of checking the condition for using a paid express train, operation 7 is an operation of selecting the order of earliest arrival as a display condition, and operation 8 is an operation of clicking the search button.

[0333] Here, when a habitual operation cluster consisting of operations 1 to 8 shown in state ST101 on the left side of Figure 10 is executed collectively, for example, as shown in state ST102 in the center of Figure 10, operation 4' is performed instead of operation 4, and an operation to input "Fujisawa" instead of "Moto-Atsugi" as the arrival station is performed.

[0334] In such a case, if only the habitual operation clustering process is performed, the cluster of operations 1 to 3 and the cluster of operations 5 to 8 will be registered as habitual operation clusters individually. In other words, the habitual operation cluster of state ST101 will essentially be divided and registered, and it will not be possible to execute them all at once.

[0335] Here, operation 4 and operation 4' have the same Intent (operation type) of "input arrival station", but the names of the arrival stations are "Hon-Atsugi" and "Fujisawa", respectively, and are different Entities (attributes).

[0336] Therefore, the operation memory execution control unit 53 sets the same Intent, “input arrival station”, for Operation 4 as shown in state ST103 on the right side of FIG. 10, and then sets the selection dialogue flag f s and option information I s Set the following.

[0337] Selection dialogue flag f s This is a flag that sets whether or not to present operation options to the user through dialogue and allow them to select an operation when, for example, an operation with the same Intent but a different Entity is performed among the operations that are performed collectively as set by the habitual operation cluster.

[0338] More specifically, the selection dialogue flag f s is set to true when a process is implemented in which operation options are presented interactively to the user to select one, and is set to false otherwise.

[0339] For example, for the habitual operation cluster shown in the state ST101 of FIG. 10, as shown in the state ST102, when an operation with the same Intent but different Entities, such as the operation indicated by operation 4', is performed instead of operation 4, the selection dialogue flag f s is set as shown by operation 4 in the state ST103 of FIG. 10.

[0340] Here, the option information I s is the information of the options specifically presented to the user when the selection dialogue flag f s is set to true, and information on the options, the number of times each option has been selected, and the total number of times the options have been selected is registered.

[0341] More specifically, in the state ST103 of FIG. 10, the state is shown when "Total: 2, Motohagi: 1, Fujisawa: 1" is registered as the option information I s with initial values respectively. There are options Motohagi and Fujisawa, and it is shown that each has been selected once and a total of 2 times. Note that the total number of times is set to 2 as the initial value.

[0342] In the case of the state ST103 of FIG. 10, although Motohagi and Fujisawa are set as options, when presented as options, the respective input probabilities are calculated and presented in descending order of input probability. When the probabilities are the same, they are presented starting from the most recently selected option. Here, the input probability is the number of times an option has been selected divided by the total number of times. In the case of the state ST103 of FIG. 10, for the options Motohagi and Fujisawa, the input probability of each is 0.5 (= 1 / 2).

[0343] <Operation determination process with the same Intent and different Entities> Next, with reference to the flowchart of FIG. 11, the operation determination process with the same Intent and different Entities will be described. Note that the operation determination process with the same Intent and different Entities in the flowchart of FIG. 11 is the process of step S62 in the flowchart of FIG. 9.

[0344] In step S101, the operation memory execution control unit 53 determines whether the number of operations stored in the dialogue cluster candidate is one or more.

[0345] If it is determined in step S101 that the number of operations stored in the dialogue cluster candidate is one or more, the process proceeds to step S102.

[0346] In step S102, the operation memory execution control unit 53 determines whether or not there is a stored cluster in the clusters already stored in the habitual operation / dialogue cluster storage unit 55, the sequence of which matches the prefix of the dialogue cluster candidate.

[0347] If it is determined in step S102 that a stored cluster whose sequence matches the beginning of the dialogue cluster candidate exists among the dialogue clusters stored in the habitual operation / dialogue cluster storage unit 55, the process proceeds to step S103.

[0348] In step S103, the operation memory execution control unit 53 stores the operation M immediately after the stored cluster match, which has a sequence that matches the dialogue cluster candidate. n Determines whether the operation n matches the Intent.

[0349] In step S103, the operation M immediately after the conversation cluster candidate and the stored cluster whose sequence matches the beginning of the conversation cluster candidate are n If the Intent of operation n matches the Intent of operation n, the process proceeds to step S104.

[0350] In step S104, the operation memory execution control unit 53 n Select dialogue flag f s is set, and the selection dialogue flag f s is set to true, i.e., the option information I s Determine whether or not is set.

[0351] In step S104, the selection dialogue flag f s is set to true and the option information I s If it is deemed that the setting is set, the process proceeds to step S105.

[0352] In step S105, the operation memory execution control unit 53 performs the operation M n Option Information I s The data reflecting the entity information of operation n is stored in the "option information I" of operation n. s Set as.

[0353] In step S106, the operation memory execution control unit 53 sets the selection dialogue flag f s and also set the selection dialogue flag f s Set to true.

[0354] In step S104, the selection dialogue flag f s is not set or is set to false, and the option information I s If it is determined that the setting is not set, the process proceeds to step S107.

[0355] In step S107, the operation memory execution control unit 53 stores the operation M immediately after the dialogue cluster candidate and the dialogue cluster candidate, which has a sequence that matches the dialogue cluster candidate. n Determines whether operation n and Entity are different.

[0356] In step S107, the operation M immediately after the conversation cluster that has a sequence that matches the conversation cluster candidate is n If the Entity of operation n is different from the Entity of operation n, the process proceeds to step S108.

[0357] In step S108, the operation memory execution control unit 53 stores the option information I s Set the initial value of the number of selections to 1.

[0358] Also, in step S101, if it is determined that the number of operations stored in the dialogue cluster candidate is not more than 1, and in step S102, if it is determined that there is no dialogue cluster in the cluster already stored in the habitual operation / dialogue cluster storage unit 55 that has a forward match with the dialogue cluster candidate in terms of sequence, then in step S103, the operation M immediately after the match of the stored cluster that has a forward match with the dialogue cluster candidate in terms of sequence n whose Intent does not match the Intent of operation n, or in step S107, if the Entity of the operation M n immediately after the match of the stored cluster that has a forward match with the dialogue cluster candidate in terms of sequence is the same as the Entity of operation n, the process ends.

[0359] Through the above process, it is determined whether the user's operation n is an operation with the same Intent but different Entities. If it is an operation with the same Intent but different Entities, then for the first time, the selection dialogue flag f s is set to true, the option information I s is set to the initial value, and thereafter, the option information I s corresponding to the new operation n is sequentially set.

[0360] <Dialogue generation example for an operation with the same Intent but different Entities> Next, with reference to FIG. 12, a dialogue generation example for an operation with the same Intent but different Entities will be described.

[0361] For example, consider the case where a dialogue cluster as shown in state ST111 as shown in FIG. 12 is set. Note that the dialogue cluster in state ST111 is the same dialogue cluster as state ST103 in FIG. 10.

[0362] In operation 4 in the left part of FIG. 12, since the selection dialogue flag f s is set to true, it is set to a state where the process of generating a dialogue by the dialogue cluster is performed.

[0363] Here, for example, as shown in the right part of Figure 12, in process SC101, when the user 11 utters an utterance such as "transfer guidance," the operation memory execution control unit 53 recognizes that an instruction has been given to execute the dialogue cluster shown in the left part of Figure 12 relating to transfer guidance.

[0364] In the process Ex101, the operation memory execution control section 53 executes the operations 1 to 3 shown in state ST111 of FIG. 12 in response to the utterance of "transfer guide."

[0365] In step Ex102, the operation memory execution control unit 53 executes the process of operation 4 and sets the selection dialogue flag f s is set to true, so the option information I s In this example, since both Hon-Atsugi and Fujisawa have an input probability of 0.5, for example, the most recent input result may be regarded as an option with a high input probability and presented.

[0366] In process Ex103, the operation memory execution control section 53 outputs a voice prompting the user to select from the presented options, such as "Where is the destination?".

[0367] Here, in process SC102, when the user utters "Motoatsugi", the operation memory execution control unit 53 recognizes that "Motoatsugi" has been selected as an option.

[0368] In the process Ex104, the operation memory execution control unit 53 inputs Hon-Atsugi as the arrival station and selects the option information I s and executes operations 5 to 8 of state ST111.

[0369] Through this series of processes, when a change occurs in the habitual operation and an operation with the same Intent but a different Entity is performed, a query dialogue with the Entity as a choice is presented, and processing is executed according to the selection.

[0370] As a result, even if a change occurs in the habitual operation, a dialogue cluster is set for the habitual operation cluster, enabling flexible response.

[0371] <Dialogue processing for operations with the same Intent but different Entities> Next, with reference to the flowchart of FIG. 13, dialogue processing for operations with the same Intent but different Entities will be described.

[0372] In addition, this process is executed when an operation with the same Intent but different Entities and with the selection dialogue flag f s set to true is performed. That is, it is a process executed when the process of operation 4 in the state ST111 of FIG. 12 is performed.

[0373] In step S131, the operation memory execution control unit 53 displays (presents) the Entities included in the option information I s as options in descending order of input probability. In the case of FIG. 12, Motohigashi and Fujisawa are presented as options. However, since the input probabilities are the same, for example, the option that was input most recently may be presented first as the option with the highest input probability.

[0374] In step S132, the operation memory execution control unit 53 outputs a system utterance that prompts the user to make a selection (response) for the option based on the inquiry phrase registered in advance for each Intent.

[0375] More specifically, the operation memory execution control unit 53 controls the response generation unit 63 to generate a system utterance that prompts the user to make a selection (response) for the option based on the inquiry phrase registered in advance for each Intent, and outputs it as data to the speech synthesis processing unit 65. The speech synthesis processing unit 65 performs speech synthesis processing on the data of the system utterance and outputs it as speech from the speech output unit 67.

[0376] Incidentally, the processing relating to the voice output of the system utterance is similar, and therefore, description thereof will be omitted hereafter.

[0377] In step S133, the operation memory execution control unit 53 determines whether or not a response utterance has been made as a result of the user's selection of an option, and repeats the same process until it is determined that a response utterance has been made.

[0378] More specifically, the operation memory execution control unit 53 determines whether or not a response utterance has been made based on the user's speech phrase that has been converted into text after the voice signal corresponding to the user's utterance input via the voice input unit 56 has been voice recognized by the voice recognition processing unit 57, and repeats the same process until it is determined that a response utterance has been made.

[0379] Incidentally, the processing relating to the user's speech is similar, and therefore will not be described below.

[0380] Then, in step S133, if it is determined that a response utterance has been made as a result of the user's selection of an option, the process proceeds to step S134.

[0381] In step S134, the operation memory execution control unit 53 executes the operation (operation 4) by the Entity of the option corresponding to the response utterance.

[0382] In step S135, the operation memory execution control unit 53 determines whether the user's response utterance is included in the option information I s Determine whether the entity is included in.

[0383] In step S135, the user's response utterance is s If it is determined that the entity is included in the list, the process proceeds to step S136.

[0384] In step S136, the operation memory execution control unit 53 selects the option information I s The number of times the entity was selected by the response utterance included in is incremented by 1.

[0385] On the other hand, in step S135, the user's response utterance is s If it is determined that the entity is not included in the list, the process proceeds to step S137.

[0386] In step S137, the operation memory execution control unit 53 selects the option information I s The newly selected Entity in the response utterance is added and registered in the request, and the count is set to the initial value of 1.

[0387] In step S138, the operation memory execution control unit 53 stores the selection information I s The total number of times information is incremented by 1.

[0388] In step S139, the operation memory execution control unit 53 selects the option information I s Determine whether there is an entity in whose input probability is greater than or equal to a specified value.

[0389] In step S139, the selection information I s If there is an entity in which the input probability is equal to or greater than a prescribed value, the process proceeds to step S140.

[0390] In step S140, the operation memory execution control unit 53 sets the selection dialogue flag f s is set to false, and the operation is set to fix the entity whose input probability is equal to or greater than a specified value, and the process is terminated.

[0391] In step S139, the selection information I s If there is no entity in the list whose input probability is equal to or greater than a specified value, the process skips step S140 and ends.

[0392] Through the above process, the selection dialogue flag f s For operations where option information I is set to true,s The entities in are presented as options in order of input probability to prompt a selection, and the process is executed based on the selected entity.

[0393] Then, for the selected Entity, the count is incremented by 1, and the total count is incremented by 1. At this time, if a new Entity other than the Entities presented as options is selected, it is registered as a new option and the count is set to 1.

[0394] Furthermore, when the input probability of an Entity is calculated sequentially and exceeds a specified value, it is assumed that the user always selects an option whose input probability as an Entity is equal to or greater than the specified value, and the selection dialogue flag f s is set to false, so that the operation is fixed to the entity whose input probability is always greater than or equal to the specified value.

[0395] In other words, once an entity has been fixed in its operation in this way, it will function as a normal habitual operation cluster without any further interaction, and multiple operations corresponding to changes in habitual operations will be executed all at once.

[0396] As a result, even if a change occurs in the habitual operation, a dialogue cluster is set and dialogue processing is performed that prompts the user to select an option.Furthermore, a specific entity is repeatedly selected based on the input probability, and when the probability exceeds a specified value, a fixed operation is performed on the specific entity, allowing for flexible response.

[0397] <Added and removed operations> (Added operations) Next, the added operations will be described with reference to FIG.

[0398] Consider a case where an habitual operation cluster consisting of operations 1 to 8 shown in state ST131 shown on the left side of Fig. 14 is generated and executed by the above-mentioned habitual operation clustering process. Note that state ST131 is the same as state ST101 in Fig. 10.

[0399] Here, when the habitual operation cluster consisting of operations 1 to 8 shown in state ST131 on the left side of FIG. 14 is executed collectively, for example, as shown in state ST132 in the center of FIG. 14, operation 4' is performed between operations 4 and 5, and "Yokohama" is entered as the intermediate destination.

[0400] In this case, when only the addictive operation clustering process is performed, both operation 4' and operation 5 have the addictive score S n is the habit threshold S th but for a time interval T n is the threshold T th If the operation is shorter than the above, the operation is regarded as a dialogue cluster, and the operation memory execution control unit 53 sets the operation 4' to the operation 5, advances the subsequent operations by one, and further sets the operation 5 to "input "Yokohama" as the stopover station", as shown in the state ST133 on the right side of FIG. 14, and sets the confirmation dialogue flag f a And, confirmation information I a Set the following.

[0401] Confirmation dialogue flag f a This is a flag that sets whether or not the user can choose to execute the operation added through dialogue when, for example, an operation is performed that is added to the collective operations set by the habitual operation cluster.

[0402] More specifically, the confirmation dialogue flag f a is set to true when an additional operation is presented interactively to inquire as to whether the operation should be executed by the user, and is set to false otherwise.

[0403] For example, for the habitual operation cluster shown in state ST131 of FIG. 14, when an operation such as that shown in operation 4' is performed between operations 4 and 5 as shown in state ST132, the selection dialogue flag f a is set for operation 5 instead of operation 4'.

[0404] Verification Information I a is the confirmation dialogue flag f a When is set to true, this is information for calculating the user's input probability, and the number of times execution is instructed and the total number of times (total number of times) are registered. Note that the initial value of the total number of times is set to 2, and the execution probability is also set to 0.5 in the initial setting.

[0405] More specifically, in FIG. 14, confirmation information I a “Total:2,exec:1” is registered, which indicates that the total number of confirmations is two and that the execution instruction has been selected once.

[0406] That is, in the case of FIG. 14, the execution probability that execution is instructed is calculated to be 0.5.

[0407] (Deleted Action) Next, the deleted operations will be described with reference to FIG.

[0408] Here, when a habitual operation cluster consisting of operations 1 to 8 shown in state ST131 (same as FIG. 14) on the left side of FIG. 15 is executed collectively, it is assumed that operation 6, “check the condition for using toll express trains,” is deleted, for example, as shown in state ST141 in the center of FIG. 15.

[0409] In such a case, if there is only habitual operation clustering processing, operation 6 will be the habitual score S n However, the habit threshold S th but for a time interval T n is the threshold T thIf the dialogue is shorter than the time required for the dialogue, the dialogue is regarded as a dialogue cluster, and the operation memory execution control unit 53 sets the operation 6 to "check the condition for using the toll express train" as shown in the state ST142 on the right side of FIG. 15, and sets the confirmation dialogue flag f a And, confirmation information I a Set the following.

[0410] For example, when operation 6 is deleted from the habitual operation cluster shown in state ST131 of FIG. 15 as shown in state ST141, the operation memory execution control unit 53 sets the confirmation dialogue flag f for operation 6 as shown in state ST142 of FIG. a Set.

[0411] Also, in state ST142 of FIG. a “Total:2,exec:1” is registered, which indicates that the total number of confirmations is two and that the execution instruction has been selected once.

[0412] That is, in the case of state ST142 in FIG. 15, the execution probability that execution is instructed is calculated to be 0.5.

[0413] <Added and deleted operation detection processes> Next, the added and deleted operation determination processes will be described with reference to the flowchart of Fig. 16. Note that the added and deleted operation determination processes in the flowchart of Fig. 16 are the processes in step S64 in the flowchart of Fig. 9.

[0414] In step S161, the operation memory execution control unit 53 calculates the time interval T n is a given threshold T th That is, it is determined whether the time interval is shorter than the threshold value.

[0415] In step S161, the time interval T n is a given threshold T th If so, the process proceeds to step S162.

[0416] In step S162, the operation memory execution control unit 53 determines whether the number of operations stored in the dialogue cluster candidate is one or more.

[0417] If it is determined in step S162 that the number of operations stored in the dialogue cluster candidate is one or more, the process proceeds to step S163.

[0418] In step S163, the operation memory execution control unit 53 determines whether or not there is a stored cluster in the clusters already stored in the habitual operation / dialogue cluster storage unit 55, the sequence of which matches the prefix of the dialogue cluster candidate.

[0419] If it is determined in step S163 that a stored cluster whose sequence matches the beginning of the dialogue cluster candidate exists among the clusters already stored in the habitual operation / dialogue cluster storage unit 55, the process proceeds to step S164.

[0420] In step S164, the operation memory execution control unit 53 performs the operation M immediately after the match of the stored cluster whose sequence matches the dialogue cluster candidate. n Thereafter, it is determined whether or not there is an operation identical to operation n, that is, whether or not operation n is an addition operation.

[0421] In step S164, the operation M immediately after the conversation cluster candidate and the stored cluster whose sequence matches the beginning of the conversation cluster candidate are n If there is no subsequent operation identical to operation n, that is, if it is determined that operation n is an addition operation, the process proceeds to step S165.

[0422] In step S165, the operation memory execution control unit 53 inputs confirmation information I a and set the initial value.

[0423] In step S166, the operation memory execution control unit 53 sets a confirmation dialogue flag f a and also set the confirmation dialogue flag fa Set to true.

[0424] In step S164, the operation M immediately after the match of the stored cluster whose sequence matches the conversation cluster candidate is n If it is determined that there is an operation identical to operation n thereafter, that is, operation n is not an addition operation, the process proceeds to step S167.

[0425] In step S167, the operation memory execution control unit 53 performs the operation M immediately after the match of the stored cluster whose sequence matches the dialogue cluster candidate. n is the same as operation n, that is, it is determined whether it is a delete operation.

[0426] In step S167, the operation M immediately after the conversation cluster candidate and the stored cluster whose sequence matches the beginning of the conversation cluster candidate are n If operation n is the same as operation n and is not considered to be a delete operation, the process proceeds to step S168.

[0427] In step S168, the operation memory execution control unit 53 n Confirm dialogue flag f a is set, and the confirmation dialogue flag f a is set to true, i.e., the confirmation information I a Determine whether or not is set.

[0428] In step S168, operation M n Confirm dialogue flag f a is set, and the confirmation dialogue flag f a is set to true, i.e., the confirmation information I a If it is deemed that the setting is set, the process proceeds to step S169.

[0429] In step S169, the operation memory execution control unit 53 performs a step of n Confirmation information I aThe data reflecting the execution information of operation n is stored in I a Then, the process returns to step S166. a Only the

[0430] On the other hand, in step S167, the operation M immediately after the match of the stored cluster whose sequence matches the conversation cluster candidate is n If operation n is not the same as operation n and is considered to be a delete operation, the process proceeds to step S170.

[0431] In step S170, the operation memory execution control unit 53 performs a process for determining whether the operation M n Delete the operation just before the operation that is the same as operation n from operation array A d The habit operability / dialogue cluster storage unit 55 stores the habit operability / dialogue cluster as the habit operability / dialogue cluster storage unit 55 .

[0432] In step S171, the operation memory execution control unit 53 stores the deletion operation array A in the habit operation / dialogue cluster storage unit 55. d It is determined whether the number of operations stored as is equal to or less than a predetermined threshold (for example, 3).

[0433] In other words, if the number of operations to be deleted is too large, they cannot be treated as the same habitual operation cluster. d A predetermined threshold is set for the number of operations stored as.

[0434] In addition, the deletion operation array A d The predetermined threshold value set for the number of operations stored in may be set arbitrarily.

[0435] In step S171, the deletion operation array A d If the number of operations stored as is equal to or less than a predetermined threshold (for example, 3), the process proceeds to step S172.

[0436] In step S172, the operation memory execution control unit 53 stores the deletion operation array Ad All operations are stored as confirmation information I a Set the initial value of and confirm dialogue flag f a is set to true and stored in the habit operability / dialogue cluster storage unit 55 as a dialogue cluster candidate.

[0437] In step S173, the operation memory execution control unit 53 sets a dummy dialogue flag f d Set the dummy dialogue flag f d Set to true.

[0438] On the other hand, in step S168, operation M n Confirmation dialogue flag f a is set to false, i.e., the confirmation information I a If it is determined that the setting is not set, the process proceeds to step S173.

[0439] In step S161, the time interval T n is a given threshold T th If it is determined in step S162 that the number of operations stored in the dialogue cluster candidate is not 1 or more, if it is determined in step S163 that there is no stored cluster whose sequence matches the dialogue cluster candidate in the beginning in the clusters stored in the habitual operation / dialogue cluster storage unit 55, or if it is determined in step S171 that the deletion operation array A d If the number of operations stored as is not less than or equal to a predetermined threshold (eg, 3), the process ends.

[0440] Through the above process, it is determined whether the user's operation n is an addition operation or a deletion operation. If it is an addition operation or a deletion operation, the confirmation dialogue flag f a is set to true, and confirmation information I a is set as the initial value, and thereafter, confirmation information I a are set in sequence.

[0441] In addition, the dummy dialogue flag f d is the habit score S n is the threshold S th This is a flag for adding an operation to be included in a dialogue cluster candidate, which is lower, to the dialogue cluster candidate by transitioning from the processing of steps S62 to S66 in the dialogue clustering process described with reference to the flowchart in Figure 9 to the processing of step S59.

[0442] That is, the dummy dialogue flag f d is a dialogue flag intended to be added to dialogue cluster candidates, so we use the dummy dialogue flag f d For operations for which the value is true, no interaction is performed with the user even when the operation is executed.

[0443] For example, operation 5 in state ST103 in FIG. 10 is immediately after operation 4, which has a different entity, so the number of transitions is F n and the transition probability P n Low habit score S n is the threshold S th However, since this is an operation that should be included in the candidate dialogue cluster, the dummy dialogue flag f d is set to true.

[0444] That is, operation 5 of state ST103 in FIG. 10 is executed by operation M n Confirmation dialogue flag f a Since the value of f is not true, the process proceeds to step S173, where the dummy dialogue flag f d is considered to be true.

[0445] As a result, in the determination in step S66 in the flowchart of FIG. 9, the dummy dialogue flag f d is determined to be true, the process proceeds to step S59, and operation 5 in state ST103 of FIG. 10 is added as a dialogue cluster candidate.

[0446] Also, for example, since the operation 6 in the state ST133 of FIG. 14 is immediately after the added operation 5, the number of transitions F n and the transition probability P n Low habit score S n is the threshold S th However, since this is an operation that should be included in the candidate dialogue cluster, the dummy dialogue flag f d is set to true.

[0447] That is, operation 6 of state ST133 in FIG. 14 is executed by operation M n Confirmation dialogue flag f a Since the value of f is not true, the process proceeds to step S173, where the dummy dialogue flag f d is considered to be true.

[0448] As a result, in the determination in step S66 in the flowchart of FIG. 9, the dummy dialogue flag f d is determined to be true, the process proceeds to step S59, and operation 6 in state ST133 of FIG. 14 is added as a dialogue cluster candidate.

[0449] Furthermore, for example, since operation 7 in state ST142 in FIG. 15 is immediately after operation 6 that has been deleted, the number of transitions F n and the transition probability P n Because the habit score is low, n is the threshold S th However, since this is an operation that should be included in the candidate dialogue cluster, the dummy dialogue flag f d is set to true.

[0450] That is, operation 7 of ST142 in FIG. 15 is operation M of the prefix-matching cluster in step S167 in the flowchart of FIG. n Since operation n is not the same as operation n, the process proceeds to step S170.

[0451] In addition, by the process of step S170, the operation M of the forward matching cluster is nWhen operations 1 through n are stored in the delete operation array Ad, in step S171, the number of operations stored is three or more, and therefore the process proceeds to step S172.

[0452] Then, in step S172, the confirmation dialogue flag f a When the value is set to true, the Tammy dialogue flag f d is set to true.

[0453] As a result, in the determination in step S66 in the flowchart of FIG. 9, the dummy dialogue flag f d is determined to be true, the process proceeds to step S59, and operation 7 in state ST142 is added as a dialogue cluster candidate.

[0454] <Example of dialogue generation for add and delete operations> Next, with reference to FIG. 17, a dialogue generation example for an added operation and a deleted operation will be described.

[0455] For example, consider the case where a dialogue cluster as shown in state ST161 in FIG. 17 is set.

[0456] Incidentally, the conversation cluster in state ST161 is the same as that in state ST133 in FIG.

[0457] In operation 5 in state ST161 of FIG. 17, the confirmation dialogue flag f a is set to true, so that the execution will be performed by the interactive cluster.

[0458] Here, for example, as shown in the right part of Figure 17, in process SC121, when the user 11 utters an utterance such as "transfer information," the operation memory execution control unit 53 recognizes that an instruction has been given to execute the dialogue cluster shown in the left part of Figure 17 relating to transfer information.

[0459] In process Ex121, the operation memory execution control section 53 executes the macro of operations 1 to 4 shown in the left part of FIG. 17 in response to the utterance of "transfer guide."

[0460] In process Ex122, the operation memory execution control unit 53 sets the confirmation dialogue flag f a is set to true, a voice prompting confirmation such as "Do you want to pass through Yokohama Station?" is output.

[0461] Here, in process SC122, when the user utters an utterance such as "Yes", the operation memory execution control unit 53 recognizes that an instruction has been given to execute the operation of "inputting Yokohama as a stopover station".

[0462] Therefore, in the process Ex123, the operation memory execution control unit 53 outputs the execution information indicating that the process of "inputting Yokohama as the transit station" has been executed as confirmation information I a The execution count "exec" and the total count (total count) are incremented by one each to reflect this, and operations 5 to 8 are executed.

[0463] Also, instead of processing SC122, as shown in processing SC123, when the user utters an utterance such as "No," the operation memory execution control unit 53 recognizes that an instruction has been given not to execute the operation of "entering Yokohama as a stop station."

[0464] In the process Ex124, the operation memory execution control unit 53 transmits the execution information indicating that the process of "inputting Yokohama as the intermediate station" has not been executed as confirmation information I a In order to reflect this, only the total number of times is incremented by 1 and updated, and operations 6 to 8 are performed.

[0465] Through this series of processes, when a change occurs in the habitual operation and an additional operation is to be performed, a dialogue is presented inquiring as to whether or not the additional operation needs to be performed, and processing is performed according to the confirmation result.

[0466] As a result, even if a change occurs in the habitual operation, a dialogue cluster is set for the habitual operation cluster, making it possible to flexibly respond to the change.

[0467] In FIG. 17, an example of utterance generation in the case of an addition operation has been described, but the case of a deletion operation is similar, so a description thereof will be omitted.

[0468] <Interaction with added and removed operations> Next, the interaction process for added and deleted operations will be described with reference to the flow chart of FIG.

[0469] This process is performed when the confirmation dialogue flag f a This process is executed when an operation n in which the value of n is set to true is performed. That is, for example, this process is executed when the process of operation 5 in state ST161 in FIG.

[0470] In step S191, the operation memory execution control unit 53 outputs a system utterance that prompts the user to respond or not to the execution, based on an inquiry phrase that is registered in advance for each Intent.

[0471] In step S192, the operation memory execution control unit 53 determines whether or not there has been a response utterance from the user instructing whether or not to execute the operation, and repeats the same process until it is determined that there has been a response utterance.

[0472] Then, in step S192, if it is determined that the user has made a response utterance instructing whether or not to execute the task, the process proceeds to step S193.

[0473] In step S193, the operation memory execution control unit 53 determines whether or not the response utterance from the user instructing whether or not to execute the operation is an utterance that approves the execution.

[0474] In step S193, if the response utterance from the user instructing whether or not to execute the task is an acknowledgment of the execution, the process proceeds to step S194.

[0475] In step S194, the operation memory execution control unit 53 executes the operation n.

[0476] In step S195, the operation memory execution control unit 53 receives confirmation information I a Increment the execution count exec by 1.

[0477] In step S196, the operation memory execution control unit 53 receives confirmation information I a Increment the Total number of times by 1.

[0478] In step S193, if the response utterance from the user instructing whether or not to execute the task is not an approval of the execution, the processes in steps S194 and S195 are skipped.

[0479] In step S197, the operation memory execution control unit 53 receives confirmation information I a The execution probability in is calculated, and it is determined whether the calculated execution probability is equal to or greater than an upper limit specified value.

[0480] In step S197, confirmation information I a If the execution probability in is equal to or greater than the upper limit specified value, the process proceeds to step S198.

[0481] In step S198, the operation memory execution control unit 53 regards the operation n as an operation to be incorporated into the normal habitual operation cluster as a result of the increase in the execution probability of the operation n as an additional operation, and sets the confirmation dialogue flag f a Set it to false and terminate the process.

[0482] Also, in step S197, confirmation information I a If the execution probability in is not equal to or greater than the upper limit specified value, the process proceeds to step S199.

[0483] In step S199, the operation memory execution control unit 53 receives confirmation information Ia The execution probability is calculated, and it is determined whether the calculated execution probability is equal to or smaller than a lower limit specified value.

[0484] In step S199, confirmation information I a If the execution probability in is equal to or less than the lower limit specified value, the process proceeds to step S200.

[0485] In step S200, the operation memory execution control unit 53 determines that the deleted operation n has not actually been used for a long time and its execution probability has decreased, so that it may be deleted, deletes it from the dialogue cluster, and terminates the process.

[0486] In step S199, the confirmation information I a If the execution probability in is not equal to or less than the lower limit specified value, the process of step S200 is skipped and the process ends.

[0487] Through the above series of processes, the confirmation dialogue flag f a For operations where is set to true, confirmation information I a The user is prompted to respond as to whether execution is possible or not, and a process is executed according to the response regarding whether execution is possible or not.

[0488] Then, when execution is instructed, the execution count "exec" is incremented by one, and the total count (Total count) is incremented by one.

[0489] Furthermore, when the execution probability is calculated sequentially and reaches or exceeds the upper limit, it is considered that the user always executes the operation n, and the confirmation dialogue flag f a Set it to false to make it a fixed operation that will always be executed.

[0490] In addition, when the execution probability is calculated sequentially and falls below a lower limit specified value, it is assumed that the user never executes operation n, and the dialogue cluster for operation n itself is deleted and set to a state where it is never executed.

[0491] That is, in either case, once the add operation becomes a fixed operation or the delete operation is deleted, it functions as a normal habitual operation cluster without any further interaction thereafter.

[0492] As a result, even if a change occurs such as an operation being added or deleted from the habitual operation, a dialogue cluster is set for the habitual operation cluster, making it possible to flexibly respond to the change.

[0493] By the above process, it is possible to easily change an addictive operation cluster in which a plurality of operations are collectively performed as described below.

[0494] For example, suppose there exists a macro consisting of a habitual operation cluster that combines a number of habitual operations performed when going to bed and includes an operation to turn off the air conditioner when the spoken phrase is "good night."

[0495] If such a macro exists, and the user says "I'm going to sleep without turning off the air conditioner" because it is getting hot, the "turn off the air conditioner" operation will be deleted from the habitual operation cluster and the confirmation dialogue flag f a is set to true and recorded as a conversation cluster.

[0496] Then, from the next time onwards, when the user utters "good night", for example, a question such as "Do you want to turn off the air conditioner?" is uttered based on the confirmation dialogue flag. If the user responds with a refusal to execute the operation such as "I don't want to turn off the air conditioner", the confirmation information I a The probability of execution decreases.

[0497] Furthermore, the execution refusal response continues, and confirmation information I for turning off the air conditioner a When the execution probability falls below a lower limit specified value, the "operation to turn off the air conditioner" is deleted from the macro consisting of the habitual operation cluster, and thereafter, the macro is executed without turning off the air conditioner without a query when the user utters "good night."

[0498] In other words, by performing the above-described operations, when the user utters an utterance such as "turning on the air conditioner while going to sleep" as a habitual operation in conjunction with the change in season (getting hotter), it is possible to change the habitual operation cluster so that the macro is executed without performing the "operation to turn off the air conditioner."

[0499] <Operations that depend on the observation context> Next, operations that depend on the observation context will be described with reference to FIG.

[0500] Consider the case where an habitual operation cluster made up of operations 1 to 8 shown in state ST181 shown on the left side of FIG. 19 is generated and executed by the habitual operation clustering process described above.

[0501] Incidentally, state ST181 is the same as state ST101 in FIG.

[0502] Here, for the departure station input in operation 3 among operations 1 to 8 shown in state ST181 on the left side of FIG. 19, c The location attribute is set as follows.

[0503] Dependent Context Type D c is information that specifies the type of context information used when the habitual operation is executed.

[0504] In this way, dependent context type D c When the location attribute is set as, for example, the location information detected (observed) by a GPS sensor or the like is stored in the observation context information C o When the Entity is matched by comparison with the Entity of operation 3, the context interaction flag f is set as shown in state ST182. c is set.

[0505] Context interaction flag f cFor example, among the operations that are performed collectively by the habitual operation cluster, the dependent context type D c The entity of the operation for which the value is set and the observation context information C o This is a flag that sets whether or not to realize a process of presenting operation options to the user through dialogue and allowing the user to select an option when an operation different from the above is performed.

[0506] For example, the vicinity of “Osaki”, which is the entity of the operation “entering “Osaki” as the departure station” in operation 3, is included in the observation context information C o In the case where the position information is detected as the position information in the state ST182 of FIG. 19, the context interaction flag f c is set to true.

[0507] Also, the context dialogue flag f c When is set to true, the context information I c is set to the initial value.

[0508] Here, the context information I c The dependent context type D is preset as the initial value. c The “Location” is set as the number of detections, the total number of detections is set to 2 (total:2), and the observation context information C o Number of inputs (C o :1) is set to 1, and the number of times the Entity inputs Osaki is set to 1 (entity:{Osaki:1}).

[0509] After that, when the user inputs the departure station, the total number of times of the context information Ic, the input entity, or the observed context information C o is incremented by 1, and the input probability of the option (input count / total number of times) is calculated. Note that, in the initial value, the total number of times is set to 2 so that the input probability of the number of times the observation context information Co is input and the input probability of the number of times the Entity is input are the same.

[0510] Therefore, in state ST182, the input probability of each of the observation context information Co and Osaki is 0.5.

[0511] <Observation context dependent operation determination processing> Next, the observation context dependent operation determination process will be described with reference to the flowchart in Fig. 20. Note that the observation context dependent operation determination process in the flowchart in Fig. 20 is the process of step S56 in the flowchart in Fig. 9.

[0512] In step S221, the operation memory execution control unit 53 stores the dependency context type D in the Intent attribute of the operation n. c Determine whether or not is set.

[0513] In step S221, the Intent attribute of the operation n includes a dependent context type D c If it is determined that the setting is set, the process proceeds to step S222.

[0514] In step S222, the operation memory execution control unit 53 stores the dependent context type D c The corresponding observation context information C o Determine whether or not the Entity of operation n matches the Entity.

[0515] In step S222, the dependent context type D c The corresponding observation context information C o If it is determined that the Entity of the operation n matches the Entity of the operation n, the process proceeds to step S223.

[0516] In step S223, the operation memory execution control unit 53 stores context information I c Set the initial value of .

[0517] In step S224, the operation memory execution control unit 53 sets the context dialogue flag f c Set the context interaction flag f cSet to true.

[0518] By the above process, the preset dependent context type D c When the observed context information Co of the operation n matches the Entity of the operation n, the context information I c The initial values ​​of the number of inputs of the observation context information Co and the number of inputs of the entity are each set to 1, the total number is set to 2, and the context interaction flag f c is set to true.

[0519] <Processing for updating context information of dialogue cluster candidates> Next, the context information update process for a dialogue cluster candidate will be described with reference to the flowchart in Fig. 21. Note that the context information update process for a dialogue cluster candidate in the flowchart in Fig. 21 is the process of step S72 in the flowchart in Fig. 9.

[0520] In step S251, the operation storage execution control unit 53 sets an unprocessed operation among the operations of the stored cluster Z in the same sequence as the dialogue cluster candidate as an operation m to be processed.

[0521] In step S252, the operation memory execution control unit 53 sets the context dialogue flag f c is true and the context interaction flag f of operation m in the stored cluster Z c Determine whether or not is true.

[0522] In step S252, the context dialogue flag f of the operation m in the dialogue cluster candidate is c is true and the context interaction flag f of operation m in the stored cluster Z c If it is determined that YES, processing proceeds to step S253.

[0523] In step S253, the operation memory execution control unit 53 calculates the context information I c Entity E c and the context information I of operation m in the stored cluster Z. c The Entity E with the highest input probability of max It is determined whether or not the and match.

[0524] In step S253, the context information I c Entity E c and the context information I of operation m in the stored cluster Z. c The Entity E with the highest input probability of max If it is determined that they do not match, the process proceeds to step S254.

[0525] In step S254, the operation memory execution control unit 53 stores the context information I c Observation context information C o Increment the input count by 1.

[0526] In step S255, the operation memory execution control unit 53 stores the context information I c The total number of times is incremented by 1.

[0527] In step S256, the operation memory execution control unit 53 stores the context information I c The context information I c Set to.

[0528] That is, the context information I c Entity E c and the context information I of operation m in the stored cluster Z. c The Entity E with the highest input probability of maxIf they do not match, the context information I c will be updated.

[0529] In step S253, the context information I c Entity E c and the context information I of operation m in the stored cluster. c The Entity E with the highest input probability of max If it is determined that they match, the processes of steps S254 and S255 are skipped, and the process proceeds to step S256.

[0530] That is, the context information I c Entity E c and the context information I of operation m in the stored cluster. c The Entity E with the highest input probability of max If they match, the context information I c is the context information I of the stored cluster Z. c It is set to the same information as

[0531] In step S257, the operation storage execution control unit 53 determines whether or not there is an unprocessed operation among the operations of the stored cluster Z in the same sequence as the dialogue cluster candidate.

[0532] If it is determined in step S257 that there is an unprocessed operation among the operations of the stored cluster Z in the same sequence as the dialogue cluster candidate, the process returns to step S251.

[0533] That is, the processes of steps S251 to S257 are repeated until all operations of the stored cluster Z in the same sequence as the dialogue cluster candidate have been processed.

[0534] Then, in step S257, if it is determined that there are no unprocessed operations among the operations of the stored cluster Z in the same sequence as the dialogue cluster candidate, or if it is deemed that all operations of the stored cluster Z in the same sequence as the dialogue cluster candidate have been processed, the process proceeds to step S258.

[0535] In step S258, the operation memory execution control unit 53 deletes the stored cluster Z having the same sequence as the dialogue cluster candidate, and ends the process.

[0536] By the above process, if there is a stored cluster Z with the same sequence as the dialogue cluster candidate, the context information I c By this, the context information of the dialogue cluster candidate I c will be updated.

[0537] Next, with reference to FIG. 22, a dialogue generation example for an observation context dependent operation will be described.

[0538] For example, consider the case where a dialogue cluster as shown in state ST201 in Fig. 22 is set. Note that the dialogue cluster in state ST201 is the same as the dialogue cluster in state ST182 in Fig. 19.

[0539] In operation 3 in the left part of FIG. 22, the context dialogue flag f c is set to true, so that the execution will be performed by the interactive cluster.

[0540] Here, for example, as shown in the right part of Figure 22, in process SC141, when the user 11 utters an utterance such as "transfer information," the operation memory execution control unit 53 recognizes that an instruction has been given to execute the dialogue cluster shown in the left part of Figure 22 relating to transfer information.

[0541] In process Ex141, the operation memory execution control unit 53 executes the macro of operations 1 and 2 shown in the left part of Fig. 22 in response to the utterance of "transfer guide." Also, it is assumed here that "Shinagawa" is detected as the observation context information Co.

[0542] In process Ex142, the operation memory execution control unit 53 executes the process of operation 3 and sets the context dialogue flag f c is set to true, so the context information I c Based on this, processing is performed.

[0543] More specifically, here, “Shinagawa” is detected as the observed context information Co, and “Shinagawa” is detected as the context information I c Therefore, the operation memory execution control unit 53 presents the options in the Entity in the order of the highest input probability. Note that, when the input probabilities are the same, the option most recently input may be presented first.

[0544] Then, in process Ex143, the operation memory execution control section 53 outputs a voice prompting the selection of the presented option, such as "Is Shinagawa the departure station?".

[0545] Here, in process SC142, when the user utters an utterance such as "Shinagawa is fine," the operation memory execution control unit 53 recognizes that "Shinagawa" has been selected as an option.

[0546] Therefore, in process Ex144, the operation memory execution control unit 53 inputs Shinagawa into the departure station and stores the context information I c Update and perform operations 4 to 8.

[0547] That is, the context dialogue flag f c If true, the context information I c Entity E with the highest input probability among the entities in max (Osaki) and current observation context information Co When (Shinagawa) does not match, alternative entities are presented, a query is spoken, and the macro is executed based on the selected entity.

[0548] Also, context information I c Entity E with the highest input probability among the entities in max (Osaki) and current observation context information C o If (Osaki) matches, Entity E is executed without any query to the user. max The macro is executed by (Osaki).

[0549] Furthermore, if a speech input other than the presented options is made, the context information I c will be added and updated.

[0550] Context Information I c Observation context information in C o When the input probability of is equal to or greater than a certain value (e.g. 0.8), the context dialogue flag f c is set to false, and the observation context information C o The macro with the entity fixed is executed.

[0551] Also, context information I c When the input probability of a specific entity in the context is equal to or greater than a certain value (e.g. 0.8), the context dialogue flag f c is set to false, and from then on a fixed macro is executed for entities whose input probability is greater than or equal to the specified value.

[0552] Through this series of processes, a change occurs in the habitual operation, and the observed context information Co is registered as the Entity E with the highest probability of input. max If they are different, the Entities are presented as options, a dialogue is displayed prompting the user to select one of them, and processing is performed according to the selection result.

[0553] As a result, even if a change occurs in the context accompanying the habitual operation, a dialogue cluster is set for the habitual operation cluster, enabling a flexible response.

[0554] <Interaction processing for observation context-dependent operations> Next, the interactive processing for the observation context dependent operations will be described with reference to the flowchart of FIG.

[0555] This process is performed according to the context dialogue flag f c This process is executed when an operation in which the value of "true" is set is performed. That is, for example, this process is executed when the process of operation 3 in state ST201 in FIG.

[0556] In step S281, the operation memory execution control unit 53 determines the current preset dependency context type D c Observation context information C o and context information I c Entity E with the highest input probability among the entities in max It is determined whether or not the above matches.

[0557] In step S281, the currently preset dependent context type D c Observation context information C o and context information I c Entity E with the highest input probability among the entities in max If it is determined that they match, the process proceeds to step S282.

[0558] In step S282, the operation memory execution control unit 53 determines the current preset dependency context type D c Observation context information C o Execute operation n using as Entity.

[0559] On the other hand, in step S281, the currently preset dependent context type D c Observation context information Co and context information I c Entity E with the highest input probability among the entities in max If it is determined that they do not match, the process proceeds to step S283.

[0560] In step S283, the operation memory execution control unit 53 outputs a system utterance that prompts the user to select (respond) to one of the options, based on a query phrase that is registered in advance for each Intent.

[0561] In step S284, the operation memory execution control unit 53 determines whether or not a response utterance has been made as a result of the user's selection of an option, and repeats the same process until it is determined that a response utterance has been made.

[0562] Then, in step S284, if it is determined that a response utterance has been made as a result of the user's selection of an option, the process proceeds to step S285.

[0563] In step S285, the operation memory execution control unit 53 executes the operation n by the Entity of the option corresponding to the response utterance.

[0564] In step S286, the operation memory execution control unit 53 stores the observation context information C o Determine whether or not is an entity of the user's response utterance.

[0565] In step S286, the observation context information C o If it is determined that the Entity is the user's response utterance, the process proceeds to step S287.

[0566] In step S287, the operation memory execution control unit 53 receives the context information I c Observation context information C contained in o The number of inputs is incremented by 1, and the process proceeds to step S291.

[0567] On the other hand, in step S286, the observation context information C o If it is determined that the Entity is not an Entity of the user's response utterance, the process proceeds to step S288.

[0568] In step S288, the operation memory execution control unit 53 checks whether the Entity selected by the response utterance is included in the context information I c Determine whether the entity is contained within.

[0569] In step S288, the Entity selected by the response utterance is added to the context information I c If it is determined that the entity is included in the object, the process proceeds to step S289.

[0570] In step S289, the operation memory execution control unit 53 stores the context information I c The input count information of the Entity selected by the user is incremented by 1.

[0571] On the other hand, in step S288, the Entity selected by the response utterance is added to the context information I c If it is determined that the object is not an Entity contained within the object, the process proceeds to step S290.

[0572] In step S290, the operation memory execution control unit 53 receives the context information I c The number of times the Entity selected by the user is entered is newly added and set as an initial value of 1.

[0573] In step S291, the operation memory execution control unit 53 receives the context information I c The total number of times is incremented by 1.

[0574] In step S292, the operation memory execution control unit 53 receives the context information I c Observation context information C o It is determined whether the input probability of is greater than or equal to a specified value.

[0575] In step S292, the context information I c Observation context information C o If it is determined that the input probability of is equal to or greater than the specified value, the process proceeds to step S293.

[0576] In step S293, the operation memory execution control unit 53 sets the context dialogue flag f c is set to false, and the observation context information C o Set it to the operation that fixes the Entity and terminates the process.

[0577] In step S292, the context information I c If it is determined that the input probability of the observation context information Co in is not equal to or greater than the specified value, the process proceeds to step S294.

[0578] In step S294, the operation memory execution control unit 53 receives the context information I c Determine whether there is an entity in whose input probability is greater than or equal to a specified value.

[0579] In step S294, the context information I c If it is determined that there is an entity in the list whose input probability is equal to or greater than a prescribed value, the process proceeds to step S295.

[0580] In step S295, the operation memory execution control unit 53 sets the context dialogue flag f c is set to false, and the operation is set to fix the entity whose input probability is equal to or greater than a specified value, and the process is terminated.

[0581] In step S295, the context information I c If there is no entity in which the input probability is equal to or greater than the specified value, the process of step S295 is skipped and the process ends.

[0582] Through the above process, the context dialogue flag f c For operations where is set to true, the observation context information C o is the context information I c Entity E is the maximum value of the entities in max Then, Entity E max The operation is executed with this as the Entity.

[0583] In addition, the observation context information C o is the context information I c Entity E is the maximum value of the entities in max If not, alternative entities are presented in the order of the input probability, the user is prompted to select one, and the process is executed using the selected entity.

[0584] Then, for the selected Entity, the count is incremented by 1, and the Total count is incremented by 1. At this time, if a new Entity other than the Entities presented as options is selected, it is registered as a new option and the count is set to 1.

[0585] Furthermore, the observation context information C o The input probability of each of and Entity is calculated sequentially, and if it exceeds a specified value, the user can input the observation context information C o Or, as an entity, it is assumed that a number of options greater than the specified value will always be selected, and the context dialogue flag f c is set to false, and the input probability of being selected is always equal to or greater than the specified value. o Or it is set to an operation fixed to the Entity.

[0586] That is, the observation context information C o Or, once an entity has a fixed operation, it will function as a normal habitual operation cluster from then on, with no further interaction taking place.

[0587] As a result, even if a change occurs in the habitual operation, a dialogue cluster is set for the habitual operation cluster, making it possible to flexibly respond to the change.

[0588] (Application example 1) In the above, dependent context type D c Although the example in which the dependent context type is location (location information) has been described, other types are also possible. For example, c may be set to dialog.

[0589] For example, dependent context type D c When the command is set to "dialog", the input operation of the arrival station for the transfer guide may be executed based on the immediately preceding dialogue.

[0590] In other words, in this case, the content of the conversation that took place within a specified time from the timing when the habitual operation cluster is executed, as well as the content of communication such as chat, messages, and SNS, is used as context information obtained from the immediately preceding conversation, and the arrival station input operation for the transfer guide is performed.

[0591] More specifically, dependent context type D c If the context dialogue flag f is set to dialog, when the arrival station input = Yokohama is included as a topic in the previous chat, c is set to true, and the context information c Yokohama will be stored in the entity.

[0592] That is, in this case, the dependent context type D of the arrival station location c is considered to be the previous dialog, not the location.

[0593] Therefore, for example, when a macro consisting of a habitual operation cluster is executed, the observed context information Co obtained based on the contents of the immediately preceding chat is added to the context information Ic When "Ikebukuro" is included instead of "Yokohama" registered as an entity in the observation context information C, the arrival station is o An utterance is made to inquire as to whether Ikebukuro should be selected, and an arrival station input operation is performed in response to the user's response, and the macro is executed.

[0594] (Application example 2) Dependent Context Type D c may be vital system context information.

[0595] That is, dependent context type D c For example, the pulse rate range may be set in vital context information such as a pulse indicating a pulse rate value, and when a macro including a music playlist selection operation is present, the pulse rate range may be set in the playlist as meta information.

[0596] In this way, for example, calming music is set in a playlist corresponding to meta information in a low pulse rate range, and music suitable for running is set in a playlist corresponding to meta information in a high pulse rate range.

[0597] Then, when a playlist is selected, the pulse rate range observation context information C o If a pulse as a is observed (detected), the context dialogue flag f c is set to true, and the context information I c The pulse range of the selected playlist is stored in the entity's meta information.

[0598] When a macro including a music playlist selection operation is executed, the observed context information C o The pulse value is the context information I c If the pulse rate is outside the range stored in the memory, the user is asked whether or not to execute a play list selection operation.

[0599] Observation context information C during cluster execution o Alternatively, playlists having meta information in which the pulse value is within the pulse range may be presented as options for an inquiry.

[0600] In addition, the dependent context type D in the playlist selection operation c For example, the "emotion" may be set to "emotion" indicating an emotion.

[0601] Here, dependent context type D c When is set to emotion, the context information is, for example, an emotion recognition result based on the user's image recognition result, pulse rate value, blood pressure value, etc., and a spoken inquiry may be made to prompt the user to select a playlist according to the emotion recognition result.

[0602] (Application example 3) Dependent Context Type D c An attribute indicating an attribute of a person in the vicinity may be set in the .

[0603] More specifically, when a macro for selecting a video content to be viewed is present, the dependent context type D c When the context information is set to attribute, the context information is set to the attribute of a person in the vicinity of the user who performs the operation of selecting video content to be viewed.

[0604] For example, if it is possible to set age groups as attributes of people around the user, the content that is available for selection in the viewing video content selection operation will have a target age range set as meta information, such as for children (up to 12 years old) or 18 years old and above.

[0605] As a result, when a macro consisting of a viewing video content selection operation is executed, the target age range set as meta information for the selected content is added to the dependent context type D consisting of the age of people around the user recognized from the image. cObservation context information C o If detected, the context interaction flag f c is set to true and the context information I c The target age range of the selected content is stored in the meta information as an entity.

[0606] Then, when the macro consisting of the video content selection operation is executed, the observation context information C consisting of the ages of people around the user who have been recognized by image recognition is o is the context information I c If the age is outside the target age range stored in the user terminal, when a viewing video content selection operation is performed, the user may be prompted to speak to inquire whether it is okay to play the video even though there is a person outside the target age nearby.

[0607] For example, when executing a macro consisting of an operation for selecting video content to be viewed, a screen may be displayed asking whether it is okay to select video content for children that is habitually viewed together with the child even when the child is not present.

[0608] (Other application examples) Dependent Context Type D c For example, the observed context information C o Depending on the online / offline status, a user who is online may be considered to be able to receive and read messages immediately, and a user who is offline may be considered not to be able to read messages immediately.

[0609] In this case, for example, the observation context information C o Depending on the online / offline information, if a user who can immediately receive and read a message is specified as a message destination, no inquiry dialogue may be performed, and if a user who cannot immediately read a message is specified as a message destination, an inquiry dialogue may be performed asking whether it is okay to include the user in the destination.

[0610] Also, dependent context type D c may be the relationship with the people in the place, for example, observation context information C o Depending on the relationship with the people present at the location, inputs and presentation-related operations related to privacy may be set to ON or OFF.

[0611] In addition, dependent context type D c may be the time period when a query such as "what's the weather?" is made, and the observation context information C o Depending on the time of day, for example, if it is morning, the weather for today may be displayed, and if it is night, the weather for tomorrow may be displayed.

[0612] (Variation 1) A dialogue flag may also be set for modifying / adding operations before / after a macro consisting of an addictive operation cluster is executed.

[0613] For example, if the user requests "set the departure station to Shinagawa and follow the usual transfer guide," the selection dialogue flag f s may be set.

[0614] In addition, after the macro is executed by uttering "the usual transfer guide", if the user utters "try again via Yokohama", the operation executed by "try again via Yokohama" is regarded as an additional operation, and the confirmation dialogue flag f a may be set so that the user is prompted as to whether or not an additional operation is required.

[0615] (Variation 2) When using the control function of a home appliance by linking with a device such as a smart speaker, if a specific operation in a macro consisting of a habitual operation cluster cannot be executed due to context dependency and an error occurs, the dependent context type D that causes the error cIf it is possible to change the parameter, it is possible to change it without uttering a query, thereby suppressing the occurrence of errors while executing the macro.

[0616] That is, for example, dependent context type D indicating the power state of an air conditioner c When a macro including an air conditioner temperature setting operation is executed based on the context information, the air conditioner may be kept powered off and the air conditioner may be turned on when the air conditioner temperature setting operation is performed when the macro is executed, thereby enabling the air conditioner temperature setting operation to be realized.

[0617] (Variation 3) It is also assumed that heterogeneous and multiple dialogue flags are set for a specific operation by the dialogue clustering process described with reference to the flowchart in FIG. 9, and multiple spoken inquiries are made when the specific operation is executed.

[0618] In such a case, depending on the content of the response to the user's inquiry, it is possible that the subsequent sequence may cause a functional inconsistency.

[0619] This inconsistency will be resolved after the operation is performed several times as the user's response habits will be reflected, but it is also possible to not ask depending on whether there is a subsequent sequence inconsistency when the user responds.

[0620] That is, for example, if there is a macro that provides transfer guidance by selecting the departure station as Osaki, either Yokohama or Hon-Atsugi as the arrival station via a selection dialogue flag, and confirming that the intermediate station is Yokohama via a confirmation dialogue flag, if the user selects Yokohama as the arrival station and then as the intermediate station at execution time, an inconsistency will occur because the arrival station and the intermediate station will be the same.

[0621] Therefore, when Yokohama is selected as the arrival station in this way, it is inconsistent for the intermediate station to also be Yokohama, so it is possible not to make an inquiry as to whether or not the intermediate station should be Yokohama.

[0622] As described above, in the present disclosure, operations that are highly habitual and do not require change are grouped together as a single macro to be executed collectively as a habitual operation cluster, and operations that may be changed depending on the situation, etc. are executed based on the response to an inquiry.

[0623] This allows the user to change the habitual operation clusters that form the macros through optimal interactions that are sensible to the user.

[0624] In addition, if the user's response to a query for a specific operation is deemed to be a habitual response, the probability information (input probability or execution probability) in the dialogue cluster is updated, and when it becomes greater than a specified value, an operation is fixed to the habitual response and the query is no longer made, allowing the user to make changes to multiple operations and execute them all at once without having to engage in unnecessary dialogue.

[0625] Furthermore, queries are made only when necessary depending on the user's current situation (context), enabling users to make appropriate changes to the operation content with little load even in situations that differ from those in which they are accustomed to performing composite operations.

[0626] <<3. Example of execution by software>> The above-mentioned series of processes can be executed by hardware, but can also be executed by software. When the series of processes are executed by software, the programs constituting the software are installed from a recording medium into a computer built into dedicated hardware, or into, for example, a general-purpose computer capable of executing various functions by installing various programs.

[0627] 24 shows an example of the configuration of a general-purpose computer. This personal computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0628] An input unit 1006 consisting of input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 for outputting a processing operation screen and an image of the processing result to a display device, a storage unit 1008 consisting of a hard disk drive for storing programs and various data, and a communication unit 1009 consisting of a LAN (Local Area Network) adapter and the like for executing communication processing via a network such as the Internet are connected to the input / output interface 1005. Also connected is a drive 1010 for reading and writing data from and to a removable storage medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0629] The CPU 1001 executes various processes according to a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory and installed in a storage unit 1008, and loaded from the storage unit 1008 to a RAM 1003. The RAM 1003 also stores data and the like necessary for the CPU 1001 to execute various processes, as appropriate.

[0630] In a computer configured as described above, the CPU 1001 loads a program stored in the memory unit 1008, for example, into the RAM 1003 via the input / output interface 1005 and the bus 1004, and executes the program, thereby performing the series of processes described above.

[0631] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0632] In the computer, the program can be installed in the storage unit 1008 via the input / output interface 1005 by mounting the removable storage medium 1011 in the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be pre-installed in the ROM 1002 or the storage unit 1008.

[0633] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0634] In addition, the CPU 1001 in FIG. 24 realizes the functions of the operation memory execution control unit 53 in FIG. 4, and the memory unit 1008 and the removable storage medium 1011 realize the functions of the user operation history memory unit 54 and the habitual operation / dialogue cluster memory unit 55.

[0635] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0636] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present disclosure.

[0637] For example, the present disclosure can take the form of cloud computing in which a single function is shared and processed collaboratively by multiple devices via a network.

[0638] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0639] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0640] The present disclosure can also be configured as follows.

[0641] <1> Calculating a habit score indicating the habit of the user operation; a control unit that registers a cluster including a plurality of operations including an operation having the habitual score higher than a predetermined threshold and an operation corresponding to a dialogue with a user; An information processing device comprising: <2> the operation in response to the dialogue with the user is an operation having the same operation type but different attributes, The control unit sets, by a selection dialogue flag, whether or not to execute a dialogue related to selection of an attribute in an operation in which the operation type is the same but the attribute is different. <1> The information processing device described in <3> When executing a dialogue related to the selection of the attribute, the control unit sets the selection dialogue flag to true, and sets information on the attribute that is an option in the dialogue as option information. <2> The information processing device described in <4> The control unit executes the dialogue by presenting to the user the attributes that are the options in the option information, and executes an operation based on the attributes that correspond to the option selected by the user. <3> The information processing device described in <5> The control unit sets the number of times each of the options in the option information is selected by the user as a selection count, and sets a total number of all the selection counts. <4> The information processing device described in <6> The control unit calculates an input probability for each of the options based on the number of selections and the total number of selections for each option in the option information, performs the dialogue by presenting the attributes to the user in an order according to the input probabilities, and performs an operation based on the attributes corresponding to the option selected by the user. <5> The information processing device described in <7> When the input probability of any of the options becomes greater than a specified value, the control unit sets the selection dialogue flag to false and executes an operation of fixing the attribute of the option having the input probability greater than the specified value. <6> The information processing device described in <8> the operation corresponding to the interaction with the user is an operation including an added operation and a deleted operation among a plurality of operations constituting the registered cluster, The control unit sets whether to execute a dialogue for confirming whether to execute the added operation or the deleted operation by a confirmation dialogue flag. <1> The information processing device described in <9> When executing a dialogue to confirm the execution of the added operation or the deleted operation, the control unit sets the confirmation dialogue flag to true, and sets the number of times the added operation or the deleted operation was executed as a result of the confirmation by the dialogue and the total number of times the confirmation was performed as confirmation information. <8> The information processing device described in <10> The control unit executes a dialogue for confirming execution of the added operation or the deleted operation, and executes the operation in response to the user's response. <9> The information processing device described in <11> The control unit calculates an execution probability of the added operation or the deleted operation in the confirmation information based on the execution count and the total count for the added operation or the deleted operation in the confirmation information. <10> The information processing device described in <12> The control unit sets the confirmation dialogue flag to false when the execution probability of the added operation or the deleted operation becomes greater than a specified value. <11> The information processing device described in <13> The control unit deletes the cluster for which the confirmation dialogue flag is set when the execution probability of the added operation or the deleted operation becomes smaller than a specified value. <11> The information processing device described in <14> the operation in response to the dialogue with the user is an observation context dependent operation that is an operation that depends on an observation context, The control unit sets whether or not to execute a dialogue related to a selection of an option in the observation context-dependent operation by using a context dialogue flag. <1> The information processing device described in <15> When executing a dialogue related to a selection of an option consisting of the observation context-dependent operation, the control unit sets the context dialogue flag to true, and sets information on an attribute dependent on the observation context, which is an option in the dialogue, and information on the observation context as context information. <14> The information processing device described in <16> The control unit executes the dialogue by presenting to the user an attribute that depends on the observation context and the observation context, which is the option in the context information, and executes an operation based on the attribute that corresponds to the option selected by the user. <15> The information processing device described in <17> The control unit sets the number of times each of the options in the context information is selected by the user as a selection count, and sets a total number of all the selection counts. <16> The information processing device described in <18> The control unit calculates an input probability for each of the options based on the number of times each option is selected in the context information and the total number of times, and executes the dialogue by presenting the attributes to the user in an order according to the input probabilities, and executes an operation based on the attributes corresponding to the option selected by the user. <17> The information processing device described in <19> When the input probability of any of the options becomes greater than a specified value, the control unit sets the context dialogue flag to false, and executes an operation to fix the attribute or the observation context that is the option having the input probability higher than the specified value. <18> The information processing device described in <20> The habit score is calculated based on a harmonic function of a time interval from a previous operation to a current operation, a number of transitions from the previous operation to the current operation, and a transition probability from the previous operation to the current operation. <1> ~ <19> 2. An information processing device according to claim 1 . <21> Calculating a habit score indicating the habit of the user operation; A cluster including a plurality of operations including an operation having a habitual score higher than a predetermined threshold and an operation corresponding to a dialogue with the user is registered. An information processing method comprising: <22> Calculating a habit score indicating the habit of the user operation; a control unit that registers a cluster including a plurality of operations including an operation having the habitual score higher than a predetermined threshold and an operation corresponding to a dialogue with a user; A program that makes a computer function as a [Explanation of symbols]

[0642] 41 information processing device, 51 input operation unit, 52 functional application program execution unit, 53 operation memory execution control unit, 54 user operation history storage unit, 55 habitual operation / dialogue cluster storage unit, 56 voice input unit, 57 voice recognition processing unit, 58 utterance meaning understanding processing unit, 59 image input unit, 60 image recognition processing unit, 61 sensor input unit, 62 sensor recognition unit, 63 response generation unit, 64 display image processing unit, 65 voice synthesis processing unit, 66 image output unit, 67 voice output unit

Claims

1. Calculating a habit score indicating the habit of the user operation; a control unit that registers a cluster including a plurality of operations including an operation having the habitual score higher than a predetermined threshold and an operation corresponding to a dialogue with a user; the operation in response to the dialogue with the user is an operation having the same operation type but different attributes, The control unit sets, by a selection dialogue flag, whether or not to execute a dialogue related to selection of an attribute in an operation in which the operation type is the same but the attribute is different. Information processing device.

2. When executing a dialogue related to the selection of the attribute, the control unit sets the selection dialogue flag to true, and sets information on the attribute that is an option in the dialogue as option information. The information processing device according to claim 1 .

3. The control unit executes the dialogue by presenting the attributes that are the options in the option information to the user, and executes an operation based on the attributes that correspond to the option selected by the user. The information processing device according to claim 2 .

4. The control unit sets the number of times each of the options in the option information is selected by the user as a selection count, and sets a total number of all the selection counts. The information processing device according to claim 3 .

5. The control unit calculates an input probability for each of the options based on the number of selections and the total number of selections for each option in the option information, performs the dialogue by presenting the attributes to the user in an order according to the input probabilities, and performs an operation based on the attributes corresponding to the option selected by the user. The information processing device according to claim 4.

6. When the input probability of any of the options becomes greater than a specified value, the control unit sets the selection dialogue flag to false and executes an operation of fixing the attribute of the option having the input probability higher than the specified value. The information processing device according to claim 5 .

7. the operation in response to the dialogue with the user is an operation including an added operation and a deleted operation from the plurality of operations constituting the registered cluster, The control unit sets whether to execute a dialogue for confirming whether to execute the added operation or the deleted operation by a confirmation dialogue flag. The information processing device according to claim 1 .

8. When executing a dialogue to confirm the execution of the added operation or the deleted operation, the control unit sets the confirmation dialogue flag to true, and sets the number of times the added operation or the deleted operation was executed as a result of the confirmation by the dialogue and the total number of times the confirmation was performed as confirmation information. The information processing device according to claim 7.

9. The control unit executes a dialogue for confirming execution of the added operation or the deleted operation, and executes the operation in response to the user's response. The information processing device according to claim 8.

10. The control unit calculates an execution probability of the added operation or the deleted operation in the confirmation information based on the execution count and the total count for the added operation or the deleted operation in the confirmation information. The information processing device according to claim 9.

11. The control unit sets the confirmation dialogue flag to false when the execution probability of the added operation or the deleted operation becomes greater than a specified value. The information processing device according to claim 10.

12. The control unit deletes the cluster for which the confirmation dialogue flag is set when the execution probability of the added operation or the deleted operation becomes smaller than a specified value. The information processing device according to claim 10.

13. the operation in response to the dialogue with the user is an observation context dependent operation that is an operation that depends on an observation context, The control unit sets whether or not to execute a dialogue related to a selection of an option in the observation context-dependent operation by using a context dialogue flag. The information processing device according to claim 1 .

14. When executing a dialogue related to a selection of an option consisting of the observation context-dependent operation, the control unit sets the context dialogue flag to true, and sets information on an attribute dependent on the observation context, which is an option in the dialogue, and information on the observation context as context information. The information processing device according to claim 13.

15. The control unit executes the dialogue by presenting to the user an attribute that depends on the observation context and the observation context that is the option in the context information, and executes an operation based on the attribute that corresponds to the option selected by the user. The information processing device according to claim 14.

16. The control unit sets the number of times each of the options in the context information is selected by the user as a selection count, and sets a total number of all the selection counts. The information processing device according to claim 15.

17. The control unit calculates an input probability for each of the options based on the number of times each option is selected in the context information and the total number of times, and executes the dialogue by presenting the attributes to the user in an order according to the input probabilities, and executes an operation based on the attributes corresponding to the option selected by the user. The information processing device according to claim 16.

18. When the input probability of any of the options becomes greater than a specified value, the control unit sets the context dialogue flag to false, and executes an operation to fix the attribute or the observation context that is the option having the input probability higher than the specified value. The information processing device according to claim 17.

19. The habit score is calculated based on a harmonic function of a time interval from a previous operation to a current operation, a number of transitions from the previous operation to the current operation, and a transition probability from the previous operation to the current operation. The information processing device according to claim 1 .

Images