Operation assistance system, operation assistance method, and operation assistance program

Abstract

An operation assistance system according to one embodiment of the present invention comprises a surrounding information acquisition unit, a surrounding information storage unit, a dialog interface unit, and a dialog scenario control unit. The surrounding information acquisition unit acquires, as surrounding information, any one or more of information pertaining to an application running on a terminal, information pertaining to terminal control, and information acquirable from a sensor provided in the terminal. The surrounding information storage unit stores surrounding information obtained by the surrounding information acquisition unit. The dialog interface unit receives information input by a user and outputs information to the user. When the surrounding information stored in the surrounding information storage unit and the information inputted to the dialog interface unit satisfy a determination condition, the dialog scenario control unit causes the dialog interface unit to output information indicating a plurality of operation proposals preliminarily associated with the determination condition, for each assumed use scene in a lump.

Description

Operation support system, operation support method, and operation support program

[0001] One aspect of the present invention relates to an operation assistance system, an operation assistance method, and an operation assistance program.

[0002] Conventionally, there are dialogue programs called "chatbots" that automatically respond to inputs such as text and voice from users. Chatbots automatically respond to user inputs according to predetermined rules. In this way, chatbots reduce the amount of manual response work that was previously done in customer support and other tasks, thereby contributing to improved productivity and reduced workloads.

[0003] Meanwhile, there is known a technology that not only responds with simple text sentences but also executes Robotic Process Automation (RPA) by distributing data input to a chatbot to an RPA tool to automate user operations. Patent Document 1 discloses a chatbot-type RPA in which a next operation suggestion according to the operation status is displayed as a button, eliminating the need to input a character string.

[0004] In such operation support by suggesting the next operation to the user according to the operation situation, as the number of operation suggestion candidates increases, it becomes difficult for the user to accurately and efficiently select the next operation.Furthermore, there is also the problem that although the operation suggestions are related to each other, it is not possible to read these relationships.

[0005] International Publication No. 2021 / 070292

[0006] The present invention provides an operation support technique that can present a large number of operation suggestions to a user so that the user can easily select an operation.

[0007] In order to solve the above problems, an operation assistance system according to one embodiment of the present invention includes a peripheral information acquisition unit, a peripheral information storage unit, a dialogue interface unit, and a dialogue scenario control unit. The peripheral information acquisition unit acquires, as peripheral information, one or more of information related to applications running on the terminal, information related to terminal control, and information obtainable from sensors equipped in the terminal. The peripheral information storage unit stores the peripheral information acquired by the peripheral information acquisition unit. The dialogue interface unit accepts information input by a user and outputs information to the user. When the peripheral information stored in the peripheral information storage unit and the information input to the dialogue interface unit satisfy a determination condition, the dialogue scenario control unit compiles information indicating a plurality of operation suggestions pre-associated with the determination condition for each expected usage scenario and outputs the information to the dialogue interface unit.

[0008] According to one aspect of the present invention, information indicating operation suggestions is output in a consolidated manner for each anticipated usage scenario, thereby providing an operation assistance technology that can present a large number of operation suggestions to the user to make a selection easily.

[0009] FIG. 1 is a diagram showing an example of the configuration of an operation support system according to a first embodiment of the present invention. FIG. 2 is a diagram for explaining an operation specification phase. FIG. 3 is a diagram for explaining an operation execution phase. FIG. 4 is a flowchart showing an example of the processing flow of the operation specification phase. FIG. 5 is a flowchart showing an example of the processing flow of the operation execution phase. FIG. 6 is a diagram showing a modified example of a summary display form on an interactive interface screen. FIG. 7 is a diagram showing a modified example of a summary display form on an interactive interface screen. FIG. 8 is a diagram showing a modified example of a summary display form on an interactive interface screen. FIG. 9 is a diagram showing an example of the configuration of an operation support system according to a second embodiment of the present invention. FIG. 10 is a diagram showing an example of stored content in a display form definition information storage unit. FIG. 11 is a diagram showing an example of a display transition related to an operation of changing the summary display form. FIG. 12 is a diagram showing an example of a display on an interactive interface screen before and after changing the summary display form. FIG. 13 is a diagram showing an example of stored content in the display form definition information storage unit before and after changing the summary display form. FIG. 14 is a flowchart showing an example of the processing flow of the operation specification phase. FIG. 15 is a flowchart showing an example of a processing flow of changing the summary display form. 16 is a diagram showing an example of a display transition related to an operation for changing the summary display form during development. FIG. 17 is a diagram showing an example of a computer that executes an operation support program.

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [First embodiment] (Configuration example) First, the configuration of an operation assistance system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the operation assistance system according to the first embodiment. In this embodiment, the operation assistance system is realized by a terminal 10 operated by a user.

[0012] 1, the terminal 10 has a dialogue interface unit 11, a chatbot 12, a first application 13a, an OS (Operating System) 13b, a sensor 13c, and a second application 14. The chatbot 12 also has a peripheral information acquisition unit 121, a peripheral information accumulation unit 122, a dialogue scenario control unit 123, an operation control unit 124, and an operation assistance definition information storage unit 125. The operation assistance definition information storage unit 125 may be provided outside the chatbot 12.

[0013] The dialogue interface unit 11 accepts information input by the user and outputs information to the user. Here, for example, the dialogue interface unit 11 displays a chat-style screen having an input field and an output field on the display or the like of the terminal 10. The dialogue interface unit 11 accepts text input via a user's operation on an input device such as a keyboard. The dialogue interface unit 11 may also accept voice input from a microphone and convert the received voice into text. Note that the input device here includes not only physical devices but also virtual keyboards displayed on a touch panel display. User operations include operations such as tapping on a touch panel constituting the touch panel display and clicking with a mouse.

[0014] The chatbot 12 is a program that automatically responds with text or the like to text information or operation information from the dialogue interface unit 11. Furthermore, the chatbot 12 can execute operations (hereinafter referred to as "response operations") according to the user's operation situation, such as operations on applications provided in the terminal 10 or the terminal 10 itself, in accordance with the requests indicated in the input text information or operation information.

[0015] The first application 13a is one or more applications provided in the terminal 10. The first application 13a executes processing in response to a user operation or sensing by a predetermined sensor.

[0016] The second application 14 is one or more applications provided on the terminal 10. The second application 14 executes processing in response to control of the chatbot 12. Note that the second application 14 may overlap with the first application 13a, and like the first application, can execute processing in response to user operation or sensing by a predetermined sensor.

[0017] Here, the peripheral information acquisition unit 121 of the chatbot 12 acquires, as peripheral information, one or more of information about the first application 13a running on the terminal 10, information about terminal control, and information obtainable from sensors provided in the terminal 10. The peripheral information includes user operations on the first application 13a, information displayed by the first application 13a, etc. The peripheral information may also include information about terminal control, such as date and time, login name, information about applications running on the terminal 10, network connection information, and hardware information, which can be acquired from the OS 13b, etc. The peripheral information may also include acceleration and GPS information obtainable from the sensors 13c provided in the terminal 10. For example, whether a specific application is running, whether the terminal 10 has been restarted, etc., are information obtainable from the OS 13b and are examples of information about terminal control. The peripheral information storage unit 122 stores the peripheral information acquired by the peripheral information acquisition unit 121.

[0018] When the peripheral information stored in the peripheral information storage unit 122 and the information input to the dialogue interface unit 11 satisfy a predetermined condition, the dialogue scenario control unit 123 causes the dialogue interface unit 11 to output information previously associated with the condition. The dialogue scenario control unit 123 identifies a response operation to be executed according to the user operation situation by determining the condition, and causes the dialogue interface unit 11 to output information regarding the execution of the identified response operation as a candidate response operation to be executed next, i.e., as an operation suggestion. The response operation to be executed can include the execution of an application to be executed or the execution of an operation on the terminal 10 itself.

[0019] For example, the dialogue scenario control unit 123 causes the dialogue interface unit 11 to display text information indicating candidates for the next response operation to be executed. Also, for example, the dialogue scenario control unit 123 acquires text information and operation information input by the user from the dialogue interface unit 11.

[0020] The dialogue scenario control unit 123 also acquires peripheral information from the peripheral information storage unit 122. Furthermore, the dialogue scenario control unit 123 acquires, from the operation assistance definition information storage unit 125, determination conditions, which are conditions for determining the corresponding operation to be executed. Furthermore, the dialogue scenario control unit 123 can instruct the operation control unit 124 to execute a corresponding operation, such as executing the second application 14, in accordance with the determination result of the condition or a user input to the dialogue interface unit 11. In other words, the dialogue scenario control unit 123 can automatically instruct the execution of a corresponding operation in accordance with the determination result of the condition, even in the absence of a user operation. Furthermore, the dialogue scenario control unit 123 can acquire the execution result of the corresponding operation from the operation control unit 124.

[0021] The operation control unit 124 executes a corresponding operation, for example, the second application 14, using the peripheral information stored in the peripheral information storage unit 122 in response to an instruction from the dialogue scenario control unit 123. Furthermore, the operation control unit 124 obtains the execution result of the corresponding operation, for example, the execution result of the second application 14, and distributes it to the dialogue scenario control unit 123. The operation control unit 124 also obtains operation control information, which is information for executing the corresponding operation, from the operation assistance definition information storage unit 125.

[0022] Here, the processing by the operation assistance system will be described with reference to specific examples. FIG. 2 is a diagram illustrating the operation identification phase in which an operation to be performed is identified. First, the dialogue scenario control unit 123 acquires a determination condition for the operation status from the operation assistance definition information storage unit 125 and instructs the peripheral information acquisition unit 121 to start acquiring peripheral information to be acquired for determining the operation status. In response to the instruction from the dialogue scenario control unit 123, the peripheral information acquisition unit 121 acquires peripheral information from the first application 13a, etc., and stores the acquired peripheral information in the peripheral information storage unit 122. Here, it is assumed that none of the first applications 13a are running. For example, the peripheral information acquisition unit 121 can refer to the task management function of the OS 13b to check whether a specific application or operation is running.

[0023] In this case, the terminal 10 outputs a message to the user on the dialogue interface screen 111 saying, "Peripheral information: Currently running application (none) has been acquired." Note that the term "running application" used here does not only refer to an application running on the terminal 10, but also includes operations that the terminal 10 is performing without using an application. Since it is desirable for messages to the user to be easy to understand, this term is used in the message. The dialogue scenario control unit 123 controls the dialogue interface unit 11 to output this message by the terminal 10.

[0024] As shown in Fig. 2, the peripheral information stored in the peripheral information storage unit 122 includes items such as "date and time," "peripheral information," and "value." In the example of Fig. 2, the "launched app" in the peripheral information acquired on "2019 / 01 / 01 10:00" is shown to be "null." This indicates that no first application 13a was running at the time the collected information was acquired.

[0025] The dialogue scenario control unit 123 acquires peripheral information from the peripheral information accumulation unit 122. Then, the dialogue scenario control unit 123 determines the operation status of the terminal 10 based on the acquired information.

[0026] For example, the determination conditions for the operation status "start of work" are "Determination Condition 1: user input = null, Determination Condition 2: peripheral information, launched application = null, Response operations: normal time; attendance management application, email application, project management application, meeting; document creation application, conference application, chat application, event; posting site, document storage application, common; shutdown, restart, app termination." This condition indicates that when there is no input from the user to the dialogue interface unit 11 and no first application 13a is running, the attendance management application, email application, project management application, document creation application, conference application, chat application, posting site, document storage application, shutdown, restart, and app termination are presented as candidate response operations.

[0027] The dialogue system developer sets the judgment criteria by proposing response operations suitable for multiple possible usage scenarios for any given operation situation. In this example, four usage scenarios are assumed at the start of a workday: "normal" and "emergency" scenarios: "meeting," "event," and "common." For each usage scenario, candidate response operations are proposed based on the next operation to be performed. Here, the "time management app" as a candidate response operation to be performed during "normal" scenarios is an application for requesting work hours and breaks, the "email app" is an application for contacting other companies, and the "project management app" is an application for checking work details. The "document creation app" as a candidate response operation to be performed during the "meeting" scenario is an application for creating minutes, the "conference app" is an application for starting a meeting, and the "chat app" is an application for sending reminders. The "publication site" as a candidate response operation to be performed during the "event" scenario is an operation for opening a webpage and checking its contents, and the "document storage app" is an operation for checking the contents of past events. As candidates for the corresponding operation to be performed "commonly" in any scenario, "shutdown" is an operation for shutting down the terminal 10, "restart" is an operation for restarting the terminal 10, and "quit app" is an operation for terminating a running application. Note that the candidate corresponding operations in the "common" usage scenario may be presented by default without being set as a particular determination condition.

[0028] When the dialogue scenario control unit 123 determines that the operation status is "start of work" based on the peripheral information from the peripheral information accumulation unit 122, it displays candidate response operations for that situation, such as an attendance management application, an email application, a project management application, a document creation application, a conference application, a chat application, a publishing site, a document storage application, shutdown, restart, and app termination, on the dialogue interface screen 111. At this time, the dialogue scenario control unit 123 displays the candidate response operations as buttons grouped by usage scenario, as shown in FIG. 2 . In this case, the visibility may be improved by using a different background color for each usage scenario. Furthermore, the distinction between multiple buttons included in one scenario may be improved by changing the density of the background color or slightly changing the RGB values.

[0029] The user selects the desired response operation from among the multiple response operation candidates displayed for each scene. The dialogue scenario control unit 123 then instructs the operation control unit 124 to execute the response operation selected by the user. The user can select the response operation by entering text, pressing a button, inputting a voice command, etc. Note that pressing a button refers to an operation such as tapping on a touch panel that constitutes a touch panel display or clicking with a mouse.

[0030] The dialogue scenario control unit 123 determines response information for the information input to the dialogue interface unit 11 based on both the input information and the peripheral information acquired by the peripheral information acquisition unit 121, and causes the response information to be output to the dialogue interface unit 11. Specifically, the dialogue scenario control unit 123 narrows down candidates for the response operation to be executed next based on the peripheral information, and specifies the response operation to be executed based on the user's input.

[0031] Assuming that the attendance management app has been selected from the suggested corresponding operations, the operation execution phase will be described with reference to FIG. 3 . FIG. 3 is a diagram for explaining the operation execution phase. First, as shown in FIG. 3 , the operation control unit 124 references the operation control information stored in the operation support definition information storage unit 125 and confirms that the information required to execute the attendance management app is the "employee ID" and "password." At this time, because the "employee ID" and "password" have not been entered by the user, the terminal 10 displays a message on the interactive interface screen 111 requesting the entry of the "employee ID" and "password" to complete this information.

[0032] The operation control unit 124 executes the attendance management application 14a using the employee ID “123456789” and password “abcdefghij” entered by the user. The attendance management application 14a is one of the second applications 14.

[0033] In this way, if the peripheral information stored in the peripheral information storage unit 122 lacks predetermined information necessary to execute a corresponding operation, in this example, the second application 14, the dialogue scenario control unit 123 outputs information to the dialogue interface unit 11 to prompt the user to input the missing information. Furthermore, the peripheral information acquisition unit 121 further acquires the information input to the dialogue interface unit 11 as peripheral information. The information acquired as peripheral information will be used when executing the identified corresponding operation, in this example, the second application 14.

[0034] (Operation Example) The flow of processing by the operation support system will be described with reference to Fig. 4 and Fig. 5. First, the processing in the operation identification phase will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the processing flow in the operation identification phase.

[0035] First, the terminal 10 acquires judgment conditions for the operation situation from the operation support definition information storage unit 125 (step S101) and distributes the acquired judgment conditions to the peripheral information acquisition unit 121 (step S102). Next, the terminal 10 acquires peripheral information necessary for judgment from the first application 13a, the OS 13b, and the sensor 13c, and stores the peripheral information in the peripheral information accumulation unit 122 (step S103). Then, the terminal 10 distributes the peripheral information stored in the peripheral information accumulation unit 122 to the dialogue scenario control unit 123 and the operation control unit 124 (step S104).

[0036] The terminal 10 also distributes the peripheral information to the dialogue interface unit 11 (step S105).Then, the terminal 10 displays (or can hide) the acquired peripheral information (step S106).

[0037] Here, the terminal 10 determines the operation status based on the peripheral information (step S107). If the operation status cannot be narrowed down based on the acquired peripheral information and information input to the dialogue interface unit 11 outside this flow (step S108, No), the terminal 10 returns to step S103.

[0038] On the other hand, if the operation situation can be narrowed down based on the acquired peripheral information and information input to the dialogue interface unit 11 outside this flow (step S108, Yes), the terminal 10 presents candidate corresponding operations, i.e., operation suggestions, linked to the narrowed down operation situation (step S109).

[0039] Here, the terminal 10 accepts an input from the user as to which candidate countermeasure operation to execute (step S110). If a candidate countermeasure operation to execute is not selected by the user's input (step S111, No), the terminal 10 returns to step S103. Furthermore, if the user ignores the candidate countermeasure operation presented by the user and performs another operation, the terminal 10 determines that step S111 is No.

[0040] On the other hand, if a candidate for a countermeasure operation to be executed is selected by the user's input (Yes in step S111), the terminal 10 issues an instruction to the operation control unit 124 to execute the coordinated countermeasure operation corresponding to the candidate for the countermeasure operation selected by the user (step S112).Then, the terminal 10 proceeds to the operation execution phase.

[0041] 5 is a flowchart showing an example of the processing flow of the operation execution phase. As shown in FIG. 5, first, the terminal 10 acquires operation control information from the operation support definition information storage unit 125 (step S201).

[0042] If the information distributed from the peripheral information acquisition unit 121 does not provide all the information necessary to execute the corresponding operation (step S202, No), the terminal 10 acquires information necessary to execute the corresponding operation from the operation control unit 124 (step S203).Then, the terminal 10 accepts input of information necessary to execute the corresponding operation by the user via the dialogue interface unit 11 (step S204) until all the information necessary to execute the corresponding operation is input (step S205, No).

[0043] When all the information required to execute the corresponding operation has been input (Yes at step S205), the terminal 10 distributes the information input via the dialogue interface unit 11 to the operation control unit 124 (step S206).

[0044] In step S202, if all the information required to execute the corresponding operation has been acquired from the information distributed by the peripheral information acquisition unit 121 (step S202, Yes), or if step S206 has been executed, the terminal 10 executes the corresponding operation (step S207). Then, the terminal 10 acquires the result of executing the corresponding operation, such as success or failure (step S208). The terminal 10 displays the acquired result of executing the corresponding operation (step S209). After the operation execution phase is completed, the terminal 10 can return to the operation identification phase.

[0045] (Operations and Effects) As described above, in the first embodiment, the peripheral information acquisition unit 121 acquires peripheral information such as information about the first application 13a running on the terminal 10, information about terminal control obtainable from the OS 13b, information obtainable from the sensor 13c provided on the terminal 10, etc. The peripheral information storage unit 122 stores the peripheral information acquired by the peripheral information acquisition unit 121. The dialogue interface unit 11 accepts information input by the user and outputs the information to the user. When the peripheral information stored in the peripheral information storage unit 122 and the information input to the dialogue interface unit 11 satisfy a determination condition, the dialogue scenario control unit 123 compiles information indicating multiple candidate response operations, which are multiple operation proposals pre-associated with the determination condition, for each expected usage scenario and outputs the information to the dialogue interface unit 11. Therefore, according to the first embodiment, the information indicating the operation proposals is collectively output for each expected usage scenario, thereby providing an operation assistance technology that can present a large number of operation proposals to the user to facilitate the user's selection. This allows the user to find the desired operation proposal quickly and reliably, and also prevents operation errors. That is, multiple usage scenarios, i.e., multiple patterns of actions, can be envisioned for a user operation situation on the interactive interface screen 111. If corresponding operation candidates, i.e., operation proposals, are presented only for the scenarios that are expected to be used most frequently, the user will need to manually perform operations for the usage scenarios that are not adopted. If a large number of operation candidates are presented, including operation proposals for not only one usage scenario but also multiple expected usage scenarios, it is difficult for the user to visually recognize them, making it difficult to select the desired candidate, and reducing the user's motivation to check them. In the first embodiment, operation proposals are structured and presented together for each usage scenario, making it easier to find the desired operation proposal.

[0046] Here, the candidate response operations that are operation suggestions include at least one of candidates for the second application 14 that may overlap with the first application 13a and candidates for operations on the terminal 10. Therefore, according to the first embodiment, it is possible to suggest various candidate response operations to the user.

[0047] Moreover, the first embodiment further includes an operation support definition information storage unit 125 that stores information indicating a plurality of operation proposals for each expected usage scenario in association with each of a plurality of determination conditions. When the peripheral information stored in the peripheral information storage unit 122 and the information input to the dialogue interface unit 11 satisfy any of the plurality of determination conditions stored in the operation support definition information storage unit 125, the dialogue scenario control unit 123 outputs information indicating the plurality of operation proposals to the dialogue interface unit 11 in a consolidated manner for each expected usage scenario, based on the information indicating the plurality of operation proposals for each usage scenario stored in the operation support definition information storage unit 125 in association with the determination condition. Therefore, according to the first embodiment, by storing various operation proposals for each determination condition in advance, it is possible to present a large number of operation proposals to the user in a manner that makes it easy for the user to select one for a variety of usage scenarios.

[0048] Furthermore, in the first embodiment, the dialogue scenario control unit 123 causes the dialogue interface unit 11 to output information indicating a plurality of operation suggestions in different display forms that are preset for each usage scenario, such as different colors, etc. Therefore, according to the first embodiment, it becomes easier to distinguish the operation suggestions included in each usage scenario, allowing the user to easily find the desired operation suggestion and preventing operational errors.

[0049] [Modifications] (Configuration Example) The display format of operation suggestions, which are candidates for multiple response operations, is not limited to buttons as shown in FIG. 2. FIGS. 6 to 8 are diagrams showing modified examples of the display format of the response operation candidates summarized for each scene on the dialogue interface screen 111. FIG. 6 shows an example of a display format using radio buttons. Here, "Register working hours and request holidays" corresponds to the attendance management app, "Check / send email" corresponds to the email app, and "Check / update status of assigned work projects" corresponds to the project management app. FIG. 7 shows an example of a display format using combo boxes to summarize response operation candidates for each scene. FIG. 8 also shows an example of a display format using icons or the like to summarize response operation candidates for each scene. Here, icons for the attendance management app, email app, and project management app are shown from left to right.

[0050] In the button summary display format shown in Fig. 2, the radio button summary display format shown in Fig. 6, and the combo box summary display format shown in Fig. 7, the content of each candidate response operation is represented by a character string. When the number of characters in the character string is small and the number of proposed response operation candidates is small, the button summary display format shown in Fig. 2 is suitable. On the other hand, when the number of proposals is small but the number of characters is large, the radio button summary display format shown in Fig. 6 is suitable. Conversely, when the number of characters is small but the number of proposals is large, the combo box summary display format shown in Fig. 7 is suitable. Furthermore, when the character string is difficult to understand and the number of proposals is small, a summary display format in which the characters are replaced with icons, etc., as shown in Fig. 8 may be used.

[0051] The dialogue scenario control unit 123 displays candidate response operations in a predetermined summary display format out of these four types of summary display formats.

[0052] Alternatively, the dialogue scenario control unit 123 may selectively adopt one of these four types of summary display format based on the number of characters in the notation representing the content of the candidate response operations defined in the judgment conditions and the number of candidate response operations. Note that, since a large number of characters and a large number of suggestions are generally difficult to read, the dialogue scenario control unit 123 may consider converting the number of characters to a smaller number and then adopting a combo box summary display format.

[0053] (Functions and Effects) As described above, in the modified example, the dialogue scenario control unit 123 outputs information indicating multiple candidate response operations, i.e., multiple operation suggestions, in a predefined summary display format for each usage scenario. Therefore, according to the modified example, one of the multiple summary display formats can be provided to the user as a default.

[0054] Furthermore, in a modified example, the dialogue scenario control unit 123 determines a summary display format for outputting information indicating multiple operation suggestions for each usage scenario based on information indicating the multiple operation suggestions to be output, such as the number of characters and the number of candidates. Therefore, according to this modified example, it is possible to suggest operations to the user in an appropriate summary display format from a variety of summary display formats depending on the information indicating the multiple operation suggestions to be output. In particular, when the operation content labels of the multiple operation suggestions are similar, a summary display format that clearly distinguishes the differences can be adopted, which also helps prevent erroneous selections.

[0055] Second Embodiment (Configuration Example) The operation assistance system may be configured to be able to change the display format of the candidate countermeasure operations, i.e., the candidate countermeasure operations. In addition to the functions described in the first embodiment, the operation assistance system of the second embodiment has a function to change the display format of the candidate countermeasure operations.

[0056] Fig. 9 is a diagram showing an example of the configuration of an operation assistance system according to the second embodiment. As shown in Fig. 9, the terminal 10 that realizes the operation assistance system according to this embodiment includes a display form definition information holding unit 126 in addition to the configuration of the terminal 10 in the first embodiment. The display form definition information holding unit 126 stores display form information that is information about the display forms of candidates for corresponding operations.

[0057] Fig. 10 is a diagram showing an example of the contents stored in the display form definition information storage unit 126. As shown in Fig. 10, the display form definition information storage unit 126 can store the display form information in the form of, for example, a form / size setting table 1261 and a color setting table 1262. The developer of the dialogue system can set default values ​​as the settings in these tables 1261 and 1262.

[0058] The form / size setting table 1261 sets a form number that specifies the display form of the candidate corresponding operations for each scene, and a font size. For example, the form number "1" indicates a button, "2" a radio button, "3" a combo box, and "4" a replacement display form. The font size is a numerical value in points.

[0059] The color setting table 1262 sets scene background colors and color information for each operation button for each scene when the display format for the corresponding operation candidates by scene is buttons. The color information is, for example, RGB values. Of course, color information using notations other than RGB values ​​may also be used, and other information such as density may also be set. In the example of FIG. 10 , the background colors are set as follows: light blue with RGB values ​​of "0, 255, 255" for scene No. 1; rose with RGB values ​​of "255, 255, 204" for scene No. 2; lime with RGB values ​​of "153, 204, 0" for scene No. 3; and so on. Furthermore, a light gray color of "192, 192, 192" is set for each button in each scene.

[0060] When displaying multiple candidate response operations for each usage scenario, the dialogue scenario control unit 123 of the chatbot 12 determines the display mode based on the display mode information stored in the display mode definition information storage unit 126. Furthermore, based on information on changes to the display mode made by the user input via the dialogue interface unit 11, the dialogue scenario control unit 123 updates the display mode information stored in the display mode definition information storage unit 126 so that the display mode is the one changed by the user.

[0061] FIG. 11 illustrates an example of a display transition associated with a change operation of the summary display mode. The upper part of FIG. 11 shows a display example of the interactive interface screen 111 based on the display mode information stored in the display mode definition information storage unit 126. In this display example, the summary display mode for each scene of candidate response operations is represented by a button. Here, the user selects, for example, the "email application" button as the button to be changed. The user can perform this selection operation by right-clicking or double-clicking the target button with the mouse, by voice input, or the like. In FIG. 11, this selection is indicated by a star. In response to this selection operation, the interactive scenario control unit 123 causes the interactive interface unit 11 to display a setting screen 112, as shown in the lower part of FIG. 11. This setting screen 112 may be switched from the interactive interface screen 111, may be superimposed on the interactive interface screen 111, or may be displayed alongside the interactive interface screen 111. Note that, while the example shown here shows the case where the button for the corresponding operation candidate, "Email App," is selected, the dialogue scenario control unit 123 also displays the setting screen 112 when an area separated by usage scenario, such as the "Normal Times" area, is selected.

[0062] The setting screen 112 includes selection item areas 1121, 1122, and 1123 for "Form," "Color," and "Size," a preview area 1124, an "Apply" button 1125, and a "Cancel" button 1126. The "Form" selection item area 1121 includes user interfaces such as checkboxes for displaying summary forms, such as the buttons shown in FIG. 2 , the radio buttons shown in FIG. 6 , the pull-down menu shown in FIG. 7 , and the icons (replace) shown in FIG. 8 . For example, checking a checkbox allows the selection of the summary form. The "Color" selection item area 1122 includes an image visually displaying various colors, such as a color palette. A corresponding color can be selected by selecting any position on the image. The "Color" selection item area 1122 also includes a user interface such as a checkbox for "Apply gradation to all operation proposals for the same scene." When multiple usage scenarios are associated with a single operation situation and multiple corresponding operation proposals exist for the same usage scenario, checking this checkbox allows for simultaneous changes to all of the proposed operations. The "Size" selection item area 1123 is provided with a slide bar or a user interface for directly and quantitatively inputting the font size, allowing the font size of the proposed content of the candidate corresponding operation to be changed. The above selection results are reflected in the preview area 1124. If the user's intention is met, pressing the "Apply" button 1125 will cause the results to be reflected on the actual interactive interface screen 111. Pressing the "Cancel" button 1126 will return the selection results made on the setting screen 112 to the original interactive interface screen 111 without being reflected on the actual interactive interface screen 111.

[0063] 12 shows an example of the display on the dialogue interface screen 111 before and after the summary display mode is changed, and FIG. 13 shows an example of the contents stored in the display mode definition information storage unit 126 before and after the summary display mode is changed. These figures show an example in which the "Email Application" button is selected on the dialogue interface screen 111 as shown in FIG. 11 and an operation is performed to change the color, for example, from light gray (192, 192, 192) to rose (255, 153, 204). In response to input of operation information indicating that the "Apply" button 1125 has been pressed on the setting screen 112, the dialogue scenario control unit 123 changes the display color of the "Email Application" button as shown in FIG. 12 and updates the RGB values ​​of operation 2 of scene No. 1, which is the corresponding display mode information, in the color setting table 1262 of the display mode definition information storage unit 126 as shown in FIG.

[0064] (Operation Example) The following describes the flow of processing in the operation identification phase by the operation support system according to the second embodiment. Note that the processing in the operation execution phase is the same as in the first embodiment, so a description thereof will be omitted.

[0065] 14 is a flowchart showing an example of the processing flow of the operation identification phase. In this embodiment, if a candidate response operation to be executed is not selected by the user's input (step S111, No), the terminal 10 further determines whether an instruction to change the summary display mode has been given by the user's input. If no instruction to change the summary display mode has been given (step S121, No), the terminal 10 returns to step S103.

[0066] On the other hand, if an instruction to change the combined display mode has been given (Yes at step S121), the terminal 10 executes the combined display mode change process, and then returns to step S103.

[0067] 15 is a flowchart showing an example of the process flow for changing the summary display format in step S122. The terminal 10 first displays the setting screen 112 (step S1221). Then, the terminal 10 accepts input to the setting screen 112 by the user (step S1222).

[0068] If neither the "Cancel" button 1126 nor the "Apply" button 1125 is selected, nor is there any operation to change the "Form", "Color" or "Size" (steps S1223, S1224, S1225, No), the terminal 10 returns to step S1222.

[0069] If a change operation has been performed in any of the selection item areas 1121, 1122, and 1113 for "Form," "Color," or "Size" (No at step S1225), the terminal 10 temporarily saves the change settings for the summary display form instructed by the change operation (step S1226). Then, the terminal 10 displays a preview of the display form based on the temporarily saved change settings in the preview area 1124 of the setting screen 112 (step S1227). Thereafter, the terminal 10 returns to step S1222.

[0070] If the "Apply" button 1125 is selected (Yes in step S1224), the terminal 10 applies the temporarily saved changed settings (step S1228). Specifically, the terminal 10 changes the display mode of the summary of the corresponding operation candidates for each scene on the interactive interface screen 111 to a display mode based on the temporarily saved changed settings in the "Form," "Color," or "Size" selection item areas 1121, 1122, and 1113. The terminal 10 also updates the settings in the form / size setting table 1261 and the color setting table 1262 of the display form definition information storage unit 126 based on the temporarily saved changed settings in the "Form," "Color," or "Size" selection item areas 1121, 1122, and 1113. The terminal 10 then terminates the display of the setting screen 112 (step S1229) and returns to step S109.

[0071] If the "Cancel" button 1126 is selected (step S1223, Yes), the terminal 10 deletes the temporarily saved changed settings (step S1230), and then ends the display of the setting screen 112 (step S1229), returning to step S109.

[0072] The processing of the summary display form for each scene of candidate corresponding operations on the setting screen 112 can also be executed when a developer sets the default values ​​of the form / size setting table 1261 and the color setting table 1262. FIG. 16 is a diagram showing an example of a display transition related to a change operation of the summary display form during development. On the development screen 113 for setting the usage scene, the developer selects any usage scene or candidate corresponding operation, thereby enabling setting on the setting screen 114. The setting screen 114 for the developer can be the same as the setting screen 112 for the user. By selecting the "Apply" button, the developer can change the default values ​​of the form / size setting table 1261 and the color setting table 1262.

[0073] According to the second embodiment, when a user performs an operation to change the summary display format or the display format of information indicating candidate countermeasures, i.e., operation suggestions, the dialogue scenario control unit 123 changes the display format based on the user's operation. Therefore, according to the second embodiment, the user can change the summary display format or the display format of information indicating operation suggestions so that it is easy for the user to distinguish.

[0074] [Other Embodiments] In the second embodiment, the contents stored in the display form definition information storage unit 126 are rewritten in response to a user's operation to change the display form, but it is also possible to store default settings and restore the contents stored in the display form definition information storage unit 126 to the default settings when the user finishes using the terminal 10. Alternatively, if the terminal 10 can identify the user by inputting a user ID or the like when starting to use the terminal 10, the display form definition information storage unit 126 may be configured to store settings for each user.

[0075] Furthermore, if the terminal 10 is used only by specific users, or if the user can be identified by the terminal 10, the operation history of the users may be stored, and the positions and order of the candidate corresponding operations, i.e., operation suggestions, may be changed or highlighted in order of frequency of operation to match the user's recent operating tendencies. This is particularly likely to reduce the number of scroll operations in the combo box display format shown in FIG. 7.

[0076] Furthermore, some functions of the operation assistance system may be realized by a server, etc. For example, in the operation assistance system, the dialogue interface unit 11, the first application 13a, and the second application 14 may be provided in the terminal 10, and the chatbot 12 may be provided in the server.

[0077] The dialogue interface unit 11 may also be a known external tool such as Slack (registered trademark). In this case, the dialogue scenario control unit 123 cooperates with the external tool by exchanging information using an API (Application Programming Interface) or the like.

[0078] The dialogue scenario control unit 123 may also make a determination based on peripheral information that combines two or more pieces of information from among information about the first application 13a, information about terminal control, and information obtainable from the sensor 13c of the terminal 10. For example, the peripheral information acquisition unit 121 obtains information indicating whether the second application 14 is running from the OS 13b, and obtains the position on the screen of the terminal 10 at which the user is looking from the sensor 13c. Then, the dialogue scenario control unit 123 determines that the predetermined condition is satisfied if the second application 14 is not running and the position at which the user is looking is included in the area where the icon of the second application 14 is located.

[0079] [System Configuration, etc.] The components of each device shown in the figures are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a hardware processor such as a CPU and a program analyzed and executed by the hardware processor, or can be realized as hardware using wired logic.

[0080] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0081] [Program] In one embodiment, the terminal 10 can be implemented by installing an operation assistance program that executes the above-described operation assistance process as package software or online software on a desired computer. For example, by executing the above-described operation assistance program on an information processing device, the information processing device can function as the terminal 10. The information processing device referred to here includes desktop and notebook personal computers. Other information processing devices also include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).

[0082] 17 is a diagram showing an example of a computer that executes an operation assistance program. The computer 1000 includes, for example, a memory 1010 and a processor 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0083] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0084] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the terminal 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing the same process as the functional configuration of the terminal 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced with an SSD (solid state drive).

[0085] Furthermore, setting data such as determination conditions and display form information used in the processing of the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. The processor 1020 then reads out the program module 1093 and program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0086] The program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the processor 1020 via the disk drive 1100 or the like. The storage medium is a non-transitory, tangible medium. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the processor 1020 from the other computer via the network interface 1070.

[0087] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0088] In short, this invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0089] DESCRIPTION OF SYMBOLS 10...Terminal 11...Dialogue interface unit 12...Chatbot 13a...First application 13b...OS 13c...Sensor 14...Second application 14a...Attendance management application 111...Dialogue interface screen 112...Setting screen 1121, 1122, 1113...Selection item area 1122...Selection item area 1124...Preview area 1125..."Apply" button 1126..."Cancel" button 113...Development screen 114...Setting screen 121...Peripheral information acquisition unit 122...Peripheral information accumulation unit 123...Dialogue scenario control unit 124...Operation control unit 125...Operation support definition information storage unit 126...Display form definition information storage unit 1261...Form / size setting table 1262...Color setting table 1000...Computer 1010...Memory 1020...Processor 1110...Mouse 1120...Keyboard 1130...Display

Claims

1. An operation assistance system comprising: a peripheral information acquisition unit that acquires, as peripheral information, one or more of information related to applications running on a terminal, information related to terminal control, and information obtainable from sensors equipped on the terminal; a peripheral information storage unit that stores the peripheral information acquired by the peripheral information acquisition unit; an interactive interface unit that accepts information input by a user and outputs information to the user; and an interactive scenario control unit that, when the peripheral information stored in the peripheral information storage unit and the information input to the interactive interface unit satisfy a judgment condition, compiles information indicating a plurality of operation suggestions that are pre-associated with the judgment condition for each expected usage scenario and outputs the information to the interactive interface unit.

2. The operation assistance system of claim 1, further comprising an operation definition information storage unit that stores information indicating the plurality of operation proposals for each of the expected usage scenarios in association with each of a plurality of judgment conditions, and wherein when the peripheral information stored in the peripheral information storage unit and the information input to the dialogue interface unit satisfy any of the plurality of judgment conditions stored in the operation definition information storage unit, the dialogue scenario control unit outputs the information indicating the plurality of operation proposals to the dialogue interface unit in a consolidated manner for each of the expected usage scenarios, based on the information indicating the plurality of operation proposals for each of the usage scenarios stored in the operation definition information storage unit in association with that judgment condition.

3. The operation assistance system according to claim 2, wherein the dialogue scenario control unit determines the display format to be output collectively for each usage scenario based on information indicating the plurality of operation suggestions.

4. The operation assistance system according to claim 2, wherein the dialogue scenario control unit outputs information indicating the plurality of operation suggestions to the dialogue interface unit in different display forms preset for each of the usage scenes.

5. An operation assistance system according to claim 3 or 4, wherein, when a user performs an operation to change the display mode, the dialogue scenario control unit changes the display mode based on the user's operation.

6. The operation support system according to any one of claims 1 to 4, wherein the operation suggestions include at least one of application candidates and operation candidates for the terminal.

7. An operation assistance method executed by a computer having a processor and a memory, comprising: an acquisition step of acquiring, as peripheral information, one or more of information related to applications running on the terminal, information related to terminal control, and information obtainable from a sensor equipped in the terminal; a storage step of storing the peripheral information acquired by the acquisition step; and a control step of, when the peripheral information stored by the storage step and information input to a dialogue interface unit satisfy a judgment condition, compiling information indicating a plurality of operation suggestions pre-associated with the judgment condition for each expected usage scenario and outputting the information to the dialogue interface unit.

8. An operation assistance program that causes a computer processor to execute the following steps: an acquisition step of acquiring, as peripheral information, one or more of information related to applications running on the terminal, information related to terminal control, and information obtainable from sensors equipped on the terminal; a storage step of storing the peripheral information acquired by said acquisition step; and a control step of, when the peripheral information stored by said storage step and the information input to the dialogue interface unit satisfy a judgment condition, compiling information indicating a plurality of operation suggestions that are pre-associated with the judgment condition for each expected usage scenario and outputting the information to the dialogue interface unit.

Images