Operation support device and scenario creation management method

Abstract

In one aspect of the present invention, a plurality of commands are managed using an object model in which a plurality of objects respectively constituting the command are represented by a tree structure. On the basis of the object model, label information is assigned to a target object to be operated and monitored among the plurality of objects for each command, and the assigned label information is managed using label management information represented by a tree structure. Furthermore, usage record information of each target object is generated, and the usage record information is managed in association with the label information corresponding to the target object. Then when one of the objects is designated as a search condition, the usage record information of the target object located under the designated object in the tree structure for each command is obtained on the basis of the label management information, a priority is calculated on the basis of the usage record information, and a candidate command, which is a search candidate, is selected and output on the basis of the calculated priority.

Description

Operation support device and scenario creation management method

[0001] One aspect of the present invention relates to an operation assistance device that uses, for example, a chatbot, and has a function for creating and managing an operation scenario of the chatbot, and a method for creating and managing the scenario.

[0002] A dialogue system called a "chatbot" is known, which automatically responds to text input from a user. A chatbot automatically responds to user input according to a pre-set scenario, which makes it possible to reduce the amount of work previously done by humans in tasks such as customer support, thereby contributing to improved work efficiency.

[0003] In addition to simple text responses, operation support systems have also been proposed that enable operations on other applications based on input to chatbots. For example, one such technology provides data input to a chatbot to a robotic process automation (RPA) tool to execute RPA, thereby automating user operations.

[0004] An RPA chatbot sets up processes called commands corresponding to the type of business in a scenario, and executes processes to support the corresponding business in response to the user's selection of the command (see, for example, Patent Document 1).

[0005] International Publication No. 2021 / 070292

[0006] However, in operation support systems that enable the operation of other applications, such as RPA chatbots, the number of commands tends to increase as the scope of business support expands. For example, when a user tries to search for commands corresponding to a desired target business in order to create a scenario, searching for the commands takes time and effort, which reduces the user's work efficiency.

[0007] The present invention has been made in light of the above circumstances, and aims to provide a technique for facilitating command searches and improving work efficiency.

[0008] In order to solve the above problem, one aspect of the present invention is to create and manage a scenario for executing multiple applications that are targets of operational assistance using commands corresponding to the applications, by managing the multiple commands using an object model in which multiple objects constituting each command are represented in a tree structure, and by assigning, for each command, labels to identify the target objects that are to be operated and monitored among the multiple objects based on the object model, and managing the assigned labels using label management information represented in a tree structure. Furthermore, usage history information for the target objects is generated, and the generated usage history information is managed in association with the labels corresponding to the target objects. When one of the objects is specified as a search condition, usage history information for the target objects located below the specified object in the tree structure is obtained based on the label management information, a priority is calculated based on the obtained usage history information, and candidate commands to be search candidates are selected and output from the multiple commands based on the calculated priority.

[0009] According to one aspect of the present invention, multiple commands are each represented by an object model that represents multiple objects in a tree structure, and by using this object model, it is possible to automatically assign labels to the target objects that are to be operated on and monitored within the command.

[0010] Furthermore, for each command, usage history information of the target object is managed in association with a label, and when a certain object is specified as a search condition, a priority is calculated based on the usage history information of the target object located below it in the tree structure, and commands that become search candidates are selected according to the calculated priority. Therefore, it is possible to select from multiple commands commands that are highly relevant to the object specified as the search condition and present them to the user as search candidates.

[0011] That is, according to one aspect of the present invention, it is possible to provide a technique that facilitates searching for commands and improves work efficiency.

[0012] FIG. 1 is a block diagram showing an example of the configuration of an operation assistance device according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the functional configuration of a scenario control unit that controls a chatbot in the operation assistance device shown in FIG. 1, together with the configuration of its peripheral units. FIG. 3 is a flowchart showing an example of the procedure and content of a label assignment process executed by the scenario control unit shown in FIG. 2. FIG. 4 is a flowchart showing an example of the procedure and content of a command search process executed by the scenario control unit shown in FIG. 2. FIG. 5 is a diagram showing an example of the configuration of a scenario file. FIG. 6 is a diagram showing an example of objects displayed on an output device. FIG. 7 is a diagram showing an example of an object model represented by a tree structure. FIG. 8 is a diagram showing an example of label assignment to target objects that are to be operated and monitored in the object model shown in FIG. 7. FIG. 9 is a diagram showing an example of usage history linked to the labels shown in FIG. 8. FIG. 10 is a diagram for explaining a method for calculating label priority and an example of a search result for candidate commands based on the label priority.

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] [One Embodiment] (Configuration Example) FIG. 1 is a block diagram showing an example of the configuration of an operation assistance device SM according to one embodiment of the present invention.

[0015] The operation assistance device SM according to one embodiment has a function of using an RPA chatbot to assist in dialogue responses and to perform proxy operations for applications in various business systems, and is configured as an information processing terminal such as a personal computer. The operation assistance device SM may also be configured as part of the functions of a server computer located on the web or in the cloud.

[0016] The operation assistance device SM includes a control unit 1 that uses a hardware processor such as a central processing unit (CPU), etc. A storage unit having a program storage unit 2 and a data storage unit 3, and an input / output interface (hereinafter, interface will be referred to as I / F) unit 4 are connected to the control unit 1 via a bus.

[0017] An input device IN using a keyboard, a mouse, or touch operation on an input sheet arranged on a display screen, and an output device OU having, for example, a display device, are connected to the input / output I / F unit 4. The input / output I / F unit 4 receives operation information input by a user on the input device IN and transmits it to the control unit 1, and also outputs display information generated by the control unit 1 to the output device OU.

[0018] The input device IN may include a voice input device such as a microphone, and the output device OU may include a printer, an external storage medium, or the like.

[0019] The program storage unit 2 is configured by combining, for example, a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD) as a storage medium that can be written to and read at any time, and a non-volatile memory such as a read only memory (ROM), and stores application programs necessary for executing various control processes according to one embodiment of the present invention, in addition to middleware such as an operating system (OS).

[0020] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an HDD or SSD as a storage medium that can be written to and read at any time, and a volatile memory such as a RAM (Random Access Memory), and the storage area is provided with storage units such as a database and buffer that store information necessary for the control unit 1 to execute various controls.

[0021] The control unit 1 includes an RPA chatbot 10 and business application execution units 12a to 12z as processing units according to an embodiment of the present invention. The chatbot 10 includes a scenario control unit 11.

[0022] The scenario control unit 11 has a function of performing dialogue response processing for the user and also of performing proxy operations for the business application execution units 12a to 12z.

[0023] The scenario control unit 11 controls the above-mentioned support functions based on a scenario file. The scenario file is defined by a combination of command information and a data set. The command information is set corresponding to, for example, each of multiple business applications that are the target of proxy operations. In other words, if there are multiple business applications, the scenario file defines multiple commands A to Z, as shown in FIG. 5, for example. Each command is logically described using processing procedures such as conditional branching and repetition related to proxy operations.

[0024] A data set is defined by management information that combines screen elements, monitoring conditions, and events. Screen elements are used for proxy operations and to acquire peripheral information within the operation support device SM, and an example of this is image information representing an "OK button." Monitoring conditions are conditions for detecting changes in peripheral information within the operation support device SM, which are one element of a trigger for a command to run, and an example of this is information such as "Was the "OK button" pressed?". An event is information based on a complex combination of monitoring conditions for a command to run, and an example of this is information such as "The OK button was pressed while the XX screen was displayed."

[0025] The management information of the above data set may be used as a control scenario for the RPA chatbot 10 as is, or may be converted (compiled) into a data format suitable for the RPA chatbot 10 and then used.

[0026] Furthermore, the scenario control unit 11 has a function of supporting the above-mentioned user operations, as well as a function of managing scenario creation according to an embodiment of the present invention.

[0027] FIG. 2 is a block diagram showing the configuration of the functions required to realize the scenario creation management function provided in the scenario control unit 11, together with the configuration of the peripheral parts.

[0028] The scenario control unit 11 includes a command information management processing unit 111 , a label assignment processing unit 112 , an object usage history information management processing unit 113 , a candidate command selection processing unit 114 , and a candidate command output processing unit 115 .

[0029] As described above, the command information management processing unit 111 manages the command information set for each of the multiple business applications that are the target of proxy operations, using a tree-structured object model. The command information storage unit 31 provided in the data storage unit 3 stores the command information. An example of the configuration of the command information will be described in the operation example.

[0030] The label assignment processing unit 112 assigns label information (hereinafter also referred to as UI labels) for identifying objects to be operated and monitored among the objects of each element constituting the command information, for each piece of command information, based on the object model stored in the command information storage unit 31. Then, the label assignment processing unit 112 stores the label information assigned to each object in the label management information storage unit 32 provided in the data storage unit 3 as label management information having a tree structure similar to the object model.

[0031] The object usage history information management processing unit 113 counts the number of operations each time an operation is performed on an object of each element in the command information, and stores the counted value of the number of operations as information representing the usage history of the object (in this example, the number of uses) in association with the corresponding label in the label management information storage unit 32.

[0032] When a screen element is specified as a search condition, the candidate command selection processing unit 114 obtains, for each command, the number of uses of target objects to be operated or monitored that are located below the tree structure of the object representing the specified screen element, based on the label management information stored in the label management information storage unit 32, and calculates a priority based on the obtained number of uses.The candidate command selection processing unit 114 then selects a candidate command to be a search candidate from among the multiple commands based on the calculated priority.An example of this command selection process will also be described in an operation example.

[0033] The candidate command output processing unit 115 generates display information for the candidate commands selected by the candidate command selection processing unit 114, and outputs the generated display information from the input / output I / F unit 4 to the output device OU.

[0034] The processing units 111 to 115 of the scenario control unit 11 described above, together with the other processing functions of the control unit 1 of the operation assistance device SM, are realized by causing the hardware processor of the control unit 1 to execute an application program stored in the program storage unit 2. Some or all of the processing units of the control unit 1 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0035] (Example of Operation) Next, an example of operation of the operation assistance device SM configured as above will be described.

[0036] (1) Management of Command Information Generally, in an accessibility API, elements of a user interface (UI) displayed on a display screen of an information processing terminal are often managed as an object model using a tree structure.

[0037] 6 shows an example of display information including screen elements displayed on the display screen of an information processing terminal. In this example, display information relating to applications 1 and 2 is displayed on display screen V, and this display information includes title bars TL1 and TL2, content items CT1 and CT2, text boxes TB1 and TB2 included in content items CT1 and CT, and registration buttons B1 and B2.

[0038] Figure 7 shows an example of the configuration of an object model corresponding to the display information shown in Figure 6. As shown in this example, each screen element of the display information is represented by an object model with a tree structure. In this display information, if a value is entered into text box TB1, for example, the chatbot 10 first searches for the corresponding object based on the tree structure of the object model, and then detects that a value has been entered by referencing the property of the input value held by the searched object.

[0039] The scenario control unit 11 of the operation support device SM focuses on the object model of the user interface (UI) as described above, and under the control of the command information management processing unit 111, configures the object model of the command as follows.

[0040] That is, the command information management processing unit 111 defines each screen element corresponding to a user interface (UI) constituting a command as an object, and then constructs an object model for each command, representing the screen elements in a tree structure, and stores the constructed object model in the command information storage unit 31 in association with the command identification information.

[0041] (2) Management of Label Management Information The scenario control unit 11 of the operation support device SM, under the control of the label assignment processing unit 112, executes the process of assigning label information to each of the above commands as follows.

[0042] FIG. 3 is a flowchart showing an example of the processing procedure and processing content of the labeling process executed by the labeling processing unit 112.

[0043] That is, the labeling processing unit 112 first selects one command in step S10 based on the command information stored in the command information storage unit 31. Then, in step S11, the labeling processing unit 112 extracts target objects to be operated and monitored from among the objects corresponding to each element constituting the command based on the object model of the selected command. Next, in step S12, the labeling processing unit 112 assigns label information to the extracted target objects.

[0044] 8 shows an example of the result of assigning label information to a command. In this example, labels "L0" to "L23" are assigned to "Terminal Screen," "Application 1," "Application 2," "Title Bar 1," "Title Bar 2," "Content 1," "Content 2," "Text Box 1," "Text Box 2," "Button 1," "Button 2," and "Button 3," which are the target objects to be operated and monitored, out of the "Terminal Screen," "Application 1," "Application 2," "Content 1," "Content 2," "Button 1," "Button 2," and "Button 3" in the tree-structured object model.

[0045] In step S13, the label assignment processing unit 112 generates label management information that represents the labels “L0” to “L23” in a tree structure similar to the object model, and stores the generated label management information in the label management information storage unit 32.

[0046] When the label information assignment process for one command is completed, the label assignment processing unit 112 determines in step S14 whether or not all commands have been selected. If the result of this determination is that unselected commands remain, the label assignment processing unit 112 returns to step S10 to select the next command and executes the label assignment process of steps S11 to S13 for the selected command. Thereafter, the label information assignment process is repeated in the same manner for all commands.

[0047] (3) Management of object usage history The scenario control unit 11 of the operation support device SM manages the object usage history as follows, under the control of the object usage history information management processing unit 113, each time one of the screen elements is operated in the display information displayed on the display screen of the output device OU.

[0048] That is, the object usage history information management processing unit 113 counts up the number of times the screen element has been operated, and then updates and stores the counted-up number of times of operation as a usage count indicating the usage history of the screen element in the label management information storage unit 32 in association with the label information of the target object corresponding to the operated screen element.

[0049] At the same time, the object usage history information management processing unit 113 reflects the count-up number of uses of the screen element in the count-up number of uses of each higher-level target object that is connected to the target object corresponding to the operated screen element in the tree structure of the object model.Then, the reflected count of uses is updated and stored in the label management information storage unit 32.As a result, in the label management information, the total value of the count of uses of all target objects subordinate to the target object, including the target object, is managed as usage history information as the count of use of the target object located higher in the tree structure of the object model.

[0050] 9 shows an example of label management information managed in the label management information storage unit 32. This example shows a case where the numbers of times of use "8", "6", "2", "6", "1", "1", "5", and "1" are stored in association with the labels "L0" to "L23" assigned to the target objects to be operated and monitored: "PC terminal screen", "application 1", "application 2", "content 1", "content 2", "button 1", "button 2", and "button 3", respectively.

[0051] (4) Selection of Candidate Commands The scenario control unit 11 of the operation support device SM executes the process of selecting candidate commands to be presented as search results as follows.

[0052] FIG. 4 is a flowchart showing an example of the procedure and content of a candidate command selection process and a selection result output process executed by the scenario control unit 11 of the operation support device SM.

[0053] That is, the scenario control unit 11 monitors the operation of specifying a search condition in step S20. When a screen element in a command being displayed is specified as a search condition in this state, first in step S21, the object usage history information management processing unit 113 identifies an object corresponding to the specified screen element by referring to the command information stored in the command information storage unit 31. Next, in step S22, the object usage history information management processing unit 113 counts up the number of uses associated with the label information of the identified object, and in step S23, updates and stores the counted number of uses in the label management information storage unit 32.

[0054] Next, in step S24, under the control of the candidate command selection processing unit 114, the scenario control unit 11 reads out, for each command, from the label management information stored in the label management information storage unit 32, the number of uses associated with the label information of the object corresponding to the screen element specified as the search condition.

[0055] At this time, the number of times the object is used that is read out is a value that reflects the number of times each target object that is located under the object in the tree structure of the object model is used, that is, the number of times the specified object itself is used plus the number of times all target objects that are located under it in the tree structure are used.

[0056] Then, in step S24, the candidate command selection processing unit 114 calculates the priority of each command based on the read number of times the specified object is used and the number of times the object located at the top of the object model is used, i.e., the total number of times all target objects constituting the command are used. More specifically, the ratio of the number of times the specified object is used to the number of times the object located at the top of the object model is calculated, and the calculated value is set as the priority of the command.

[0057] In step S25, the candidate command selection processor 114 compares the priority calculated for each command with a preset threshold value and selects commands whose priority is equal to or greater than the threshold value. In step S26, the candidate command selection processor 114 then sorts the selected commands in descending order of priority.

[0058] 10 shows an example of the priority calculation process and command selection process described above, as well as an example of a search result for candidate commands selected based on the calculated priorities. In this example, "Content 1" shown in FIG. 8 is specified as a search criterion, and the priority of the command is determined by dividing the number of times "Content 1" has been used (6) by the number of times the top-level object, "Terminal Screen," has been used (8). Then, based on the calculated priority for each command, multiple candidate commands above a threshold are selected, and the selected candidate commands are sorted in descending order of priority, resulting in multiple candidate commands X, A, Y, ... being obtained as a search result.

[0059] Finally, in step S27, under the control of the candidate command output processing unit 15, the scenario control unit 11 of the operation assistance device generates display information for displaying the sorted candidate commands X, A, Y, ..., and outputs the generated display information from the input / output I / F unit 4 to the output device OU for display.

[0060] (Effects) As described above, the operation assistance device SM according to one embodiment manages each screen element constituting a command using a tree-structured object model, and uses this object model to assign label information to objects corresponding to the operation target and monitoring target screen elements included in the command. The operation assistance device SM then manages the number of uses of the objects corresponding to the operation target and monitoring target elements in association with the label information. When an arbitrary screen element is specified as a search criterion, the device calculates a priority for each command based on the total number of uses of the specified screen element object itself and each of its subordinate target objects, as well as the total number of uses of the target objects corresponding to all screen elements included in the command. Then, candidate commands with a priority higher than a threshold are selected, and the selected candidate commands are sorted in descending order of priority, thereby generating and displaying display information for the candidate commands.

[0061] Therefore, by representing the screen elements that make up a command as an object model with a tree structure, and using this object model, it becomes possible to automatically assign label information to the target objects that are to be operated and monitored within the command, thereby eliminating the workload on the user involved in assigning label information.

[0062] Furthermore, for each command, the number of times a target object is used is managed in association with label information, and when a certain object is specified as a search condition, a priority is calculated based on the number of times the target object is used, including those located below it in the tree structure of the object model, and candidate commands are selected according to the calculated priority. Therefore, it is possible to select and present candidate commands from multiple commands, taking into consideration not only the number of times the target object specified as a search condition is used, but also the number of times its related target objects are used below it.

[0063] In the embodiment, the number of uses is used as the object usage history information, but the usage frequency may be used instead of the number of uses. Also, various methods can be used to calculate the priority.

[0064] In addition, the hardware configuration of the operation assistance device SM, the processing functions of the control unit 1, particularly the scenario control unit 11, and their processing contents can be modified in various ways without departing from the spirit of this invention.

[0065] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0066] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0067] SM...operation support device IN...input device OU...output device 1...control unit 2...program storage unit 3...data storage unit 4...input / output I / F unit 10...chatbot 11...scenario control unit 12a to 12z...business application execution unit 111...command information management processing unit 112...label assignment processing unit 113...object usage history information management processing unit 114...candidate command selection processing unit 115...candidate command output processing unit 31...command information storage unit 32...label management information storage unit

Claims

1. An operation assistance device having a function for creating and managing scenarios for executing multiple applications that are targets of operation assistance using commands corresponding to the applications, comprising: a first processing unit that manages the multiple commands using an object model in which multiple objects that make up each command are represented in a tree structure; a second processing unit that, based on the object model, assigns labels to identify the target objects to be operated and monitored among the multiple objects for each command, and manages the assigned labels using label management information represented in a tree structure; a third processing unit that generates usage history information for the target objects, and manages the generated usage history information in association with the labels corresponding to the target objects; a fourth processing unit that, when one of the objects is specified as a search condition, acquires, based on the label management information, the usage history information of the target objects located below the specified object in the tree structure for each command, and calculates a priority based on the acquired usage history information; and a fifth processing unit that selects and outputs candidate commands to be search candidates from the multiple commands based on the priority.

2. The operation assistance device of claim 1, wherein the fourth processing unit calculates, based on the label management information, the priority for each command by dividing the sum of the usage history information of the specified object and the target objects located below it in the tree structure by the sum of the usage history information of all the target objects that make up the command.

3. The operation assistance device of claim 1, wherein the fifth processing unit selects as the candidate command a command that satisfies at least one of a first condition that the priority is equal to or greater than a threshold value and a second condition that the priority is selected in descending order.

4. A scenario creation and management method for creating and managing, by an information processing device, scenarios for executing multiple applications that are the target of operational assistance using commands corresponding to the applications, the method comprising the steps of: managing the multiple commands using an object model in which multiple objects that constitute each command are represented in a tree structure; assigning labels to identify the target objects to target objects that are to be operated and monitored among the multiple objects for each command based on the object model, and managing the assigned labels using label management information represented in a tree structure; generating usage history information for the target objects and managing the generated usage history information in association with the labels corresponding to the target objects; when one of the objects is specified as a search condition, acquiring usage history information for the target objects located under the specified object in the tree structure based on the label management information, for each command, and calculating a priority based on the acquired usage history information; and selecting and outputting candidate commands to be search candidates from the multiple commands based on the priority.

Images