Automation program creation device, automation program creation method, automation program creation program, computer-readable recording medium, and device on which the program is recorded

Abstract

To facilitate management of created automation program.SOLUTION: A creation apparatus of an automation program that causes a series of operation for automation target software to be recorded and executed includes: a variable definition unit 42 for defining variables referred in sub-activities constituting the automation program; an automation program creation unit 25 for generating the automation program constituted by activities based on the variables defined by the variable definition unit 42; an outline activity display unit 43 for displaying the activities constituting the automation program generated by the automation program creation unit 25 and the sub-activities constituting the activities; and an outline label creation unit 44 for automatically creating, by a natural language, an outline label indicating an outline of the activities and the sub-activities displayed by the outline activity display unit 43 by acquiring variable information relating to the variables referred by any of the sub-activities.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an automation program creation device, an automation program creation method, an automation program creation program, a computer-readable recording medium, and a device on which the program is recorded. [Background technology]

[0002] In Robotic Process Automation (RPA), tasks that humans used to perform in a computer desktop environment are automated by having a program execute those tasks. Conventionally, a method for reflecting human tasks in a program has been known in which human operations are recorded in a form that can be recognized by a computer.

[0003] The basic concept of RPA is to reduce the amount of code that needs to be written compared to regular programming and to support users who are unfamiliar with programming in generating automation programs.

[0004] For example, in order to understand the flow of work or business, when the same label is assigned to multiple pieces of log data, a configuration has been proposed in which the pieces of log data are displayed as one piece of log data (see Patent Document 1).

[0005] However, this technology requires users to assign labels themselves, which can lead to problems, particularly when users do not correctly understand the content, making it difficult to assign accurate labels and understand the content even when looking at the labels. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2020-123048 Summary of the Invention [Problem to be solved by the invention]

[0007] An object of the present disclosure is to provide an automation program creation device, an automation program creation method, an automation program creation program, a computer-readable recording medium, and a device on which the program is recorded, which enable easy management of created automation programs.

[0008] According to one aspect of the present invention, an automation program creation device for recording and executing a series of operations on software to be automated can include: a variable definition unit for defining variables referenced in one or more sub-activities constituting the automation program; an automation program generation unit for generating an automation program consisting of one or more activities including one or more sub-activities based on the variables defined in the variable definition unit; a summary activity display unit for displaying one or more activities constituting the automation program created by the automation program generation unit and the sub-activities constituting the activities; and a summary label generation unit for automatically generating, in natural language, summary labels representing summaries of the one or more activities or sub-activities displayed in the summary activity display unit by acquiring variable information related to variables referenced in any of the sub-activities. With the above configuration, the activity summary representing the summary of each activity can be expressed in natural language somewhat automatically by using variables defined by the user during programming, thereby realizing an even easier-to-use environment.

[0009] According to another aspect of the present invention, there is provided an automation program creation method for creating an automation program for recording and executing a series of operations on software to be automated, the method comprising the steps of: defining, in a variable definition section, variables referenced in one or more sub-activities constituting the automation program; creating, in an automation program generation section, an automation program made up of one or more activities including one or more sub-activities based on the variables defined in the variable definition section; automatically generating, in a general label generation section, general labels representing summaries of the one or more activities and sub-activities constituting the automation program created by the automation program generation section, using natural language information relating to variables referenced in any of the sub-activities; and displaying, in a general activity display section, the general labels generated by the general label generation section. This allows the activity summaries representing the summaries of each activity to be expressed in natural language somewhat automatically by using variables defined by the user during programming, thereby realizing an even easier-to-use environment.

[0010] Furthermore, according to another aspect of the present invention, an automation program creation program for creating an automation program that records and executes a series of operations on software to be automated can be made to realize on a computer the following functions: a variable definition section defines variables referenced in one or more sub-activities that constitute the automation program; an automation program generation section creates an automation program consisting of one or more activities, including one or more sub-activities, based on the variables defined in the variable definition section; a summary label generation section automatically generates, in natural language, a summary label that represents an overview of one or more activities and sub-activities that constitute the automation program created by the automation program generation section, by obtaining variable information related to variables referenced in one of the sub-activities; and a summary label generated by the summary label generation section in a summary activity display section.

[0011] Furthermore, a computer-readable recording medium or device storing the program according to another aspect of the present invention is a medium storing the program according to any of the above aspects or a device storing the program. Examples of the recording medium include magnetic disks such as CD-ROM, CD-R, CD-RW, flexible disk, magnetic tape, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, Blu-ray (trade name), UHD BD (trade name), and HD DVD (AOD (trade name)), optical disks, magneto-optical disks, semiconductor memories, and other media capable of storing a program. The program also includes programs distributed by downloading over a network such as the Internet, as well as those stored on the recording medium. The device storing the program also includes general-purpose or dedicated devices on which the program is implemented in an executable form such as software or firmware. Furthermore, each process and function included in the program may be executed by computer-executable program software, or each process may be implemented by hardware such as a predetermined gate array (FPGA, ASIC), or by a combination of program software and partial hardware modules that implement some of the hardware elements. In this specification, a computer-readable medium also includes a non-transitory tangible medium or a transitory propagating signal. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram showing a series of operations to be automated. [Figure 2] 1 is a block diagram showing a hardware configuration of an automated program creation device according to an embodiment of the present invention; [Figure 3] FIG. 3 is a functional block diagram of the processor unit in FIG. 2. [Figure 4] FIG. 2 is a block diagram showing the software configuration of an automation program creation program according to an embodiment of the present invention. [Figure 5] FIG. 10 is a schematic diagram showing the interaction between the automation program creation program and spreadsheet software. [Figure 6] FIG. 1 is a schematic diagram showing interactions between an automation program creation program and a Web browser. [Figure 7] FIG. 10 is a schematic diagram showing an example of interaction between an automation program creation program and software to be automated. [Figure 8] 1 is a flowchart showing the procedure for creating an automation program using RPA software. [Figure 9] FIG. 10 is a schematic diagram showing an outline procedure creation screen. [Figure 10] 10 is a flowchart showing a procedure for setting detailed settings for importing spreadsheet software. [Figure 11] FIG. 10 is a schematic diagram showing a spreadsheet software acquisition range designation screen. [Figure 12] 10 is a flowchart showing the procedure for setting up transcription. [Figure 13] FIG. 10 is a schematic diagram showing a general transcription detail setting screen. [Figure 14] FIG. 10 is a schematic diagram illustrating an example of an operation record screen. [Figure 15] FIG. 10 is a schematic diagram showing an activity list screen. [Figure 16] FIG. 10 is a schematic diagram showing a screen for setting an operation record screen. [Figure 17] FIG. 10 is a schematic diagram illustrating an example of a table data transcription output detailed setting screen. [Figure 18] FIG. 18 is a schematic diagram showing a state in which the upper left cell of the table in FIG. 17 is selected. [Figure 19] FIG. 10 is a schematic diagram showing a state in which column name options are displayed. [Figure 20] FIG. 10 is a schematic diagram showing a table navigation screen. [Figure 21] FIG. 10 is a schematic diagram showing a table operation screen. [Figure 22] FIG. 22A is a schematic diagram showing an example of an expense list, and FIG. 22B is a schematic diagram showing an example of a newly generated table. [Figure 23] FIG. 2 is a schematic diagram illustrating tables to be joined. [Figure 24] This is a table in which the table in Figure 23 is joined by "customer code." [Figure 25]FIG. 10 is a schematic diagram showing tables to be stacked vertically. [Figure 26] This is a table that compiles the table in Figure 25 using the table stacking function. [Figure 27] This is a table that compiles the table in Figure 23 using the table stacking function. [Figure 28] FIG. 10 is a schematic diagram showing a table from which a difference is to be calculated; [Figure 29] FIG. 29 is a schematic diagram showing a table obtained by taking the differences in FIG. 28. [Figure 30] FIG. 29 is a schematic diagram showing a table extracted from common parts of FIG. 28. [Figure 31] FIG. 10 is a schematic diagram showing a state in which the title of an article is manually selected on a target Web page. [Figure 32] FIG. 32 is a schematic diagram showing a state in which titles of multiple articles have been automatically selected on the Web page of FIG. 31. [Figure 33] FIG. 10 is a schematic diagram illustrating an example of an output detail setting screen. [Figure 34] 33. FIG. 34 is a schematic diagram showing an example of a web scraping extract preview display displayed in the web scraping extract preview display field on the output detail setting screen of FIG. [Figure 35] FIG. 10 is a schematic diagram showing another example of a web scraping extraction preview display. [Figure 36] 10 is the first part of a flowchart showing the procedure for setting detailed Web import settings. [Figure 37] FIG. 15 is a schematic diagram showing a state in which options are displayed by clicking on the operation record screen of FIG. 14; [Figure 38] FIG. 38 is a schematic diagram showing a screen after selection from the input candidate list in FIG. 37. [Figure 39] FIG. 39 is a schematic diagram showing an example in which "Get value (variable name)" is selected in FIG. 38. [Figure 40] FIG. 10 is a schematic diagram showing how operation recording related to the menu of software to be automated is performed. [Figure 41] FIG. 10 is a schematic diagram showing how semi-transparent windows are displayed in overlapping relation to two applications. [Figure 42]FIG. 42 is a schematic diagram showing the positional relationship between the semi-transparent window and the application in FIG. 41. [Figure 43] FIG. 10 is a schematic diagram illustrating an example of a transcription source application having a table UI. [Figure 44] 10 is a schematic diagram showing a general activity display field in a state where a general activity detail display area is expanded. FIG. [Figure 45] 45 is a schematic diagram showing a general activity display field in a state where the general activity detail display area in FIG. 44 is collapsed. FIG. [Figure 46] This is a schematic diagram showing the state in which the "Set Variables" activity has been moved from "Low-Cost Excel Read" in Figure 44 to an external one. [Figure 47] 47 is a schematic diagram showing a general activity display field in a state where the general activity detail display area in FIG. 46 is collapsed. FIG. [Figure 48] FIG. 2 is a schematic diagram showing an editor screen. [Figure 49] FIG. 10 is a schematic diagram showing how an activity display field is automatically updated when source code is updated on the editor screen. DETAILED DESCRIPTION OF THE INVENTION

[0013] Embodiments of the present invention will be described below with reference to the drawings. However, the embodiments described below exemplify an automation program creation device, an automation program creation method, an automation program creation program, a computer-readable recording medium, and a device on which the program is recorded, embodying the technical concepts of the present invention. The present invention does not limit the automation program creation device, the automation program creation method, the automation program creation program, the computer-readable recording medium, and the device on which the program is recorded to the following. Furthermore, this specification does not in any way specify the components shown in the claims to the components of the embodiments. In particular, the dimensions, materials, shapes, and relative positions of components described in the embodiments are not intended to limit the scope of the present invention, and are merely illustrative unless otherwise specified. Note that the size and positional relationships of components shown in each drawing may be exaggerated for clarity. Furthermore, in the following description, the same names and symbols indicate components that are identical or of the same quality, and detailed description will be omitted as appropriate. Furthermore, the elements constituting the present invention may be configured with the same components, so that multiple elements can be served by one component, or conversely, the functions of one component can be shared by multiple components. [Embodiment 1]

[0014] The automation program creation device is a device that assists users in creating automation programs, commonly referred to as RPA (Robotic Process Automation), that automate tasks previously performed by users in a desktop environment of a personal computer. FIG. 1 is a conceptual diagram illustrating a method for creating an RPA program using the automation program creation device of the present invention to execute a transcription task that outputs data from a file or application to a form in a file or application. The spreadsheet software SS, web browser WB, and automation target software GS, located on the left side of FIG. 1, are source applications that display data to be transcribed in the transcription task. The spreadsheet software SS, web browser WB, automation target software GS, and email software MS, located on the right side of FIG. 1, are destination applications that output data to be transcribed in the transcription task. In this embodiment, the automation target software GS refers to all applications that can be operated by a user in a desktop environment. For example, the automation target software GS, which serves as the transcription source application in this embodiment, refers to any application that can be operated by a user other than the spreadsheet software SS and web browser WB. Furthermore, for example, the automation target software GS as the transcription destination application in this embodiment indicates an application that can be operated by the user other than the spreadsheet software SS, the web browser WB, and the email software MS.

[0015] In this embodiment, a transcription task is defined as a task of acquiring data from a source application and outputting it to a destination application. For example, a transcription task of transcribing data from a data file of spreadsheet software SS to another data file of the same spreadsheet software SS is shown in Fig. 1 as a task of acquiring data from a data file of spreadsheet software SS as a source application (shown on the left in Fig. 1) and outputting the data or data based on the data to a data file of spreadsheet software SS as a destination application (shown on the right in Fig. 1). Similarly, a task of transcribing data from a page being viewed in a web browser to a form in a specific software is shown in Fig. 1 as a task of acquiring data from a web browser WB as a source application (shown on the left in Fig. 1) and outputting the data or data based on the data to automation target software GS as a destination application (shown on the right in Fig. 1).

[0016] When a user creates an RPA program that performs such a transcription task, the user must consider which data to treat as variables and how to handle those variables. For example, the user must consider what data to acquire from the transcription-source application as what variables, how to edit the acquired data and process it into data to be output to the transcription destination, and how to output which variables to the transcription-destination application. Creating a program while considering how to handle variables can be difficult for users who are unfamiliar with programming. In this embodiment, as shown in FIG. 1, the data acquired from the transcription-source application is stored as a table, making it easy to specify how to handle variables. Details will be described later.

[0017] 2 shows the configuration of a general-purpose computer PC that functions as the automation program creation device of this embodiment. The computer PC includes a display unit 11 such as a display, an input unit 12, a bus 13, a communication unit 14, a processor unit 20, a memory unit 15, and a storage unit 16. An automation program creation program is installed in the computer PC, causing the computer PC to function as the automation program creation device. In this embodiment, the general-purpose computer PC functions as the automation program creation device, but a dedicated automation program creation device may also be configured as hardware.

[0018] The display unit 11 displays each application shown in FIG. 1 and screen information provided by the automation program creation program. The display unit 11 may be configured with, for example, a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. The display unit 11 may be configured with a touch panel, in which case the touch panel serves both as the display unit 11 and the input unit 12. The input unit 12 includes input devices such as a keyboard 12a, a mouse 12b, and a touchpad 12c. The bus 13 constitutes a data transmission path for exchanging data between components. The communication unit 14 is a component for data communication with external devices, and standardized communication interfaces and modules such as Ethernet (product name) and IEEE802.x can be used. In this way, connection with external devices can be performed wired or wirelessly. The processor unit 20 is a component that performs various calculations and processing, and can be suitably realized by a CPU, MPU, SoC, ASIC, or the like. The memory unit 15 expands load modules during execution of a computer program and stores temporary data generated during execution of the computer program. The memory unit 15 can be a RAM configured with volatile memory such as SRAM or SDRAM, or a ROM configured with non-volatile memory such as EEPROM. The storage unit 16 is a component for storing various information such as automation program execution files, data, and setting information. The storage unit 16 is configured with a hard disk, semiconductor memory, portable media, etc.

[0019] The input unit 12 also functions as a member for accepting input on an object selection acceptance screen superimposed and displayed by an object selection acceptance unit 38 (to be described later). (Processor unit 20)

[0020] 3 is a functional block diagram showing the functions of the processor unit 20 of the computer PC that functions as the automation program generation device of this embodiment. The processor unit 20 has an automation program generation unit 25 and an automation program execution unit 26. The automation program generation unit 25 is a functional block for generating an automation program, and has an outline procedure creation unit 21, an acquisition range designation unit 22, a transcription designation unit 23, an operation record setting unit 48, a variable definition unit 42, an outline activity display unit 43, an outline label generation unit 44, an outline name input unit 45, a source code display unit 46, and a code conversion unit 47. The automation program execution unit 26 is a functional block for executing the automation program generated by the automation program generation unit 25.

[0021] The outline procedure creation unit 21 is a functional block for setting an outline procedure that indicates the transfer of data between applications in the automated transcription work. The outline procedure creation unit 21 has a repetition setting unit 31. The repetition setting unit 31 is a functional block for setting a procedure that is to be repeated among the outline procedures set by the outline procedure creation unit 21.

[0022] The acquisition range designation unit 22 is a functional block for specifying the range of data in the transcription-source application in which data to be acquired exists. It displays screen information for designating the range and accepts user designations for the screen information. In this embodiment, data in the transcription-source application includes data contained in a file handled by the transcription-source application and data contained in screen information displayed by a function of the transcription-source application. For example, if the transcription-source application is spreadsheet software SS and data is acquired from a file displayed in a tabular format by the spreadsheet software SS, this corresponds to acquiring data contained in the file handled by the transcription-source application. In this case, a designation of the range in the file in which the data to be acquired exists is accepted. In particular, if data is acquired from a file displayed in a tabular format by the spreadsheet software SS, the range in which the data to be acquired exists is specified by specifying the rows and columns of the table. Furthermore, for example, if the transcription-source application is automation software GS and a character string displayed by a function of the software is acquired as data, this corresponds to acquiring data contained in screen information displayed by a function of the transcription-source application. In this case, a designation of the range in the screen information in which the character string to be acquired exists is also accepted. Although details will be described later, if it is difficult to specify data using a method specific to the target application, such as specifying data using rows and columns in spreadsheet software SS, the data acquisition range is specified using the function of operation record setting unit 48, which will be described later.

[0023] The acquisition range designation unit 22 also includes an automatic table recognition execution unit 35. The automatic table recognition execution unit 35 is a functional block that extracts data from files that are handled by spreadsheet software SS but have a format that makes it difficult for users to specify the range of data to be acquired because the data arrangement is inconsistent, and then formats the data into a format that makes it easier for users to specify the range. For example, in a file that contains both parts intended as a table and parts that are not, the unit recognizes the parts surrounded by ruled lines as the parts intended as a table and extracts the table. Furthermore, for example, the unit recognizes cells that are joined across multiple columns as cells with a single column and extracts the table. Furthermore, for example, when extracting data from multiple files, the unit adds a file name indicating the source of each row as a new column to extract the table. In this way, a table is generated that is formatted in a format that makes it easy for users to specify the range by appropriately applying known algorithms. The user specifies the range in which the data to be acquired exists by specifying rows and columns in the formatted table.

[0024] In this way, the data of the transcription source application for which the range of data to be acquired is specified by the acquisition range designation unit 22 is held in the memory unit 15 as table data.

[0025] When the transcription source application is a spreadsheet software SS, table data is generated by arranging data contained in the transcription source application for each column displayed on the spreadsheet software SS. Even when a table is recognized by the automatic table recognition execution unit, data is arranged based on the formatted table to generate table data. That is, the spreadsheet software SS is an application that arranges data according to rows and columns, which are rules that are easy to use for generating table data. When the transcription source application is a web browser WB, data viewed by the user through the web browser WB is arranged according to HTML tags. Therefore, table data is generated by arranging data contained in the transcription source application for each tag. That is, the web browser WB is an application that arranges data according to tag information, which is a rule that is easy to use for generating table data. Thus, when the transcription source application is an application that arranges data according to rules that are easy to use for generating table data, the table data is generated according to those rules. In this embodiment, examples of applications that arrange data according to rules that are easy to use for generating table data include, but are not limited to, the spreadsheet software SS and the web browser WB. If the data of the transcription-source application is arranged according to certain rules and these rules are well known, a method for generating table data from the data of the transcription-source application can be set in advance in the acquisition range specification unit 22, so that data can be arranged according to rules that are easy to use for generating table data. In this case, as with the spreadsheet software SS and the web browser WB, the type of transcription-source application is set by the outline procedure creation unit 21, and the corresponding table data generation method is applied, generating table data from the data of the transcription-source application. The data is arranged in the table data according to the rules for arranging data in the transcription-source application. The column names of the generated table data may be set appropriately by the user.

[0026] Furthermore, if the acquisition range designation unit 22 does not support the data arrangement rules of the automation target software GS, which is the transcription source application, for example, if the automation target software GS is an application used only within a certain company, it is difficult to generate table data in accordance with the data arrangement rules of the automation target software GS. In such cases, as described above, the data acquisition range is designated by the function of the operation record setting unit 48. When designating the data acquisition range using the function of the operation record setting unit 48, the data acquisition range and the column names of the table data are designated in accordance with the user's operation. More specifically, the position information or coordinate information in the object hierarchy related to the data designated by the user is associated with the designated column name. By associating them in this manner, table data is generated in which data included in the first screen information displayed by the automation target software GS and data included in the second screen information displayed by the automation target software GS and located at the same position as the data are arranged in the same column.

[0027] In this way, data from the transcription-source application for which the range of data to be acquired has been specified by the acquisition range designation unit 22 is acquired as table data by an appropriate method according to the type of the transcription-source application, and the table data is stored in the memory unit 15. The table data stored in the memory unit 15 is given a name so that it can be distinguished from other table data similarly stored in the memory unit 15, but the name may be assigned by the user. In this case, the acquisition range designation unit 22 accepts the name to be assigned to the table data.

[0028] The transfer specification unit 23 is a functional block for specifying data to be transferred to a transfer destination application. The transfer specification unit 23 includes a transfer destination specification unit 23a, a transfer unit specification unit 23b, a table operation unit 23c, and a transfer data preview unit 23d.

[0029] The transcription destination designation unit 23a is a functional block for designating a form into which data in the transcription destination application is to be written, and displays screen information for designating the form and accepts user designations for the screen information. In this embodiment, the form of the transcription destination application is the target where a user makes some designation based on the data to be transcribed in conventional transcription work, and includes a text box where the user inputs text data, a combo box where the user selects any option from predetermined options, and a radio button that indicates which of multiple options has been selected.

[0030] The transcription unit designation unit 23b is a functional block for designating the unit of transcription of data acquired from the range designated by the acquisition range designation unit 22. That is, the transcription unit designation unit 23b designates the range of data to be transcribed in one transcription. As described above, data to be transcribed to the transcription destination application is stored in the memory unit 15 as table data, so the transcription unit designation unit 23b accepts designation of the number of rows as the unit of data transcription. Note that, when transcription to the transcription destination application, the unit of which is designated by the transcription unit designation unit 23b, is subject to repetition in the outline procedure, the unit designated by the transcription designation unit 23b is treated as the unit of transcription.

[0031] The table operation unit 23c is a functional block for manipulating data stored in the memory unit 15 as table data and editing the data to be written to the transfer destination application. The operations on the table data by the table operation unit 23c include a filter operation that determines which rows to display depending on whether the data included in a certain column satisfies a predetermined condition, and a table join operation that joins multiple table data according to conditions such as common column items.

[0032] The transfer data preview unit 23d is a functional block that displays the table operated by the table operation unit 23c and the unit into which data specified by the transfer unit designation unit 23b is to be transferred to the table. The user checks the information displayed by the transfer data preview unit 23d to confirm the transfer unit that the user set and the results of the table operation.

[0033] The operation record setting unit 48 is a functional block for recording user operations on applications. The operation record setting unit 48 also functions as part of the acquisition range designation unit, and records operations by the user specifying the range of data to be acquired from the transcription source application. The operation record setting unit 48 also functions as part of the transcription destination designation unit 23a, and records operations by the user specifying the form into which data is to be written from the transcription destination application. Note that the user operations recorded by the operation record setting unit 48 are not limited to user operations on the transcription source application or the transcription destination application, and may record user operations on any application.

[0034] The operation recording unit 48 includes an object selection receiving unit 38 , an operation selecting unit 39 , an element identifying unit 40 , and an operation executing unit 41 .

[0035] The object selection accepting unit 38 accepts a user's selection of an object displayed by the automation target software GS. More specifically, it displays an object selection accepting screen that accepts user operations, superimposed on the display screen of the automation target software GS. When a user instructs an operation using a cursor displayed on the display screen of the computer PC functioning as the automation program creation device of this embodiment, the application on the display screen located in the foreground among the display screens corresponding to the cursor accepts the operation. Therefore, the object selection accepting unit 38 places the object selection accepting screen in front of the automation target software GS on the display screen of the display unit, thereby accepting user operations via a mouse, keyboard, etc., prior to the automation target software GS. Note that the user operations accepted by the object selection accepting unit 38 by displaying the object selection accepting screen are not limited to operations performed solely with the user's intention to select an object, but also include operations intended to instruct some kind of processing on the object. The former operation is an operation prompted to the user, for example, with a message stating, "Please select one of the objects." The latter operation is an operation prompted to the user, for example, with a message stating, "Please perform the operation to be performed on the automation target software GS in the task."

[0036] The object selection receiving screen in this embodiment is a translucent window TW that covers the entire display screen. The translucent window TW is preferably transparent enough to allow the user to visually recognize the automation target software behind the translucent window TW (see, for example, FIG. 14, which will be described later). This allows the object selection receiving unit 38 to receive operations intended to target the automation target software GS while allowing the user to visually recognize the automation target software GS and perform operations intended to target the automation target software GS. Furthermore, the translucent window TW is preferably not transparent enough to be visible when displayed over the automation target software GS. Because the object selection receiving screen receives operations on behalf of the automation target software GS intended by the user, the automation target software GS does not change in response to the operations, even though the user has performed the operations. This allows the user to visually grasp the current state in which the object selection receiving screen is accepting operations on behalf of the automation target software GS. Furthermore, the object selection receiving screen is preferably displayed so as to cover the entire display screen. According to this, even if there is a plurality of automation target software GS, as long as these plurality of automation target software GS are displayed on the display screen, it is possible to accept user operations without changing the range in which the object selection acceptance screen accepts operations. Therefore, there is no need for processes such as changing the position or size of the object selection acceptance screen depending on the automation target software GS to be operated.

[0037] When the selection acceptance screen is superimposed, the software to be automated may be displayed in a smaller window size instead of full-screen on the display screen. This configuration has the advantage that by displaying the software to be automated in a smaller size rather than full-screen, objects other than the software to be automated are easier to see.

[0038] The operation selection unit 39 is a functional block that enables the user to specify the operation content for an object selected by a user operation accepted by the object selection acceptance unit 38. For the operation record setting unit 48 to accurately record user operations, it is necessary to record the object targeted by the user operation and the operation content. In this embodiment, the object selection acceptance unit 38 accepts the user operation on an object selection acceptance screen displayed, and the operation content for the selected object is selected by the operation selection unit 39. This allows the operation record setting unit 48 to record the object targeted by the operation and the operation content for the object before the automation target software GS accepts the user operation. Therefore, even if an object dynamically changes in response to a user operation, such as when an object disappears due to an operation, the operation can be recorded regardless of changes in the display screen state. An output list window 161 that displays the operation content for the object selected by the operation selection unit 39 may also be displayed on the display screen (see, for example, FIG. 37, described later). This allows the user to check the operation content in the output list window 161 when creating an automation program, thereby providing the advantage of visually grasping the creation status.

[0039] The operation selection unit 39 can also display operation content options, which list multiple options for operations on a specified object. This automatically displays possible operations for the object specified by the user, allowing the user to select one. This limits and stabilizes the input content to a certain extent, improving the accuracy of recording by the operation record setting unit 48. The displayed selection options may also allow the user to select how the operation content is reproduced in the program. For example, an operation to input text into a form can be realized by the user operating the keyboard or pasting the contents of the clipboard. Using the options displayed by the operation options 39, the user can select which method to use to input text in the automation program, thereby ensuring reliable automation. In addition to inputting text, for example, when clicking on an object, the object can be identified using coordinate information or based on the hierarchical relationship of the objects that include the object. Therefore, it is preferable for the user to select the appropriate method.

[0040] When the operation record setting unit 48 functions as part of the function of the acquisition range designation unit 22, it is also possible to select storage in a specific memory area of ​​the memory unit 15 as an operation on an object that can be designated by the operation selection unit 39. In this case, the automation target software GS that is the target of user operation is the transcription source application. When the operation selection unit 39 selects storage of data in the memory unit, it is preferable to be able to set whether or not the data is stored as table data in the memory unit 15. Furthermore, when the data is stored as table data in the memory unit 15, it is preferable to be able to specify the table data to which the data acquired from the object belongs and the column of the table data in which the data acquired from the object is placed.

[0041] When the operation record setting unit 48 functions as part of the function of the transcription specification unit 23, transcription of tabular data stored in the memory unit 15 can also be selected as an operation for an object that can be specified by the operation selection unit 39. In this case, it is preferable to be able to select which column item of the tabular data to transcribe for the selected object, that is, to be able to assign the column item of the tabular data to an input form included in an object of the automation target software GS. In this way, assigning tabular data to each input form of an object of the automation target software has the advantage of making it easier to create an automation program.

[0042] The element identification unit 40 is a functional block for identifying the attributes of a selected object. When the object selection receiving unit 38 receives the selection of an object, the element identification unit 40 acquires information about the type of object the selected object is, using a library that associates objects with information related to the objects. In this embodiment, the library is, for example, Microsoft's UI Automation (product or service name).

[0043] The information acquired by the element identification unit 40 will be described using an example in which the selected object is a combo box. When the selected object is a combo box, the selected object has options, and the user can select information to input into the object from the options. The element identification unit 40 identifies the selected object as a combo box and acquires information about the options that can be input into the object as information related to the object. The information acquired by the element identification unit 40 is preferably displayed as options for the operation content displayed by the operation selection unit 39. When the selected object is a combo box, the object itself has options for information to be input into the object, and by reflecting the options in the options displayed in the operation selection unit 39, operations that can be executed on the software GS to be automated are specified, thereby improving the accuracy of recording user operations.

[0044] When the element identification unit 40 acquires information related to an object, it is preferable that the user's operation be reflected in the automation target software GS. This is because some objects allow information related to the object to be acquired more accurately by accepting a user's operation, and it is preferable to acquire information before and after the operation is accepted. Methods for reflecting the user's operation in the automation target software GS include temporarily hiding the object selection acceptance unit and having the operation record setting unit 48 execute the operation content in the automation target software GS.

[0045] The operation execution unit 41 executes an operation on an object of the automation target software GS, the operation on which is specified by the operation selection unit 39. The operation by the operation execution unit 41 is executed after the object is selected and the operation content for the object is selected. In addition, as described above, the operation execution unit 41 may execute an operation on the selected object during the process in which the element identification unit 40 acquires information related to the object.

[0046] The operation record setting unit 48 may also be able to accept an instruction to temporarily stop the operation record. This makes it possible to temporarily stop the operation record function as needed, thereby increasing the flexibility of operations on the software to be automated, for example by excluding operations that you do not want to record. Furthermore, in response to an instruction to temporarily stop the operation record, the superimposed display of the object selection acceptance screen may be canceled, and input from the input unit such as a mouse or keyboard may be accepted.

[0047] The automation program generation unit 25 also generates an automation program that executes an operation selected based on the operation selection unit 39 for a specified GUI element. The automation program generation unit 25 also generates an automation program made up of one or more activities including one or more sub-activities based on variables defined in the variable definition unit 42. The automation program execution unit 26 is a component that reads the automation program execution file generated by the automation program generation unit 25 and stored in the storage unit 16, executes the automation program execution file, and reproduces the recorded series of operations.

[0048] The variable definition section 42 is a component for defining variables to be referenced in any of one or more sub-activities that constitute an automation program.

[0049] The outline activity display unit 43 is a component for displaying one or more activities that constitute an automation program created by the automation program generation unit 25 and the sub-activities that constitute the activities. The outline activity display unit 43 can also group multiple sub-activities that constitute an automation program and display them as an activity. This makes it possible to automatically group a series of operations and display them on the outline activity display unit 43, making it possible to provide the user with an easy-to-view summary of even complex operations.

[0050] The outline label generation unit 44 is a component that acquires variable information relating to variables referenced in any of the subactivities and automatically generates, in natural language, an outline label 224 that represents an outline of one or more activities or subactivities displayed in the outline activity display unit 43. The outline label generation unit 44 can also display, in the outline activity display unit 43, variable information relating to variables referenced in a subactivity that is not included in the activity, or variables referenced in the activity.

[0051] With this configuration, an activity overview, which represents the outline of each activity, can be expressed in natural language somewhat automatically by using variables defined by the user during programming, thereby creating an even easier-to-use environment.

[0052] The general activity display section 43 can also be provided with a general activity headline section 222 that shows an outline of the general activity, and a general activity detail display area 225 that displays an outline of each activity displayed in the general activity headline section 222. This allows the user to easily check the outline, summary, and details of each activity.

[0053] Furthermore, the general activity display unit 43 can also be provided with a display switching unit that can switch between a detailed display mode that displays the general activity detailed display area 225 and a general display mode that hides the general activity detailed display area 225 for each general activity headline section 222. This makes it possible to switch between expanded display and collapsed display for each activity.

[0054] Furthermore, the general activity display section 43 can assign a different color to each variable included in the general label 224 displayed in the general activity headline section 222 or the general activity detail display area 225. This provides an environment in which the variables are color-coded and the overview of the automation program is more easily visible.

[0055] On the other hand, the general label generator 44 can also generate the general label 224 in natural language according to predetermined rules. Here, the predetermined rules may include at least one of the following: inputting an argument that is used internally; acquiring an argument that is assigned internally; and not determining an argument as an input if the argument does not refer to an external value through reassignment.

[0056] The outline name input section 45 is a component that allows the user to input, by means of a character string or the like, an outline name 223 that indicates the content of the activity, which is displayed in each outline activity headline section 222. This allows the user to assign any name that is easy for the user to understand to each activity, which makes the outline of the automation program easier to understand in addition to the automatically created natural language.

[0057] The source code display unit 46 is a component for displaying the source code of the automation program created by the automation program generation unit 25. The code conversion unit 47 is a component for converting the content described in the source code displayed in the source code display unit 46 into a summary activity displayed in the activity display field.

[0058] 4 to 7 are diagrams showing the relationship between software and data possessed by a computer PC that functions as an automation program creation device of this embodiment. The RPA software RS shown in FIGS. 4 to 7 is an automation program generation program. When the RPA software RS is installed, the processor unit 20 functions as an automation program generation unit 25 and an automation program execution unit 26, and is therefore shown in FIGS. 4 to 7 as a functional block included in the RPA software RS.

[0059] As shown in Figure 4, the RPA software RS acquires data from existing applications such as spreadsheet software SS, web browser WB, and other automation target software GS, and writes data to these existing applications. Data acquisition and writing to existing applications are performed by the functions of each functional block of the automation program generation unit 25 when an automation program is generated, and by the automation program execution unit 26 when the generated automation program is executed. The automation program generation unit 25 also generates table data based on data acquired from the transcription source application, stores the table data in the memory unit 15, and generates an automation program by setting the writing of data to the transcription destination application by referring to the table data stored in the memory unit 15. The automation program execution unit 26 executes the transcription task based on the generated automation program.

[0060] When using the RPA software RS to support the creation of automation programs, the RPA software RS exchanges data via API with an external program such as spreadsheet software SS, which reads data files such as xlsx, such as Microsoft Excel (product or service name), as shown in Figure 4. In addition, the web browser WB (data format is html files) and the RPA software RS communicate between processes using tools such as Selenium to retrieve, write, and manipulate data.

[0061] The automation program generation unit 25 also supports the creation of automation programs in table data format. The automation programs created by the automation program generation unit 25 are saved as RPA program files in the storage unit 16 or the like. The automation program execution unit 26 then reads out the saved RPA program files and executes the automation programs. This automatically executes a series of predefined operations.

[0062] FIG. 5 shows the relationship between the RPA software RS and the spreadsheet software SS. The RPA software RS communicates with the spreadsheet software SS via COM to operate the spreadsheet software SS. By operating the spreadsheet software SS, the RPA software RS obtains the values ​​of cells in a table displayed by the spreadsheet software SS as data acquisition from a file, and writes data to cells in a table displayed by the spreadsheet software SS as data writing. In this embodiment, the spreadsheet software SS is Excel, and the file displayed by the spreadsheet software SS has the extension .xlsx. In this case, the RPA software RS may write data to a file using the Excel file operation library process.

[0063] Figure 6 shows the relationship between the RPA software RS and the web browser WB. When retrieving data from a web browser, the RPA software RS has the user operate the web browser. At this time, it obtains the path of the element clicked by the user with a pointing device such as a mouse via an extension function previously added to the web browser WB. The RPA software RS uses an Xpath as a search expression based on the obtained path to extract the elements of the path corresponding to the Xpath. When the RPA software RS executes the created automation program, it identifies the path of the form displayed on the web browser WB via the web browser WB's extension function and transcribes data into the elements of the identified path.

[0064] When the RPA software RS generates an Xpath from a path obtained by the web browser WB's extension function, it generates an Xpath that can extract elements at the same level as the element corresponding to that path, and can extract multiple elements at the same level as the element specified by the user operating the web browser WB. In this case, if the user specifies multiple elements that are determined to be at the same level, an Xpath can be generated from the common part of the paths of the multiple elements, improving the accuracy of generating an Xpath suitable for extracting elements at the same level.

[0065] FIG. 7 shows the relationship between the RPA software RS and the automation target software GS, whose operations are automated using the operation record setting unit 48. The RPA software RS and the automation target software GS communicate with each other via UIAutomation. UIAutomation is a library provided by Microsoft. When creating an automation program, the automation program generation unit 25 accepts user operations performed on the automation target software GS. More specifically, the automation program displays a translucent window TW in front of the automation target software GS, allowing the display information of the automation target software GS to be visible. In this embodiment, the computer PC accepts operations through the application corresponding to the screen information displayed in the foreground. Therefore, operations intended by the user to be performed on an application behind the translucent window TW are accepted as operations on the translucent window TW. The automation program generation unit 25, which accepts user operations using the translucent window TW function, passes the mouse position specified by the accepted user operation to UIAutomation. As a result of the mouse position being passed, UIAutomation receives information related to the UI element of the user interface at the mouse position. The information related to the UI element here includes the position of the UI in the hierarchy and UI attributes (for example, whether data can be entered or whether it is a combo box). Based on this information related to the UI element, the automation program generation unit 25 displays operation content options indicating the content of the operation to be performed on the UI element. The automation program generation unit 25 accepts the operation content selected by the user for the UI element, records the operation content and the UI element in association with each other, and returns the operation content and information specifying the UI element to UIAutomation, causing it to execute the operation on the automation target software GS.

[0066] When the created automation program is executed, information specifying the UI element and the operation content associated with the UI element are sent to UIAutomation, and a predetermined operation is executed on the UI.

[0067] Note that when the RPA software RS supports the creation and execution of an automation program for the automation target software GS, the method is not limited to using UIAutomation or the like. For example, an API provided by the OS, which is the environment in which the automation target software GS and the RPA software RS are executed, may be used. As an example, the Win32API (product or service name) provided by the Windows (product or service name) OS may be used to monitor operation events sent from the OS to each application and generate an automation program. Furthermore, the operation contents indicated in the automation program may be performed on the automation target software GS via the Win32API, thereby executing the automation program.

[0068] The RPA software RS of this embodiment has multiple modes for creating automation programs. One of the modes is navigation mode, which is a guided mode for creating an automation program while guiding the user through the steps required to create the automation program that matches the automation program the user is trying to create. Figure 8 shows a flowchart for creating an automation program that performs transcription work using the RPA software RS in navigation mode.

[0069] First, in step S901 of FIG. 8, the outline procedure creation unit 21 displays an outline procedure creation screen as screen information for the user to create an outline procedure. (Outline procedure creation screen 110)

[0070] FIG. 9 shows an example of the outline procedure creation screen 110. The user operates the outline procedure creation screen 110 to create an outline procedure. The outline procedure created by the user corresponds to a transcription task, and the user specifies a transcription source application and a transcription destination application. Therefore, even if an option "Extract data from spreadsheet software SS" is displayed and the user selects this option, it can be assumed that the spreadsheet software SS has been selected as the transcription source application. In this way, the automation program creation device of this embodiment prompts the user to create an outline procedure by displaying, as options, the applications used in the transcription task, which are relatively easy to specify. In addition, screen information is displayed for the user to specify information necessary to retrieve data from the transcription source application and information necessary to write data to the transcription destination, according to the outline procedure created by the user selecting an application. This reduces the difficulty for the user to create an automation program.

[0071] The outline procedure creation screen 110 will be described in detail using Figure 9. By first creating an outline scenario and then creating a detailed scenario, it becomes easier to create a scenario based on the user's workflow. The items in the center of the screen are selectable outline operations (activities). Of the outline operations listed in the operation candidate list 112 in the center of the screen, you can add the desired item to the operation selection list 114 on the right side of the screen by clicking on it to select it and then pressing the "Add" button 113. Furthermore, if you press the "Next" button 117, and activities have been added to the operation selection list 114 on the right, you will be transitioned to a screen for making detailed settings for each outline operation in the order of the added items. On the other hand, if there are no activities in the operation selection list 114, no transition will occur. Pressing the "Discard and return" button 118 will cancel the creation navigation settings and exit.

[0072] On the outline procedure creation screen 110, an outline procedure icon 111 in the operation candidate list 112 is added to the operation selection list 114, thereby selecting a transcription source application and a transcription destination application. In the example of FIG. 9, when the outline procedure icon 111 called "spreadsheet software" in the form "What do you want to extract data from?" is added to the operation selection list 114, the spreadsheet software SS is selected as the transcription source application. The icon called "Web" corresponds to the web browser WB, and the icon called "any application" corresponds to the automation target software GS. Also, in the example of FIG. 9, when the outline procedure icon 111 called "spreadsheet software" in the form "What do you want to write data to?" is added to the operation selection list 114, the spreadsheet software SS is selected as the transcription destination application.

[0073] The outline procedure creation screen 110 also includes a repetition setting icon 119. The user creates an outline procedure that includes steps to be repeatedly executed by combining the outline procedure with the repetition setting icon 119. The repetition setting unit 31 displays screen information related to repetition based on the outline procedure that includes the repetition setting icon 119. In FIG. 9, when the user selects the "Make specified range repeatable" button 115 while selecting an outline procedure icon 111 arranged as an outline procedure, the repetition setting icon 119 is inserted above the selected outline procedure. The selected outline procedure icon 111 is displayed one level lower than the repetition setting icon 119, indicating that it is an outline procedure for which repetition has been set.

[0074] In the example of the first embodiment, the spreadsheet software SS is selected as the transcription source application, and the spreadsheet software SS and the web browser WB are selected as transcription destination applications. Also, some of the operations for writing data to the automation target software GS, the spreadsheet software SS, and the web browser WB as transcription destination applications are set to be repeated.

[0075] As shown in FIG. 8, after displaying the outline procedure creation screen 110 in step S901, it is determined in step S902 whether the creation of the outline procedure has been completed. If not, step S902 is repeated; if completed, the process proceeds to step S903.

[0076] In step S903, the automation program creation unit determines whether a transcription source application was selected in step S901. If the automation program creation unit determines that a transcription source application has been selected, it starts setting the acquisition range in step S904 according to the application selected as the transcription source application. After the automation program creation unit completes setting the acquisition range in step S904, it proceeds to step S905. Also, if the automation program creation unit determines in step S903 that a transcription source application has not been selected, it proceeds to step S905. (Setting the import range for spreadsheet software SS)

[0077] FIG. 10 is a flowchart showing details of the acquisition range specification executed by the acquisition range specification unit 22 when the spreadsheet software SS is selected as the transcription source application. In step S1101, the acquisition range specification unit 22 determines whether the mode is an automatic table acquisition mode in which automatic table recognition is performed. If the acquisition range specification unit 22 determines that the mode is an automatic table acquisition mode, in step S1102, the acquisition range specification unit 22 performs automatic table recognition using the function of the automatic table recognition execution unit. The automatic table recognition is performed, a formatted table is generated, and the process proceeds to step S1103. If the acquisition range specification unit 22 determines in step S1101 that the automatic table acquisition mode is not in the automatic table acquisition mode, the process proceeds to step S1103. In step S1103, the acquisition range specification unit 22 specifies the range in which the data to be acquired exists, in the table formatted by the automatic table recognition execution unit or in the table displayed in table format by the spreadsheet software SS. After the acquisition range is specified in step S1103, the acquisition range specification unit 22 specifies the name of the table data to be generated from the acquired data in step S1104. In step S1104, the acquisition range designation unit 22 may accept a name for the table data from the user, or may designate a name that can be distinguished from other table data already stored in the memory unit 15. In step S1105, the acquisition range designation unit 22 generates table data in which the top row of the acquisition range designated in step S1103 is used as a column heading, and assigns the name designated in step S1104 to the table data and stores it in the memory unit 15.

[0078] FIG. 11 shows a spreadsheet software acquisition range specification screen 130 displayed by the acquisition range specification unit 22 when the automatic table acquisition mode is not selected. This diagram is a screen for accepting user operations in steps S1103 and S1104 in FIG. 10 described above. In FIG. 11, a form labeled "File Name," "Sheet Name," and "Cell" is displayed to accept the specification of the table range to be executed in step S1103. Also, in FIG. 11, a form labeled "Name of Extracted Table" is displayed to accept the specification of the name to be executed in step S1104. In the example of FIG. 11, the name "Expense List" is specified. Furthermore, in FIG. 11, a trigger button 138 for executing step S1105 is provided. The name of the table data specified in step S1104 is a variable name for retrieving the table data from the memory unit 15 in the RPA software RS.

[0079] 8, after the acquisition range has been specified, the automation program creation unit determines in step S905 whether a transcription destination application was selected in step S901. If the automation program creation unit determines that a transcription destination application has been selected, it executes transcription settings in step S906. After completing transcription settings in step S906, the automation program creation unit proceeds to step S907. If the automation program creation unit determines in step S905 that a transcription destination application has not been selected, the automation program proceeds to step S907.

[0080] 12 is a flowchart showing details of settings related to transcription executed by transcription designation unit 23 when an arbitrary application is selected as the transcription destination application. In step S1301, table data to be called from memory unit 15 is selected. In step S1302, table operation unit 23c functions to accept an operation on the table data selected in step S1301 and edit the table data in accordance with the operation. In step S1303, transcription unit designation unit 23b functions to designate a unit to be transcribed to the transcription destination application for the table data called in step S1301 or the table data edited in step S1302. After accepting the designation of the unit to be transcribed in step S1303, transcription destination designation unit 23a functions to designate a form to which data will be transcribed in the transcription destination application.

[0081] In step S1304, the transcription destination designation unit 23a displays screen information for accepting the user's designation of the transcription destination form according to the transcription destination application. If the transcription destination application is spreadsheet software SS, it accepts designation of the transcription destination file, sheet, cell, etc. If the transcription destination application is web browser WB, it launches the web browser WB, acquires user operations on the web browser WB via an extension function of the web browser WB, and identifies the form selected by the user on the web browser WB. If the transcription destination application is automation target software GS, it identifies the form using the function of the operation record unit.

[0082] Figure 13 shows a general transcription detail setting screen 150 that is displayed as one form of transcription setting in step S906. The general transcription detail setting screen 150 shown in this figure shows one form of the transcription specification unit 23. The general transcription detail setting screen 150 is used to set details of data transcription. In the example of Figure 13, the general transcription detail setting screen 150 displays an activity display field 121 on the left side of the screen, and a transcription detail specification field 152 from the center to the right of the screen. In this example, the general transcription detail setting screen 150 displays a detailed setting screen for the activity selected in the activity display field 121, in this case "Record operations and perform input / output."

[0083] The details of data transcription, in which data within the range set in the acquisition range designation unit 22 in FIG. 3 is transcribed as source data in a table format, are designated by the user in the transcription designation unit 23.

[0084] The transcription detail specification field 152 in FIG. 13 is provided with a transcription data specification field 153 and an operation execution determination field 156 as one form of the transcription specification unit 23 . (Transfer data specification field 153)

[0085] The transfer data specification field 153 has fields corresponding to the table data selection executed in step S1301, the table operation executed in step S1302, and the transfer unit specification executed in step S1303 shown in the flowchart of Fig. 12. By operating the transfer data specification field 153, the user can set details of the data to be transferred to the transfer destination application.

[0086] The transfer data designation field 153 has a table data reference / non-reference designation field 155 and a table data designation field 153b. The user can designate whether or not to reference table data stored in the memory unit 15 by operating the table data reference / non-reference designation field 155. In the example of FIG. 13, referencing table data is selected. Furthermore, the user operates the table data designation field 153b to designate table data including data to be transferred or data for generating the data to be transferred. In other words, the transfer data designation field corresponds to step S1301 in the flowchart of FIG. 12 described above. The example of FIG. 13 indicates that table data named "Expense List" is selected.

[0087] The transcription data specification field 153 also has a transcription unit preview field 154 as one form of the transcription unit preview section 24. The transcription unit preview field 154 displays the format of the data to be transcribed based on the detailed settings of the transcription data set by the user through operation of the transcription data specification field 153. The transcription unit preview field 154 is also a component that allows the user to edit the table data in step S1302 of the flowchart in Figure 12 described above. When the user operates the transcription unit preview field 154, the table navigation screen 380 in Figure 20 is displayed, and the user's table operations are accepted.

[0088] The transfer data specification field 153 includes a transfer unit specification field 153c as one form of the transfer unit specification section 23b. By operating the transfer unit specification field 153c, the user specifies the target to be transferred in a single data transfer to the transfer destination application. In other words, the transfer unit specification field 153c corresponds to step S1303 in the flowchart of FIG. 12. The transfer unit from the table data specified in the table data specification field 153b can be selected from "row by row," "entire table," and "grouping." If "row by row" is selected, the table data for the variable name selected in the table data selection field 153b is transferred line by line in a single transfer. If "entire table" is selected, the entire table data is transferred in a single transfer. Furthermore, if "grouping" is selected, data from rows with the same column value in the table data is entered. For this reason, "grouping" allows the selection of column items in the table data for the variable name selected in the table data selection field 153b. 13, the column item "Date" is displayed along with the option "Grouping." In this case, in one transcription, data with the same value in the column named "Date" among the table data of the variable name selected in the table data selection field 153b is transcribed. (Operation decision column 156)

[0089] Furthermore, in the transcription detail specification field 152 in Fig. 13, an operation determination field 156 is provided below the transcription unit specification field 153. The operation determination field 156 is a component that allows the user to specify a form as a data transcription destination in step S1304 of the flowchart in Fig. 12 described above.

[0090] The operation determination field 156 is displayed when automation target software GS is specified as the transfer destination application. The user initiates operation recording from the operation determination field 156 using the operation record setting unit 48 to specify a form in the automation target software GS. The operation determination field 156 shown in FIG. 13 is a component for determining operations that will not be repeated during execution of the automation program. However, a separate operation determination field for determining operations that will be repeated may also be provided. In this case, it is preferable that the operation determination field for determining operations that will not be repeated and the operation determination field for determining operations that will be repeated are displayed on the same screen. Here, "displayed on the same screen" includes a layout in which the two operation determination fields can be displayed by scrolling. This allows the user to set both operations that will be repeated and operations that will not be repeated from the same screen, making it easy to create automation programs for tasks that involve a mixture of repeated and non-repeated operations. (Operation Record Screen 160)

[0091] FIG. 14 shows an example of a display screen including an object selection receiving screen displayed by the function of the object selection receiving unit 38 included in the operation record setting unit 48. In FIG.

[0092] The operation record screen 160 shown in FIG. 14 displays an "Expense Settlement" form GS1 displayed by an expense settlement application serving as the automation target software GS. The operation record setting unit 48 displays a semi-transparent window TW in front of the object selection reception screen generated by the automation target software GS. The semi-transparent window TW may be transparent as long as it is transparent enough to allow the user to see the screen information that overlaps the semi-transparent window TW and is located behind the semi-transparent window TW. Displaying a semi-transparent window allows the user to recognize that some screen information is being displayed superimposed on the screen information generated by the automation target software GS, so it is preferable that the semi-transparent window TW is not transparent enough to allow the user to see it. Note that in FIG. 14, the frame of the semi-transparent window TW is inside the frame of FIG. 14 to clearly show the semi-transparent window TW on the diagram, but in reality it is displayed on the entire screen.

[0093] The user performs an operation on the operation record screen 160. More specifically, the operation record setting unit 48 accepts the user's operation targeting a specific object included in form GS1 through the semi-transparent window TW, and identifies the object in form GS1 selected by the user. This allows the acquisition range designation unit 22 to specify the acquisition range of data in the automation target software GS, which is the transcription source application. This also allows the transcription destination setting unit to identify the form into which data is written in the automation target software GS, which is the transcription destination application. In the example of Figure 14, the form with the description "Applicant" among the "Expense Reimbursement" form GS1 of the automation target software is the selected object. To make the operation easier to understand, the position of the mouse cursor is represented by a circle.

[0094] FIG. 37 shows a state in which, after the object selection receiving unit 38 has received the selection of an object, the operation selecting unit 39 has displayed operation content options 171 that indicate selectable operations for the object.

[0095] The operation content options 171 have options for the operation content to be performed on the object. In FIG. 37, these are "mouse operation" 171a and "input" 171b. The mouse operation may have more detailed operation content such as "click" and "double click". These "click" and "double click" can be displayed in a submenu from "mouse operation" 171a.

[0096] Furthermore, the operation content options 171 have options indicating how to implement the operation when executing the automation program. In Fig. 37, the options for "input" 171b include "text input" 172a and 172c, which reflect the user's keyboard operation, and "paste input" 172b and 172d, which reflect the contents of the clipboard. This makes it easier for the user's intentions to be reflected in the automation program.

[0097] Furthermore, the operation content options 171 display options for selecting information to be input for the selected object from table data stored in the memory unit 15. In FIG. 37, these options are "Table input (column name): Applicant" 172e and "Table input (column name) Amount" 172f. These options are preferably displayed in FIG. 37 when the automation target software GS is treated as the transfer destination application. This allows the transfer destination assignment to be easily set using the table data stored in the memory unit 15.

[0098] Here, the automation target software GS in Fig. 37 is an application that displays a character string in an application ID form 186 when an application button 187 is clicked after information has been entered in each form from the "Applicant" form to the "Amount" form, as shown in Fig. 38. Therefore, for example, when obtaining a character string to be displayed in the application ID form 186, the operation execution unit 41 executes the selected operation for the selected object one by one, making it easy for the user to select the character string to be displayed in the application ID form 186.

[0099] FIG. 39 shows a screen displayed when acquiring a display string in the application ID form 186. In this embodiment, the operation selected by the operation selection unit 39 is reflected in the automation target software GS by the operation execution unit 41, making it easy for the user to understand that the desired character string will be displayed in the application ID form 186. The user selects the application ID form 186, and selects the option "Get value" 191a from the operation content options 191 displayed by selecting the application ID form 186, thereby storing the character string in the application ID form 186 in the memory unit 15. At this time, it is preferable that the user be able to set the format in which the acquired data will be stored in the memory unit 15. FIG. 39 shows a display that accepts the setting of the variable name to be stored in the memory unit 15. It is preferable that, in addition to the variable name, the user be able to specify the table data containing the data and the position in the table data in which the acquired data will be placed. Here, selecting the option "Get value (variable name: )" from the options in the operation content options 191 allows the user to specify a variable name. This option constitutes one aspect of the variable definition unit 42. The user enters any character string as the variable name.

[0100] After the form into which the data is to be written is specified, the operation record setting unit 48 displays screen information that allows the user to select the data to be written into the form based on the table data called from the memory unit 15 by the transcription designation unit 23. When the table data is edited by the table operation unit 23c, the screen information may be displayed based on the edited table data. Alternatively, the screen information may be displayed based on the unit designated by the transcription unit designation unit 23b.

[0101] The user selects the data to be written into the form from the options for data to be written displayed by the operation record setting unit 48. The operation record setting unit 48 accepts the user's selection and associates the identified form with the data to be written into the form. As a result, the transcription destination designation unit 23a designates the form in the transcription destination application into which the data is to be written and the data to be written into the form.

[0102] In this embodiment, the operation of writing data to the automation target software as the transcription destination application is set as a repeated process. Therefore, the automation program generated by the automation program generation unit is output as a program that repeatedly executes the process of writing data in units specified by the transcription unit designation unit into a form specified by the transcription destination designation unit 23a. More specifically, after the process of transcribing data in the specified units into a form displayed by the automation target software GS is executed, an automation program is generated that writes data moving to the next data in the specified units into a similar form similarly displayed by the automation target software GS.

[0103] The operation record setting unit 48 writes the data selected by the user into the specified form. By executing the automation target software immediately after selecting an operation via the operation record screen, the user can check whether the selected data is written into the specified form without any errors, which also makes debugging easier.

[0104] As shown in FIG. 8, the automation program creation unit proceeds to step S907 to determine whether there are any operation record procedures for which details have not been set. The operation record procedures for which the presence or absence is determined are not the operation record procedures included in the setting of the acquisition range of the transcription source application in step S904 or the transcription setting to the transcription destination application in step S906, but are procedures set as procedures corresponding to user operations performed before and after the procedure for transcribing data during transcription work. If it is determined in step S907 that there are any unset operation record procedures, the process proceeds to step S908 to execute operation record setting. In step S908, a semi-transparent window TW is displayed by the function of the operation record setting unit 48, and user operations performed on the semi-transparent window TW are recorded. After recording of the user operations in step S908 is completed, the process proceeds to step S909 to generate screen information indicating the contents of the automation program based on the contents set in at least one of steps S901, S904, S906, and S908. If it is determined in step S907 that there are no unset operation record procedures, the process proceeds to step S908, where screen information showing the contents of the automation program is generated based on the contents set in at least one of steps S901, S904, and S906. (Activity list screen 220)

[0105] FIG. 15 shows an activity list screen 220 displayed in step S909, which shows the contents of the automation program to be created. On the activity list screen 220, the created procedures are displayed, grouped into the general procedure units set in step S901. The activity list screen 220 has a general activity display field 221, which is one aspect of the general activity display unit 43, on the left side of the screen. In the general activity display field 221, the created activities are displayed, grouped into general activities. On the activity list screen 220 in FIG. 15, variables are displayed in orange. In this embodiment, the table data itself is treated as a variable in the generated automation program, and therefore the name given to the table data itself is treated as a variable name. Furthermore, each column item included in the table data is also treated as a variable name because it is referenced during transcription.

[0106] 8, after displaying the activity list screen 220 in step S909, the automation program creation unit creates an automation program in step S910. The automation program is generated based on the content set in at least one of steps S901, S904, S906, and S908. [Embodiment 3] (Operation Record Screen 280)

[0107] 16 shows an operation record screen 280 when the form of the transcription destination application specified by the user through the function of the transcription destination specification unit 23a is a table UI displayed in a table format. Note that in this figure, the semi-transparent window is not displayed for the sake of explanation.

[0108] When the operation record screen 280 in FIG. 16 detects that the user has selected a table UI with the mouse, a table data transfer details setting window 301 is displayed, as shown in the operation record screen 300 in FIG. 17, for setting details for transferring table data to the table UI. The table data transfer details setting window 301 includes a table input position specification field 302 and a table selection field 303. First, to specify the table UI to output, the user is prompted to select the upper left cell of the table in the table input position specification field 302. After the user's selection is detected and the upper left cell of the table UI is specified, the operation of selecting the upper left cell of the table UI is reflected in the table UI. In FIG. 18, the daily report management application, which is the automation target software GS, displays the selected cell in the table UI in a blue frame. Furthermore, the table input position specification field 302 displays "Selected," indicating that the selection of the upper left cell of the table is complete.

[0109] Next, select a table. In the Table Data Transfer Advanced Settings window 301 in Figure 18, the table selection field 303 has a table movement direction specification field 304, a row insertion specification field 305, a row deletion specification field 306, and a "Try moving the table" button 307. Pressing the "Try moving the table" button 307 allows you to check whether the table UI has been selected correctly. In this check, the RPA software accepts a user operation using the corresponding keyboard key, such as "Right: →", and you can check whether the selected area moves in the table UI when you operate the keyboard.

[0110] The table displayed in the RPA software corresponds to the table UI of the target software, i.e., the table UI of the daily report management app that is the transfer destination application. As an example, when transferring data to the second column from the left of the table UI, the operation record screen 320 in Figure 19 shows the state in which the data to be transferred to this column is selected. This operation record screen 320 allows you to see how each column in the table UI of the target software GS corresponds to each column of the table data imported for RPA program creation in the RPA software, as well as the appearance of the setting screen at that time. In the figure, the horizontal bar L1 indicates that the table UI of the target software GS is reflected in the setting screen of the RPA software RS. Below that, the column name options 321 display the names of the column items in the imported table data as options. The column name options 321 allow you to select the column name corresponding to the table data to be transferred to the target software GS when the RPA software RS is executed. For example, if you select and set "Item" from the column name options 321 displayed in the pull-down menu of the second column from the left of L1, when you run the RPA software RS, "Item" from the imported table data will be transcribed into the second column from the left of the automation target software GS. By setting it in this way, the data in the table data that will be written to a specific column in the table UI displayed in the automation target software GS is specified.

[0111] The settings made on the operation record screen 320 are replaced with the corresponding user operations and recorded. For example, assume that the table UI displayed on the operation record screen 280 is such that pressing the F2 key places the selected cell in an input state, pressing the Tab key moves the selected cell one column to the right, pressing the Shift key and the Tab key together moves the selected cell one column to the left, and pressing the ↓ key moves the selected column one row down. When the items "unit price," "item," and "delivery date" are associated in the operation record screen 320, starting from the leftmost column and proceeding to the right, the settings are replaced with the following operations and recorded. [operation] Click in the top left corner of the table. Repeat the following for each row: Press F2 key (to enter input mode) Paste: "Unit Price" Press Tab key once Press F2 key (to enter input mode) Paste: "Item" Press Tab key once Press F2 key (to enter input mode) Paste: "Delivery date" Press Shift+Tab three times Down arrow key press (Table operation)

[0112] Next, we will explain in detail how to edit table data using the table operation unit 23c. The function of editing table data stored in the memory unit 15 is called "table navigation." For example, on the general transcription detailed setting screen 150 in Figure 13, table operations can be called up to edit table data.

[0113] 20, it is also possible to select and execute the type of table operation that corresponds to the desired table operation. The table data after the operation is executed is stored in memory unit 15. At this time, the table data stored in memory unit 15 may be given a name specified by the user and stored in memory unit 15, or may be given a name that can be distinguished from table data already stored in memory unit 15 and stored in memory unit 15.

[0114] Furthermore, when multiple pieces of table data are stored in memory unit 15 and multiple pieces of table data are selected in the table data selection in step S1301 in the flow shown in Fig. 12, it is also possible to combine the multiple pieces of table data or extract the differences between the multiple pieces of table data. Fig. 21 is another example of Fig. 20. For example, on table operation screen 390 in Fig. 21, when an expense list such as that in Fig. 22A is generated as Table 1, selecting "narrow table by conditions" generates an activity "narrow Table 1 by conditions."

[0115] For example, if you create a scenario in which expenses over 1,000 yen are only submitted for application, you can narrow down the expense list in Figure 22A by setting "Filter expense list by amount > 1,000," and a table like the one in Figure 22B will be generated. The newly generated table in this way can also be used as a variable. For example, if you name the table in Figure 22B "List of expenses subject to application," you can use this table in other activities.

[0116] This function can also be used to specify branching. For example, with normal branching, the user must identify the location in the code where branching should occur (in this case, "Amount > 1000"), which is a time-consuming process. In contrast, this function allows the data to be treated as a single table, so by using it in conjunction with the loop process in the creation navigation, it is possible to perform branching operations without having to insert the branching process yourself. (Table operation function)

[0117] In addition, various operations can be performed on multiple tables, such as joining, stacking, extracting differences, extracting common parts, narrowing down tables, adding columns, and deleting columns. These table operation functions are executed by the table operation unit 23c in Fig. 3. As an example of the table operation function, a table joining function for joining multiple tables will be described below with reference to Figs. 23 and 24. (Table join function)

[0118] The table join function allows you to join multiple tables using a specific column name. For example, if you have two tables as shown in Figure 23, you can join them using a specific column name, in this case "Customer Code," to generate the table shown in Figure 24.

[0119] In this way, by combining table data, it is possible to generate an automated program for the transcription work that treats data in table data stored as separate table data as associated data. In the example of Figure 23, table data containing "customer name" and table data containing "email address" are stored as separate table data. The table merging function generates an automated program for the transcription work that transcribes "customer name" and "email address" related to that "customer name" into multiple forms on the same screen. (Table stacking function)

[0120] The table stacking function also allows multiple tables that specify the same items to be linked in the column direction. For example, as shown in Figure 25, two tables that have common items (here, three items: "Customer Name," "Customer Code," and "Status") can be combined into a single table as shown in Figure 26 by using the table stacking function. Note that, as shown in Figure 23 above, if the table stacking function is used on tables that have uncommon items, the cells for the non-existent items will be blank, as shown in Figure 27.

[0121] In this way, by stacking table data vertically, it is possible to generate an automated program for transcription work that treats the same type of data stored in separate table data as data stored in the same table data. In the example of Figure 25, the two tables all have the same column items, so the data in each row is the same type of data. Because the table stacking function treats the data as a single table data as shown in Figure 26, by specifying transcription for the table data using the transcription specification unit 23, an automated program for transcription work that performs the same transcription for multiple table data is generated. (Differential extraction function)

[0122] Next, we will explain the differential extraction function, which extracts the differences between multiple tables. The differential extraction function retains rows in a specific column of one table that do not exist in a specific column of the other tables. For example, consider the two tables shown in Figure 28, where the field names for customer name, customer code, and status are recorded in Table 1 on the left, and customer code and email address are recorded in Table 2 on the right. To extract only those customers for which there are no email addresses, use the differential extraction function. Here, rows for which there is no customer code in Table 2, a specific column in Table 1, are extracted from Table 1. As a result, a table is created, as shown in Figure 29, listing pseudonymous banks for which there are no customer codes in Table 2 and no email addresses.

[0123] In this way, by extracting differences from multiple data tables, it is possible to generate an automated program for the transcription process, which transcribes the data resulting from narrowing down data from one data table to another data table based on a condition. In the example of Figure 28, the table on the left is narrowed down based on the table on the right. This makes it possible to generate an automated program for transcribing data from the data table on the left that does not have a corresponding "customer code" in the data table on the right. (Common part extraction function)

[0124] Next, we will explain the common part extraction function, which extracts common parts of tables. With the common part extraction function, for multiple tables, only rows in a specific column of one table that exist in a specific column of another table are retained. For example, consider an example where only customers with email addresses are extracted from the table shown in Figure 28 above. Here, from the customer code, which is a specific column in Table 1, rows where the customer code in Table 2 exists are extracted from Table 1. As a result, a table is created, as shown in Figure 30, that lists pseudonymous manufacturing companies whose customer codes exist in Table 2 and whose email addresses exist.

[0125] In this way, by extracting common parts from multiple table data, similar to the differential extraction function, it is possible to generate an automated program for the transcription process that transcribes the data resulting from narrowing down data from one table data to another table data based on a condition. In the example of Figure 28, the table on the left is narrowed down based on the condition of the table on the right. This makes it possible to generate an automated program that transcribes data from the table data on the left that has a "customer code" that corresponds to the data in the table data on the right.

[0126] In the examples of Figures 23, 25, and 28 mentioned above, two tables are arranged horizontally, i.e., in the row direction, but it goes without saying that this is not limited to this, and that table operation functions such as the table join function, table vertical stacking function, and differential extraction function can be executed on multiple tables arranged in any position, such as vertically or diagonally. (Table filtering function)

[0127] The table filtering feature allows you to filter by a specific column. You can also enter conditions for multiple columns. (Column Addition Function)

[0128] The Add Column feature allows you to add columns to a spreadsheet, including formulas. (Column deletion function)

[0129] The column deletion function is a function for deleting unnecessary columns.

[0130] The various operations performed on columns in the table data described above can also be performed on rows, and conversely, operations performed on rows can be replaced with operations performed on columns. [Embodiment 4: Web scraping] (Web Import Advanced Settings)

[0131] After the element data to be acquired from the Web browser WB is specified, acquisition range specification unit 22 displays Web data acquisition setting screen 540. The Web data acquisition setting screen allows the user to select whether or not to acquire element data at the same level as the element data specified by the user. When acquiring element data at the same level, the screen also accepts a setting as to whether the element data at the same level should be arranged in the same column or row. Based on these settings, the data acquired from the Web browser WB is stored in memory unit 15 as table data.

[0132] Figure 32 shows a screen displayed when the user selects to acquire element data at the same hierarchical level as the element data specified by the user in Figure 33. When the user selects to acquire element data at the same hierarchical level, the acquisition range designation unit 22 creates a search expression that can extract element data at the same hierarchical level based on the path of the element data specified by the user. In Figure 32, the element data that corresponds to the path extracted by the search expression is highlighted.

[0133] 36 is a flowchart executed by the acquisition range setting unit when the acquisition range is set by the acquisition range setting unit in step S904 of FIG. 8 when the Web browser WB is specified as the transcription source application. Note that this flowchart shows a flowchart executed by the acquisition range setting unit from a state in which the page from which data is to be acquired is displayed in the Web browser WB. The launching of the Web browser WB and the display of the page from which data is to be acquired may be separately set by the operation record setting unit 48.

[0134] In step S5701, the acquisition range setting unit accepts designation of element data to be displayed on the Web browser WB. In step S5702, the acquisition range setting unit acquires the path of the element data designated by the user on the Web browser WB via an extension function of the Web browser WB. In step S5703, the acquisition range setting unit highlights the element data corresponding to the acquired path based on the path. Figure 31 shows the state in which the element data designated by the user is highlighted in step S5703. The user can identify the element data they designated.

[0135] In step S5704, the acquisition range setting unit determines whether to acquire element data at the same level as the element data specified by the user in step S5701. If element data at the same level as the element data specified by the user is to be acquired, in step S5705, a search expression using Xpath for acquiring element data at the same level as the element data specified by the user in step S5701 is generated based on the path acquired in step S5702. In this embodiment, the search expression generated in step S5705 is based on the path acquired in step S5702, but the search expression may be generated based on multiple paths. For example, the search expression may be generated based on the paths of multiple element data, accepting designation of element data that the user recognizes as element data at the same level as the element data specified in step S5701. In this case, the accuracy of generating a search expression that can extract element data at the same level as the user's intention is improved. FIG. 32 shows the element data extracted by the search expression generated in step S5705 highlighted. If it is determined in step S5704 that element data at the same layer as the element data specified by the user is not to be acquired, the process proceeds to step S5708, where setting of the data acquisition range from the Web browser WB is completed.

[0136] After step S5705 is completed, the acquisition range setting unit accepts, in step S5706, a selection of the arrangement direction of element data at the same level. The arrangement direction of element data selected in step S5706 determines whether element data at the same level are arranged in the same column or row when data acquired from the Web browser WB is stored as table data in memory unit 15. FIG. 34 shows the format of table data stored in memory unit 15 when element data at the same level are set to be arranged in the same column in step S5706. After the selection in step S5706 is completed, the acquisition range setting unit accepts, in step S5707, the addition of data to be acquired. The data specified in step S5707 is arranged according to the arrangement selected in step S5706 when stored as table data in memory unit 15. For example, if element data at the same level are selected to be arranged in the same column in step S5706, the data specified in step S5707 is acquired as data to be arranged in a new column. 35 shows the form of table data held in memory unit 15 when element data at the same layer is set to be arranged in the same column in step S5706 and two types of data are added in step S5707. Depending on the settings in each step, the form of table data shown in FIGS. 34 and 35 may be displayed as a preview of the data to be acquired.

[0137] After the data addition in step S5707 is completed, the process proceeds to step S5708, where the setting of the data acquisition range from the Web browser WB is completed.

[0138] As described above, according to the present invention in each embodiment, the user is guided to acquire data groups that can be acquired as table data as table data, thereby facilitating the creation of automation programs. In particular, if the software to be automated is a spreadsheet software such as Excel, automatic recognition and table range setting can be performed. Furthermore, if the software to be automated is a web browser, the repetition direction during web scraping can be easily set. Furthermore, if the software to be automated is another general-purpose program, selecting "data acquisition" as the operation content displays a screen that allows variable names to be set, allowing table data to be named and referenced, providing the advantage of facilitating the creation of automation programs.

[0139] The operation record function will be described in detail below. (Recording of menu operations of software to be automated)

[0140] An example of recording an operation related to a menu of software to be automated will now be described with reference to FIG. 40. First, the procedure of operating an expense reimbursement app to obtain an "application ID" is recorded on an operation record screen 740. Here, on the operation record screen 740 in FIG. 40, the "Edit" menu 188 in the menu field of the expense reimbursement app shown on the left is clicked from a semi-transparent window TW. At the timing of the click, all levels of the "Edit" menu 188 are temporarily opened, and the element identification unit 40 acquires "Copy," a sub-item of the "Edit" menu 188. In the example of FIG. 40, the element identification unit 40 selects an object in the "Edit" menu 188 that has options, and therefore a "Menu" category 742 is displayed in the input candidate list 741. Furthermore, below the "Menu" category 742, selection items in the same hierarchical relationship as in the expense reimbursement app—here, "Copy (C)" 744 or "Paste (P)" 745 of "Edit"—are listed as operation content options 743.

[0141] In the above example, the operation recording function is executed while one piece of automation target software is running. However, the present invention is not limited to this configuration. The operation recording function can also be executed with multiple programs as automation target software. As an example, Figures 41 and 42 show the execution of two applications GSa and GSb. As shown in these figures, even when multiple applications GSa and GSb are running, the entire display screen can be covered with a translucent window 750. If you click on the position of an object in the automation target software from within this translucent window 750, the click is received by the translucent window 750, not the automation target software. Due to this specification, an operation to select an object, such as a click, is not immediately reflected in the automation target software.

[0142] Some menu items in the software to be automated that are selected as the target for operation recording disappear when clicked, making it impossible to obtain the correct data. In contrast, the automation program creation program of this embodiment does not receive clicks in the software to be automated, but rather receives clicks from the automation program creation program itself, making it possible to appropriately control the timing of data acquisition. This is particularly useful when setting operation details for objects whose behavior changes when clicked, or whose information becomes unavailable (for example, the screen disappears when the "close" button is pressed).

[0143] Here are some examples of operations that can be set in an operation record. Commonly configurable operations include clicking with specified coordinates, normal clicking without specifying coordinates, focusing, key input, and table input. Operations for text boxes include text input, clipboard pasting, and keyboard input. Operations for compo boxes include selecting an element. Operations for check boxes include checking on and off. For menus, operations include: Get hierarchy → click, Get hierarchy → check on (checkable), Get hierarchy → check off (checkable). For ribbons (often found in office software), operations include XPath support and Get hierarchy → click. Operations for scrolling include scrolling down and scrolling up. (Low level record mode)

[0144] The operation record setting unit 48 can execute low-level recording. In this embodiment, low-level recording is a record that records the movement of the mouse operated by the user, the up and down of the mouse button, and the keys pressed on the keyboard. In this mode, the operations performed by the user are recorded as they are, so there is no need to display an object selection acceptance screen or operation content options that would hinder the reflection of the operations in the automation target software GS. Note that mouse button down refers to the state in which the user presses the button with their finger, and mouse button up refers to the state in which the finger is released and the mouse button returns to its original position. (Table UI import function)

[0145] It is also possible to provide a table UI import function that imports information as two-dimensional table data from the table UI of the automation target software GS, which is the transcription source application. This table UI import function will be explained based on Figure 43. Here, the table UI of the expense settlement app, which is the import source, or transcription source application, is assumed to be the UI shown in Figure 43. Here, the user is prompted to make the following settings. 1. Click the top left cell of the table in the table UI of the expense reimbursement app shown in Figure 43. 2. In the table UI, set which column and which value to retrieve. Here, we retrieve the column values ​​of the column items "Occurrence Date," "Means," and "Amount." 3. Specify the end condition. Here, you can choose either "Until an empty row is reached" or "Maximum number of rows

[10] ".

[0146] As a result of making these settings, the following operations are generated as internal processing of the automation program creation program. Note that the acquisition method differs depending on the control of the transcription source application, so the following is just an example. Click in the top left corner of the table. Repeat the following until you reach an empty line or have copied 10 lines. Add new row. Copy the cells and save them in the "Occurrence Date" column of a new row. Tab key x2 Copy the cells and save them in the "Means" column of a new row. Tab key x1 Copy the cell and save it in "Amount" in a new row. Shift+Tab x3 Down arrow key (Other variations)

[0147] In this embodiment, the object selection acceptance screen is a translucent window. However, the object selection acceptance screen is not limited to a translucent window. It is sufficient for the object selection acceptance screen to notify the user that the object selection acceptance screen is active. For example, a message such as "Object selection acceptance in progress" may be displayed on the display screen, an icon indicating that the object selection acceptance screen is active may be superimposed, the shape of the mouse cursor may be changed, the screen may flash, a sound may be emitted, or, in the case of a tablet with a vibrator function, the vibrator may be continuously or intermittently activated. Any other suitable means may be used to visually, audibly, or tactilely recognize that the object selection acceptance function is active. It is sufficient for the user to recognize that the object selection acceptance function is active. Furthermore, the object selection acceptance screen is displayed on the front of the display screen to make it easier for the user to see. However, as described above, it may be displayed on only a portion of the display screen, as long as the user is aware that the object selection acceptance function is active. For example, a widget for the object selection acceptance function may be placed in a corner of the display screen. It is sufficient that the visibility of the objects in the software to be automated is maintained to the extent that it does not interfere with the display of the input form or operations of the software to be automated that the user wants to record using the operation recording function. In other words, the entire software to be automated does not need to be visible at all times, and the condition for the object selection reception screen is that the display necessary for operation, such as input forms, is secured.

[0148] The activity list screen 220 in FIG. 15 shows a state in which detailed settings for importing various data and settings for the transcription operation have been completed. The section "Record operations and perform input / output" in this diagram is the general activity heading section 222, which displays the general activity corresponding to the flow. A general activity detail display area 225 is provided below the general activity heading section 222. The general activity detail display area 225 displays "Launch app," "Repeat table," and "Exit app," listing the specific operations, i.e., activities, that make up the general activity "Record operations and perform input / output." The general activity unit after various settings have been completed corresponds to the general procedure created in FIG. 9. Here, within the general activity heading section 222, "Record operations and perform input / output" is displayed as a general name 223, which is manually entered by the user in the general name input section 45 in FIG. 3, while the general label 224 to the right of it, "Enter expense list," is automatically generated by the general label generation section 44 in FIG. 3.

[0149] As described above, the automation program creation program according to this embodiment allows the operations of the created automation program to be displayed together in outline activity units. Furthermore, when displaying the outline activities on the activity list screen 220, appropriate variables are selected from those handled by the operations and are given summaries in natural language, allowing the user to grasp the summaries. In other words, the user does not need to create the outline of the created automation program from scratch, but can create it to a certain extent automatically, thereby reducing the labor required for creating the automation program.

[0150] In conventional RPA, as the construction of the scenario that defines the automation program progresses, the length of the scenario increases, resulting in a lack of visibility. To address this issue, attempts have been made to improve visibility by dividing the operations that make up the automation program into blocks or groups.

[0151] However, although conventional RPA has implemented functions such as grouping and blocking flows to reduce the burden on users when operating, editing, and checking scenarios, the burden on users is still heavy.

[0152] In contrast, in the automation program creation program of this embodiment, when parts of the created automation program are displayed in groups, appropriate variables are selected from those handled in the operation and a natural language summary is added, allowing the user to grasp the summary.

[0153] First, the created automation program is input into the general label generation unit 44, which automatically analyzes the reading and writing of variables inside and outside the general activity. Then, a list of variables that are used inside the general activity and also used outside is obtained. Furthermore, it is determined whether each variable is used for input or whether it stores the results of data acquisition, and this is reflected in the display. This display is displayed in natural language on the general activity display unit 43.

[0154] Also, the overview activity can be expanded to display details or made compact by folding, improving user visibility, as shown in Figures 44 and 45. Furthermore, the internal and external data flows are visualized by the overview activity display unit 43, making it possible to clearly show the operations being performed inside the automation program.

[0155] Furthermore, when an operation within a general activity is edited, added, deleted, etc., the general label generation unit 44 automatically reanalyzes the internal data flow and updates the display content in the general activity display unit 45.

[0156] On the other hand, apart from the data flow analysis display by the general label generating section 44 and the general activity display section 45, the user can also give any name via the general name input section 45.

[0157] An example in which two general activities have been created and are displayed on the activity list screen is shown in Fig. 44. In this example, two general activities, "Low-cost Excel reading" and "Expense settlement," are displayed in the general activity display field 831. In addition, a general activity detail display area 225 is displayed below each general activity headline section 222. (Expand button 839)

[0158] To the left of each general activity headline portion 222 displayed in the general activity display field 831 is an expand button 839, which constitutes one form of a display switching section. In the example of FIG. 44 , the expand button 839 is displayed as a ▽ or △, and pressing the expand button 839 allows the general activity detail display area 225 to be switched between expanded and collapsed display. For example, FIG. 44 shows the general activity detail display area 225 in an expanded state, with the expand button 839 displayed as a ▽. Pressing the expand button 839 in this state results in the display shown in FIG. 45 , where the expand button 839 changes to a △, the general activity detail display areas 225 a and 225 b of each general activity are collapsed, and only the general activity headline portion 222 is displayed. In this way, by changing the icon of the expand button 839 to a △ or ▽, it is possible to distinguish between a detailed expanded display and a simple collapsed display. Furthermore, even in the collapsed display, variables used outside the automation program are displayed, so the general display is not cluttered and the user can easily understand the content of the activity. In addition, the relationship between input and acquisition is also visualized, which has the advantage that it is easy to understand what the operation is for, even when the display is collapsed.

[0159] 44 and 45, a summary name 223 is provided on the left side of the summary activity heading section 222, and a summary label 224 is provided on the right side. The summary name 223 is a text box in which the user can freely set a name, and in this example is shown as a blue area (shown by cross-hatching in FIG. 44). The summary label 224 is a character area that is automatically generated by the summary label generation section 44, and is displayed as an orange area (shown by diagonal lines in FIG. 44). Items whose summary labels 224 are displayed in different colors represent variables. In this way, the display color differs depending on the type of variable. Furthermore, by using a different display color for each variable, it is possible to obtain the advantage that it is easier for the user to visually distinguish between multiple variables.

[0160] In the example of Fig. 44, the name of the general activity "Read Expenses from Excel" input by the user is displayed in the general name 223 on the left side of the general activity heading section 222a in the upper row, and "Get [Expense List]" is displayed as the outline of the automatically generated general activity in the general label 224 to the right of that. Similarly, the name of the general activity "Expense Settlement" is displayed in the general name 223 on the right side of the general activity heading section 222b in the lower row, and "Enter [Expense List]" is displayed as the outline in the general label 224 to the right of that. (Outline label 224)

[0161] The general label 224 is automatically generated by the general label generating unit 44 according to a predetermined rule. The general label 224 is automatically generated according to a predetermined rule, for example, if the general label 224 is automatically generated according to the following rule: · Input anything that is used internally as an argument; ·Get what is assigned internally; -If an argument does not refer to an external value by reassignment, it is not considered as input; etc.

[0162] Here, a variable declared outside a general activity is an activity outside the general activity to which the general label 224 is assigned, which defines the value to be treated as the variable. In Fig. 44, for the general activity to which the general label "Input expense list" is assigned, the activity "Get table from Excel" is an activity outside the general activity. In the "Get table from Excel" activity, the data to be obtained from Excel is obtained as an "expense list," so this is equivalent to the data to be obtained from Excel being defined as a value to be treated as an "expense list."

[0163] Among the general activities in Figure 44, "Expense settlement" has an internal activity that uses "Expense list," "Item name," "Amount," and "Use" as variable names. Of these, the variable declared outside the "Expense settlement" general activity is "Expense list," which is also declared in "Get table from Excel," an activity included in the general activity "Low cost Excel read." Therefore, "Expense list" is used for general label 224.

[0164] In addition, the text following the variable name is determined, for example, by the following example program code (outline: treated as an argument inside the activity → input, assigned internally → obtained). var a = 1; OutlineActivity(() => { a = 2; })

[0165] The above code defines the operation "get a." Another example of code is as follows: var a = 1; OutlineActivity(() => { Console.WriteLine(a); })

[0166] The above code defines the operation of "entering a." Other code examples include the following: var a = 1; OutlineActivity(() => { a = 2; a = a+1; })

[0167] The above code defines the operation "get a." Further code examples include the following: var a = 1; var b = 2; OutlineActivity(() => { b = 3; Console.WriteLine(a); })

[0168] The above code defines the operation of "input a and get b." Further code examples include the following: var a = 1; OutlineActivity(() => { a = 3; Console.WriteLine(a); })

[0169] The above code defines the operation "get a." Further code examples include the following: var a = 1; OutlineActivity(() => { Console.WriteLine(a); a = 3; })

[0170] The above code defines the operation "input a and get a".

[0171] Furthermore, the user can change which activities are grouped into which general activities by rearranging the activities on the screen of Fig. 44. If this operation changes the activities grouped into each general activity, the general label 224 is also updated. (Automatic update of summary activities)

[0172] As an example of the general activity automatic update function for automatically updating general activities, the general activity display field 851 in FIG. 46 shows a state in which the activity "Set Variable" has been moved from the general activity "[Low Cost Excel Import] [Get Low Cost List]" displayed in the general activity headline section 222a on the screen in FIG. 44 to outside the general activity. Here, because "filename" is used outside the general activity, a new activity "Set Variable" is specified before the general activity "Expense Excel Import," and "Set [Low Cost List.xlsx] to [filename]" is specified as its summary. Furthermore, because "filename" is also used as an input inside the general activity "Expense Excel Import," the general activity display in the general activity headline section 222a is automatically updated from "Get [Low Cost List]" in FIG. 44 to "Enter [filename] and get [Low Cost List]" in FIG. 46. In this way, because "filename" is used as an input both inside and outside the general activity, the general activity display is automatically updated accordingly. In other words, even if the user modifies the automation program, the display of the general activity is automatically updated, saving the user the trouble of manually rewriting the general outline each time debugging, etc. Also, Figure 47 shows the general activity display field 851 in Figure 46 with the general activity detail display areas 225a and 225b closed. In this way, the activity "Set variable" that has been moved to the outside is also displayed, saving the user the trouble of having to rewrite the general outline each time the automation program is updated from the state in Figure 45.

[0173] In this way, the user can easily grasp the outline of the created automation program. That is, it is possible to check multiple outline activities included in one automation program together on one screen, etc., in an outline display mode that omits detailed display and compacts it. Also, even in the outline display mode, by describing some content in the outline activity heading section 222, the outline can be grasped while still displaying it compactly.

[0174] Furthermore, the user can check the details of the sub-activities that make up the general activity and easily access the information they need by selecting a desired general activity and switching to a detailed display mode that displays the general activity detailed display area 225. In this way, by being able to switch between a compact display and a detailed display as needed, it is possible to achieve both ease of viewing and the ability to grasp the details. (Editor screen 870)

[0175] Furthermore, the code of the created automation program can be checked. Furthermore, a user skilled in programming can directly edit the automatically generated code. To edit the code of such an automation program, the editor unit 49 of FIG. 3 is used. Specifically, an editor screen 870 shown in FIG. 48 is used as one embodiment of the editor unit 49. The editor screen 870 has a source code display field 871 on the right side and an activity display field 872 on the left side. The activity display field 872 displays, for each outline activity that constitutes the automation program, each of the sub-activities that make up the outline activity, similar to the outline activity display field 831 of FIG. 44 described above. The source code display field 871 also constitutes one embodiment of the source code display unit 46. The source code display field 871 displays the source code of the automation program that was automatically generated to realize a series of operations defined in the created automation program. In other words, whereas a display format such as that shown in source code display field 871 is normally difficult for general users to understand, the automation program creation program of this embodiment can realize an environment that is easier to understand and that makes it easier to utilize the automation program by displaying an outline of the automation program in natural language, as shown in activity display field 872. Furthermore, whereas in the past, in order to realize such a natural language notation, the user had to separately and manually describe the contents of each activity, the automation program creation program of this embodiment can express the outline of each activity in natural language somewhat automatically by using variables defined by the user during programming, thereby realizing an even easier-to-use environment.

[0176] The user can directly edit the source code from the source code display field 871. Furthermore, as shown in the editor screen 880 of Fig. 49, when the user changes the source code from the source code display field 871, the display content of the outline activity in the activity display field 872 is updated accordingly. In this way, even a user skilled in programming can express the created or updated automation program as an outline activity in an easy-to-understand manner in natural language, which has the advantage of allowing efficient use of past assets when checking the outline of the automation program at a later date.

[0177] The code conversion unit 47 in FIG. 3 converts the content written in source code displayed in the source code display field 871 into a summary activity displayed in the activity display field 872. The code conversion unit 47 analyzes the content written in source code and expresses it as natural language. For example, this process involves extracting nodes from the source code using a tool and assigning IDs to the nodes. When generating the UI, the IDs are copied to each element. Furthermore, a correspondence table between ID → View and ID → SyntaxNode is prepared, and the two are associated via IDs. This allows the source code to be converted into the corresponding natural language. [Industrial Applicability]

[0178] The automation program creation device, automation program creation method, automation program creation program, and computer-readable recording medium and device on which it is recorded of the present invention can be suitably used in RPA software that supports the creation of automation programs that automate operations on spreadsheet software such as Excel, web browsers, and other software to be automated. [Explanation of symbols]

[0179] 11...Display section 12...input section; 12a...keyboard; 12b...mouse; 12c...touchpad 13...Bus 14…Communications Department 15...Memory section 16...Storage section 20...Processor section 21...Outline Procedure Creation Department 22... Acquisition range specification section 23...Transfer designation section; 23a...Transfer destination designation section; 23b...Transfer unit designation section; 23c...Table operation section; 23d...Transfer data preview section 24...Transfer unit preview section 25...Automation program generation unit 26...Automation program execution unit 31...Repeat setting section 38...Object selection reception section 39...Operation selection section 40...Element identification section 41...Operation execution unit 48...Operation record setting section 110...Outline procedure creation screen 111...Outline procedure icon 112...List of operation candidates 113..."Add" button 114...Operation selection list 115..."Repeat specified range" button 117...Next button 118... "Discard and return" button 119...Repeat setting icon 121...Activity display column 123... "Extract with automatic table recognition" button 124... "Specify the range of a spreadsheet" 125..."Extract using record function" button 130...Spreadsheet software acquisition range specification screen 132...Spreadsheet import details specification field 138...Next button 150...General posting detailed setting screen 152...Transfer details specification field 153...Transfer unit specification field; 153b..."Data to input" selection field; 153c..."Repeat unit" selection field; 153d...Grouping detailed setting field 154..."Input unit preview" column 155..."Enter the results of other operations" field 156...Operation decision field 157..."Preparation Action" setting field 159..."Post-event action" setting field 160...Operation record screen 161...Output list window 170...Operation record screen 171...Operation content options; 171a..."Mouse operation"; 171b..."Input" 172...input candidate list; 172a, 172c..."text input"; 172b, 172d..."paste input"; 172e..."table input (column name): applicant"; 172f..."table input (column name) amount" 180...Operation record screen after dynamic changes are reflected 186…“Application ID” field 188...Edit menu 190...Operation record screen 191...Operation options; 191a..."Get value" 202..."Specify individual cells to export" field 208...Next button 220...Activity list screen 221...Summary activity display column 222...Outline activity headline section 223…Short name 224...Outline label 225...Outline activity details display area 231..."File Selection" button 232..."Select folder" button 234..."Auto Extract" button 235...Preview button 236..."Manual Extraction" button 237..."Settings" button 262..."Confirm" button 280...Operation record screen 300...Operation record screen 301...Table Data Transfer Advanced Settings Window 302...Table input position specification field 303...Table selection column 304…Table movement direction specification field 305...Line insertion specification field 306...Line deletion specification field 307... "Try moving the table" button 310...Operation record screen 320...Operation record screen 321...Column name options 358..."OK" button 380...Table navigation screen 390…Table operation screen 513..."Open Web Page" setting field; 513b..."Settings" button 514... "Selection of extraction target" field: 514b... "Settings" button 515... "Repeat Settings" field; 515b... "Settings" button 516..."Output Settings" field; 516b..."Settings" button 540... Output detail setting screen 740...Operation record screen 741...Input candidate list 742... "Menu" category 743...Operation options 744... "Copy (C)" 745... "Paste (P)" 750...Translucent window 831...Summary activity display column 839…Expand button 851...Summary activity display column 870...Editor screen 871...Source code display field 872...Activity display column 880...Editor screen SS: Spreadsheet software WB: Web browser GS...software to be automated; GS1...expense settlement application "expense settlement" form RS…RPA software PC...computer TW...Translucent window L1: Area corresponding to the table UI GSa, GSb… Applications

Claims

1. An automation program creation device that records and executes a series of operations on software to be automated, a variable definition section for defining variables to be referenced in any of one or more sub-activities constituting the automation program; an automation program generation unit for generating an automation program consisting of one or more activities including one or more sub-activities based on the variables defined in the variable definition unit; a summary activity display unit for displaying the one or more activities constituting the automation program created by the automation program generation unit and sub-activities constituting the activities; a general label generation unit that automatically generates, in natural language, a general label that indicates an outline of the one or more activities or sub-activities displayed on the general activity display unit by acquiring variable information related to variables referenced in any of the sub-activities; An automated program creation device comprising:

2. 2. The automated program creation device according to claim 1, The outline activity display unit groups a plurality of sub-activities that make up the automation program and displays them as an activity.

3. 3. The automated program creation device according to claim 1, The outline label generating unit is capable of displaying, on the outline activity display unit, variable information relating to variables referenced in a sub-activity not included in an activity or variables referenced in an activity.

4. The automated program creation device according to any one of claims 1 to 3, The outline activity display unit a summary activity headline section showing an outline of the one or more activities and sub-activities constituting the activities; a summary activity detail display area in which a summary is displayed for each activity displayed in the summary activity headline section; An automated program creation device comprising:

5. 5. The automated program creation device according to claim 4, The outline activity display section, for each outline activity headline section, a detailed display mode for displaying the outline activity detailed display area; a summary display mode in which the summary activity detail display area is not displayed; An automated program creation device having a display switching unit that can switch between the above.

6. 6. The automated program creation device according to claim 4 or 5, The outline activity display section displays variables included in outline labels displayed in the outline activity headline section or the outline activity detail display area by assigning different colors to each variable.

7. The automated program creation device according to any one of claims 1 to 6, The general label generating unit is configured to generate the general label from the natural language in accordance with a predetermined rule.

8. 8. The automated programming device according to claim 7, The predetermined rule is: Enter the ones used internally as arguments; Gets what is assigned internally; If an argument does not refer to an external value by reassignment, it is not considered as an input; An automated program creation device comprising at least one of the above.

9. The automated program creation device according to any one of claims 4 to 6, further comprising: The automation program creation device further comprises a general name input section for each of the general activity headline sections, which allows input of a general name indicating the content of the activity to be displayed in the general activity headline section.

10. The automated program creation device according to any one of claims 1 to 9, the automation program is a program that automates an operation of transcribing data acquired from a first application that is a transcription source into a second application that is a transcription destination, The automated programming device further comprises: obtaining the data from the first application; A transcription operation of the data to the second application a summary procedure creation unit for selecting; an acquisition range designation unit for selecting a range of data to be acquired from among data included in any data file of the first application selected by the outline procedure creation unit; a transcription unit designation unit for setting a data transcription unit for transcribing the data within the range set by the acquisition range designation unit as source data in a table format to the second application; a transcription unit preview unit for displaying transcription data to be transcribed into the second application program selected by the outline procedure creation unit based on the data transcription unit set by the transcription unit designation unit; an automation program generation unit that generates an automation program for transcribing tabular source data of the first application selected by the outline procedure creation unit within a range set by the acquisition range designation unit into the second application selected by the outline procedure creation unit based on a data transcription unit set by the transcription unit designation unit; An automated programming device comprising:

11. The automated program creation device according to any one of claims 1 to 10, further comprising: an automation program creation device comprising a source code display unit for displaying the source code of the automation program created by the automation program creation unit;

12. The automated program creation device according to claim 11, further comprising: an automation program creation device comprising a code conversion unit for converting content written in source code displayed in the source code display unit into the one or more activities and sub-activities constituting the activities displayed in the outline activity display unit.

13. An automation program creation method for creating an automation program that records and executes a series of operations on software to be automated, a step of defining variables to be referenced in any of one or more sub-activities constituting the automation program in a variable definition section; a step of generating an automation program composed of one or more activities including one or more sub-activities based on the variables defined in the variable definition unit, in an automation program generation unit; a step of automatically generating, in a natural language manner, a summary label generating unit that generates a summary label representing an outline of one or more activities and sub-activities that constitute the automation program created by the automation program generating unit, by acquiring variable information related to variables referenced by any of the sub-activities; a step of displaying the general label generated by the general label generation unit on a general activity display unit; An automation program creation method including:

14. An automation program creation program that creates an automation program that records and executes a series of operations on software to be automated, a function of defining, in a variable definition section, variables to be referenced in any of one or more sub-activities constituting the automation program; a function of generating an automation program consisting of one or more activities including one or more sub-activities based on the variables defined in the variable definition unit, in an automation program generation unit; a function of automatically generating, in a natural language manner, a summary label that represents an outline of one or more activities and sub-activities that constitute the automation program created by the automation program generation unit, by acquiring variable information related to variables referenced in any of the sub-activities; a function of displaying the general label generated by the general label generation unit in a general activity display unit; An automation program that allows a computer to achieve the above.

15. A computer-readable recording medium or storage device storing the program according to claim 14.

Images