Information processing apparatus, information processing method, and information processing program

Abstract

To provide an information processor, an information processing method, and an information processing program capable of visually and intuitively understanding a cause of error occurrence when an error occurs during automatic operation.SOLUTION: An information processing apparatus according to the present embodiment includes a screen shot unit that records, in a storage area, screen shots before and after an error occurred during an automatic operation of an application, and a screen display unit that displays, on a confirmation screen, the screen shots before and after the error recorded in the storage area.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Recently, there has been an increasing demand for automated application operations known as RPA (Robotic Process Automation). There are automation tools for RPA that automate operations.

[0003] In RPA, identifying the cause of an error when an automated operation fails is an important issue. Conventionally, only a screenshot of the screen immediately after the error occurred was recorded. This made it difficult to identify the cause of the error if the original screen was not captured or if the screen itself disappeared.

[0004] Additionally, a similar effect can be achieved by taking screenshots at every RPA processing step, but this was not a practical method because the binary data of the images would increase disk space.

[0005] Furthermore, even if it were possible to obtain screenshots before and after the error occurred, there was no viewer that allowed intuitive comparison.

[0006] Conventionally, an example of an RPA system is the system disclosed in Patent Document 1. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2020-115246 Summary of the Invention [Problem to be solved by the invention]

[0008] However, Patent Document 1 does not disclose any method for identifying the cause of an error when an error occurs during automatic operation.

[0009] The present invention has been made in view of the above, and when an error occurs during automatic operation, An object of the present invention is to provide an information processing device, an information processing method, and an information processing program that enable the cause of an error to be visually and intuitively understood. [Means for solving the problem]

[0010] In order to solve the above-mentioned problems and achieve the object, the present invention provides an information processing device that automatically operates an application in accordance with a definition that specifies a series of operations, and is characterized by comprising: a screenshot means that records screenshots before and after an error that occurs during the automatic operation of the application in a memory area; and a screen display means that displays the screenshots before and after the error recorded in the memory area on a confirmation screen.

[0011] Furthermore, according to one aspect of the present invention, the screenshot means may take a screenshot each time a screen item is automatically operated, and for screenshots taken before an error, only the number of screenshots recorded immediately before the error that is set may be retained in the memory area.

[0012] According to another aspect of the present invention, the screenshot means may store the screenshot before the error in the storage area in a FI / FO manner.

[0013] According to another aspect of the present invention, the confirmation screen may display a list of thumbnails of screen shots before and after the error occurred.

[0014] According to another aspect of the present invention, the confirmation screen may display a comparison image in which the difference between the screen shots before and after the error is highlighted.

[0015] Furthermore, according to one aspect of the present invention, when three or more comparative images of screen shots are displayed on the confirmation screen, differences between the different images may be highlighted differently.

[0016] According to another aspect of the present invention, the highlighting of the difference may include highlighting using a highlight color.

[0017] Furthermore, according to one aspect of the present invention, when the confirmation screen displays comparative images of three or more screen shots, the differences between the different images may be highlighted in different colors to visualize the gradual differences in the screen changes.

[0018] According to another aspect of the present invention, the automatic operation may be an automatic operation based on RPA (Robotic Process Automation).

[0019] Furthermore, according to one aspect of the present invention, the definition is composed of a first layer that defines the flow, a second layer that defines the robot execution, and a third layer that is the internal implementation of the screen operation processing, and the definition of the screenshot means may be implemented in the third layer.

[0020] According to another aspect of the present invention, the application may include an accounting application.

[0021] In addition, in order to solve the above-mentioned problems and achieve the object, the present invention may be an information processing method executed by an information processing device that automatically operates an application in accordance with a definition that specifies a series of operations, and may include a screenshot process that records screenshots before and after an error that occurs during the automatic operation of the application in a memory area, and a screen display process that displays the screenshots before and after the error recorded in the memory area on a confirmation screen.

[0022] In addition, in order to solve the above-mentioned problems and achieve the object, the present invention is characterized in that it is an information processing program to be executed on an information processing device that automatically operates an application in accordance with a definition that specifies a series of operations, and is an information processing program for executing a screenshot process that records screenshots before and after an error that occurs during the automatic operation of the application in a memory area, and a screen display process that displays the screenshots before and after the error recorded in the memory area on a confirmation screen. [Effects of the Invention]

[0023] According to the present invention, when an error occurs during automatic operation, it is possible to visually and intuitively understand the cause of the error. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a diagram for explaining how to take a screen shot when a robot is automatically executing an application to be operated. [Figure 2] FIG. 2 is a diagram showing an example of a screen shot. [Figure 3] FIG. 3 is a diagram showing an example of a screen shot. [Figure 4] FIG. 4 is a diagram for explaining a case where a screen shot is taken every time immediately before an operation on a screen item. [Figure 5] FIG. 5 is a block diagram showing an example of the configuration of an information processing device according to this embodiment. [Figure 6] FIG. 6 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. [Figure 7] FIG. 7 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. [Figure 8] FIG. 8 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. [Figure 9] FIG. 9 is a diagram for explaining a specific example of the process of the control unit of the information processing device according to the present embodiment. [Figure 10] FIG. 10 is a diagram for explaining a specific example of the process of the control unit of the information processing device according to the present embodiment. [Figure 11] FIG. 11 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. [Figure 12] FIG. 12 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. [Figure 13] FIG. 13 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. [Figure 14] FIG. 14 is a diagram for explaining a specific example of processing by the control unit of the information processing device according to the present embodiment. [Figure 15] FIG. 15 is a diagram for explaining a specific example of the process of the control unit of the information processing device according to the present embodiment. [Figure 16] FIG. 16 is a diagram for explaining a specific example of the process of the control unit of the information processing device according to the present embodiment. [Figure 17] FIG. 17 is a diagram for explaining a specific example of the process of the control unit of the information processing device according to the present embodiment. [Figure 18] FIG. 18 is a diagram for explaining a specific example of the process of the control unit of the information processing device according to the present embodiment. [Figure 19] FIG. 19 is a diagram for explaining a specific example of the process of the control unit of the information processing device according to the present embodiment. [Figure 20] FIG. 20 is a diagram for explaining a specific example of the process of the control unit of the information processing device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0025] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an information processing device, an information processing method, and an information processing program according to embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to the embodiments.

[0026] [1. Overview] Recently, there has been an increasing demand for automated application operations known as RPA (Robotic Process Automation).

[0027] RPA has an automation tool for automating operations. Specifically, it allows users to register (define) multiple operations and then execute the registered operations, allowing them to easily execute standardized operations from the second time onwards.

[0028] How to respond when an automated operation fails is an important issue, and screenshot logs are useful when investigating the cause of the failure. Screenshot logs (1) allow for a more intuitive understanding of the situation at the time of failure than text logs of processing content, and (2) contain more information, making them useful for identifying the cause. Specifically, this is because they can record error messages displayed on the screen and input values ​​for items that have not yet been operated.

[0029] Figure 1 is a diagram to explain how to take a screenshot when a robot is automatically executing an application to be operated. In Figure 1, it is assumed that operations 1 and 2 performed by the robot on the application to be operated are successful, but operation 3 fails. An example of a failure is when the value entered in a text box in "operation 2" is invalid, causing an error message to be displayed, making it impossible to operate the item to be entered in "operation 3." A screenshot is taken immediately after the error occurs, and the cause of the failure can be investigated using the screenshot.

[0030] Figures 2 and 3 are diagrams showing examples of screenshots. Most systems take a screenshot of the screen immediately after an error occurs and notify the user. This system is technically easy to implement.

[0031] Figure 2 shows Screenshot Example 1. Screenshot Example 1 shows a case where an error occurred because a dialog box that the robot did not expect appeared, preventing the process from continuing. The dialog box "Error: Some fields have not been entered" is displayed, and this screenshot is taken.

[0032] Figure 3 shows Screenshot Example 2. Screenshot Example 2 is a case where the screen transition expected by the robot did not occur, the item to be operated on could not be found, and an error occurred; the message "500 Internal Server Error Please contact your system administrator" is displayed, and this screen shot is taken.

[0033] Examples of logic by which the robot determines that an operation has failed include (1) when the screen item to be operated does not exist on the screen, (2) when a dialog box (such as an error) that the robot did not expect appears, and (3) when a screen other than the one to be operated is displayed.

[0034] Next, we will explain the issues with screenshots. First, as a business issue (1), there are cases where the cause of an error cannot be determined from just the screenshot taken immediately after the error occurs. Examples of cases where the items entered using RPA cannot be confirmed include when the input screen is partially hidden by an error dialog, the user is redirected to an error page, or the items entered in the application are cleared.

[0035] As a business issue (2), there are cases where it is not possible to take a screenshot immediately after an error occurs. Irregular cases where the screen disappears include when the RPA execution timeout occurs and the system is forced to close, when a human accidentally closes the screen during automatic operation, or when the computer on which the robot is running is forced to close.

[0036] For this reason, it is desirable to have a screenshot of the screen just before an error occurs. However, there are implementation issues (1) and (2) below when taking a screenshot of the screen just before an error occurs.

[0037] (Implementation Issues (1)) Since the process of taking a screenshot for each operation must be set in the definition (L2), the definition creation becomes cumbersome and the definition itself becomes complicated. L (layer) will be explained later.

[0038] (Implementation Issues (2)) It is necessary to take a screenshot every time just before operating a screen item, but this would result in a large screenshot log (and of course, video is not practical either). Specifically, because images are binary data, a large amount of disk space is required to store them, and due to the characteristics of RPA, a large number of images are taken during a series of execution processes.

[0039] Fig. 4 is a diagram for explaining a case where a screen shot is taken every time immediately before an operation on a screen item. In Fig. 4, for example, operations 1 to 98 are successful, operation 99 fails, and a screen shot (image data No. 100) is taken when an error occurs. Screen data Nos. 1 to 100 are stored in the image data storage area, and No. 99 is the screen shot taken immediately before the error occurs and is necessary for identifying the cause of the error, but Nos. 1 to 98 are not necessary for identifying the cause of the error.

[0040] (Solution to implementation issue (1)) This can be solved by incorporating the screenshot capture process into the internal processing (L3) of the screen operation process. The definition is simplified because the user creating the RPA definition (L2) can incorporate the process without being aware of it.

[0041] (Solution to implementation issue (2)) A screenshot is taken every time a screen item is operated, but the image data from the previous capture is discarded and only the most recent one is saved. This allows you to take screenshots before and after an error occurs without taking up space in the image data storage area.

[0042] Solutions to implementation issues (1) and (2) will be described in detail later.

[0043] [2. Configuration] The configuration of information processing device 100 according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the configuration of information processing device 100 according to this embodiment.

[0044] The information processing device 100 is, for example, a commercially available desktop personal computer. Note that the information processing device 100 is not limited to a stationary information processing device such as a desktop personal computer, but may also be a portable information processing device such as a commercially available notebook personal computer, a PDA (Personal Digital Assistant), a smartphone, or a tablet personal computer.

[0045] 5, the information processing device 100 includes a control unit 102, a communication interface unit 104, a storage unit 106, and an input / output interface unit 108. The units included in the information processing device 100 are connected to each other so as to be able to communicate with each other via any communication path.

[0046] The communication interface unit 104 communicably connects the information processing device 100 to a network 300 via a communication device such as a router and a wired or wireless communication line such as a dedicated line. The communication interface unit 104 has a function of communicating data with other devices via the communication line. Here, the network 300 has a function of connecting the information processing device 100 and the server 200 so that they can communicate with each other, and is, for example, the Internet or a LAN (Local Area Network). Note that the data stored in the storage unit 106, which will be described later, may be stored in the server.

[0047] An input device 112 and an output device 114 are connected to the input / output interface unit 108. The output device 114 may be a monitor (including a home television), a speaker, or a printer. The input device 112 may be a keyboard, a mouse, a microphone, or a monitor that cooperates with a mouse to achieve a pointing device function. Note that, hereinafter, the output device 114 may be referred to as the monitor 114, and the input device 112 may be referred to as the keyboard 112 or the mouse 112. Furthermore, an operation performed by a user on the screen (GUI, etc.) of the output device (monitor) 114 using the input device 112 may be simply referred to as a "user operation."

[0048] Various databases, tables, files, etc. are stored in the storage unit 106. Computer programs that work in conjunction with an OS (Operating System) to issue commands to a CPU (Central Processing Unit) to perform various processes are recorded in the storage unit 106. The storage unit 106 can be, for example, a memory device such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a fixed disk device such as a hard disk, a flexible disk, an optical disk, etc.

[0049] The storage unit 106 includes a definition file 106a, a parameter master 106b, a log file 106c, an image file 106d, and the like.

[0050] The definition file 106a is a file for storing definitions (also called "RPA robot execution definitions") that stipulate a series of operation commands for the automatic operation unit (RPA system) 102a to automatically operate an application. A "robot or RPA robot" is a unit that defines a series of operation commands required for one processing execution (one loop), which is defined for each application to be operated. Details of the "RPA robot execution definition" will be described later.

[0051] The log file 106c is a file for storing an execution log created by the automatic operation unit (RPA system) 102a. The execution log includes, for example, three levels of logs: a summary log, a detailed log, and a detailed log (see FIGS. 8 to 11).

[0052] The parameter master 106b is a master for registering various parameters, for example, The number of steps recorded immediately before an error is registered, which defines the number of steps recorded in the image file 106d immediately before an error occurs in the screenshot unit 102a1. The number of steps recorded immediately before an error can be set in the parameter master 106b by an operator.

[0053] Image file (storage area) 106d is a file for storing screen shots taken by screen shot unit 102a1.

[0054] The control unit 102 is a CPU or the like that performs overall control of the information processing device 100. The control unit 102 has an internal memory for storing control programs such as an OS, programs that define various processing procedures, required data, etc., and executes various information processing operations based on these stored programs.

[0055] The control unit 102 is configured to be able to access the definition file 106a, parameter master 106b, log file 106c, image file 106d, etc. stored in the storage unit 106. Note that these master files may be provided in other locations (e.g., server 200) as long as they are accessible by the control unit 102.

[0056] The control unit 102 conceptually includes an automatic operation unit (RPA system) 102a and an application 102b.

[0057] The automatic operation unit (RPA system) 102a executes automatic operations on the application 102b in accordance with a definition that prescribes a series of operations on the application 102b and is registered in a definition file 106a.

[0058] The automatic operation unit (RPA system) 102a conceptually includes a screenshot unit 102a1 and a screen display control unit 102a2. The automatic operation unit 102a is incorporated into the RPA described above. In addition to being incorporated into the RPA, it may also be incorporated into other automatic operation tools such as RDA (Robotic Desktop Automation).

[0059] The screen shot unit 102a1 records screen shots taken before and after an error occurs during automatic operation of an application in the image file 106d.

[0060] The screenshot unit 102a1 may take a screenshot every time just before automatically operating a screen item, and for screenshots before an error, only the screenshots of the number of steps recorded just before the error set in the parameter master 106b may be stored in the image file 106d.

[0061] The screen shot unit 102a1 may store the screen shots before the error in the image file 106d by the number of recording steps immediately before the error in a FI / FO (First In / First Out) method.

[0062] The screen display control unit 102a2 displays a confirmation screen on the monitor 114 for displaying the screen shots before and after the error recorded in the image file 106d. The confirmation screen may display a list of thumbnails of the screen shots before and after the error. The confirmation screen may also display a comparison image that highlights the difference between the screen shots before and after the error.

[0063] When displaying comparison images of three or more screenshots on the confirmation screen, the differences between the different images may be highlighted differently. Highlighting of the differences may include highlighting using highlight colors. Note that highlighting is not limited to highlight colors, and various displays such as highlighting by changing the color, font, size, brightness, etc. of text, or highlighting with arrows may be used, and any display format that can distinguish the differences may be used.

[0064] When three or more comparative images of screenshots are displayed on the confirmation screen, the differences between the different images may be highlighted in different colors to visualize the gradual differences in the screen changes.

[0065] The application 102b includes, for example, an accounting application for inputting various slips such as order entry, purchase entry, order entry, and sales entry, and for printing out various slips. The present invention is also applicable to applications other than these. The applications are stored in the memory unit 106, and the applications stored in the memory unit 106 that are executed by the control unit 102 are conceptually referred to as the application 102b of the control unit 102 in terms of function.

[0066] [3. Specific examples of processing] A specific example of the processing by the control unit 102 of the information processing device 100 in this embodiment will be described with reference to Fig. 6 to Fig. 20. Fig. 6 to Fig. 20 are diagrams for explaining a specific example of the processing by the control unit 102 of the information processing device 100 in this embodiment.

[0067] The automatic operation unit (RPA system) 102a of the control unit 102 executes the following process (flow) in accordance with the RPA robot execution definition (program and data) stored in the definition file 106a. The process (flow) shown below indicates the process executed by the automatic operation unit 102a in accordance with the RPA execution definition, but for ease of explanation, the RPA execution definition may be described as the subject of the operation.

[0068] (3-1. Common (basic) definition) The common definitions of the RPA robot execution definitions will be described with reference to Figures 6 to 11. First, the basic common definitions will be described.

[0069] (RPA flow definition) Figure 6 shows the execution flow, including the processes before and after the execution of an RPA robot. The process assumes that the data to be input to the robot is prepared in Excel or similar and then inputted repeatedly.

[0070] 6, when the RPA flow is started (step S1), an RPA data source is read (step S2). Here, for example, a data source prepared in Excel or the like is read.

[0071] Execute the RPA data source loop (step S3). Here, loop processing is performed for the number of data items read.

[0072] When the loop starts (step S4), the RPA robot is executed (step S7). Detailed settings for how the robot should move are made separately. The success or failure of the execution result is determined. If the execution result is successful (step S8), the next loop is executed (step S10). If the execution result is unsuccessful (step S9), a handling pattern is selected according to the operation. (1) If it is a continue-on-error type (processing continues even after an error occurs) (step S11), the next loop is executed (step S10). (2) If it is a stop-on-error type (processing stops when an error occurs) (step S12), the RPA scenario is terminated (step S13).

[0073] When the loop ends (step S5), the RPA flow ends (step S6).

[0074] (RPA robot execution definition) This section explains the detailed definition of "RPA robot execution." Robot behavior is defined by combining processing components. RPA robot execution definition consists of (1) Layer 1 (L1): RPA flow definition, (2) Layer 2 (L2): robot execution definition, and (3) Layer 3 (L3): internal implementation of screen operation processing.

[0075] (1) Layer 1 (L1): RPA flow definition This is the "RPA flow definition" mentioned above. It is the layer set by the RPA execution configurer.

[0076] (2) Layer 2 (L2): Robot execution definition This is the detailed execution definition of the robot. It is the layer set by the RPA execution configuration person. Operation processes corresponding to the screen items are prepared as components, and these are combined to define the robot's behavior. In general RPA tools, various processing components are prepared in addition to screen operation processes.

[0077] For example, the screen shot capture process is a process that can be incorporated when you want to take a screen shot at any time. The standby process is a process that waits for time-consuming processes such as screen transitions and registration processes. It is common to set the standby time during setup. The screen item value acquisition process is a process that acquires what values ​​have been entered on the screen. The branching process is a process that branches the robot's behavior based on conditions such as the value of a screen item. The loop process is a process used when repeated input is required, such as for detail items.

[0078] (3) Layer 3 (L3): Internal implementation of screen operation processing This is the internal implementation of the "screen operation processing component" that is incorporated into the robot execution definition. This is a layer that the RPA execution configuration setter does not need to be aware of. By incorporating common processes into this layer, RPA configuration can be simplified while providing high functionality.

[0079] Figure 7 is a diagram showing an example of the processing flow of the internal implementation of (1) Layer 1 (L1): RPA flow definition, (2) Layer 2 (L2): robot execution definition, and (3) Layer 3 (L3): screen operation processing.

[0080] In FIG. 7, in the L1:RPA flow definition, when the PRA flow is started (step S21), a series of pre-processing steps are performed (step S22), the RPA robot is executed (step S23), a series of post-processing steps are performed (step S24), and then the RPA flow is terminated (step S25).

[0081] In L2: Robot Execution Definition, when RPA robot execution is started (step S31), for example, input is made into a text box (step S32), input is made into a check box (step S33), input is made into a radio button (step S34), a series of processes are executed (step S35), the registration button is pressed (step S36), and the RPA robot execution is terminated (step S37).

[0082] In the internal implementation of L3: Screen Operation Processing, when the operation processing starts (Step S41), the operation target item is identified (Step S42). For example, the operation target item is identified based on the ID or coordinates. If the identification of the operation target item fails (Step S44), that is, if the operation target item does not exist, an error occurs and "RPA robot execution" is immediately interrupted.

[0083] If the target item for operation is successfully identified (step S43), the input data is acquired (step S45). Specifically, the data to be input into the target item is acquired from the data read in by "Read RPA data source."

[0084] If the acquisition of input data fails (step S47), an error is detected and the "RPA robot execution" is immediately interrupted.

[0085] If the input data is successfully acquired (step S46), a series of pre-processing steps are executed (step S48), an operation process is executed (step S49), a series of post-processing steps are executed (step S50), an execution log is output and stored in the log file 106c (step S51), and the operation process is terminated (step S52).

[0086] (Execution log) The execution log is assumed to have three levels: summary log, detailed log, and detailed log. A summary log is a log created for each execution instruction of an RPA flow. A detailed log is a log output for each processing unit (L1: each step in the RPA flow definition) in one execution of an RPA flow. A detailed log is a log output for each processing unit (L2: each step in the RPA robot execution definition) in the RPA robot execution definition. Note that the execution log may also include data on the success and error (failure) of the execution results.

[0087] 8 is a diagram showing an example of a summary log. The summary log may include data such as SEQ, log level, execution definition name, execution start date and time, execution end date and time, execution ID, and execution type CD.

[0088] 9 is a diagram showing an example of a detailed log. The detailed log may include data such as an execution ID, a sequence number, a log level, execution details, the name of the executing robot, the name of the application to be operated, a loop sequence number, the execution start date and time, and the execution end date and time.

[0089] 10 is a diagram showing an example of a detailed log. The detailed log may include data such as an execution ID, a detailed sequence number, a sequence number, a log level, execution details, an operation target item name, input information, and execution date and time.

[0090] 11 is a diagram showing an example of a screenshot log. The screenshot log may include an execution ID, a detail SEQ, a detailed SEQ, and an example of screen data.

[0091] (3-2. Implementation and Processing of Screen Shot Unit 102a1) Screen shot unit 102a1 records screen shots taken before and after an error occurs during automatic operation of an application in image file (storage area) 106d. Screen shot unit 102a1 may also take a screen shot every time immediately before automatically operating a screen item, and store only the screen shots taken before the error that correspond to the number of steps recorded immediately before the error set in parameter master 106b in image file 106d. Screen shot unit 102a1 may also store the screen shots taken before the error in image file 106d using a FI / FO method.

[0092] The implementation and processing of the screenshot unit 102a1 will be explained in association with the solutions to the above-mentioned implementation problems (1) and (2).

[0093] (Solution to implementation issue (1)) Setting the screen shot capture process in the definition (L2) for each operation makes the definition creation cumbersome and the definition itself becomes complicated. This problem is solved in this embodiment by incorporating the screen shot capture process into the internal processing (L3) of the screen operation processing. The definition can be simplified because the process can be incorporated without the user creating the RPA definition (L2) being aware of it.

[0094] The reason for implementing the screen shot capturing process in the internal process (L3) of the screen operation process will be described in detail with reference to Figures 12 and 13. Figure 12 is a diagram for explaining the case where the screen shot capturing process is implemented in the RPA definition (L2). In Figure 12, steps equivalent to those in Figure 7 are assigned the same step numbers.

[0095] In Figure 12, when implementing the screenshot capture process in the RPA definition (L2), the screenshot creation process must be set in the RPA definition (L2) for each operation. In the example shown in Figure 12, the screenshot capture process must be set for each text box input, check box input, radio button input, and register button operation. This makes the creation of the RPA definition (L2) cumbersome, and the definition itself becomes complicated.

[0096] Fig. 13 is a diagram for explaining in detail the case where the screenshot capturing process is implemented in the internal process (L3) of the screen operation process. In Fig. 13, the same step numbers as in Fig. 7 are assigned.

[0097] In FIG. 13, the definition can be simplified by incorporating the screenshot capture process (step T1) before the identification of the operation target item (step S41) of the internal process (L3) rather than in the RPA definition (L2). As a result, a screenshot can be captured when the "immediate process in the RPA definition (L2)" has been successfully completed. For example, if the step in question is "(2) Checkbox input (step S33)," a screenshot will be captured when "(1) Textbox input (step S32)" has been successfully completed.

[0098] An error is detected for each processing step of the internal processing (L3), and if an error occurs (step S44), a screen shot is taken.

[0099] (Solution to implementation issue (2)) To address the issue of the large data volume of the screenshot log if a screenshot is taken every time a screen item is operated (identifying the item to be operated (step S42)), a screenshot is taken every time a screen item is operated (identifying the item to be operated (step S42)), but the image data from the previous shot is discarded and only the most recent one is kept. This makes it possible to take screenshots before and after an error occurs without putting pressure on the image file 106d (storage area).

[0100] FIG. 14 is a diagram for explaining a case where, for example, a screen shot is taken every time immediately before an operation on a screen item is performed, but image data from the previous shot is discarded and only the latest image is kept.

[0101] In FIG. 14, for example, in the case where operations 1 to 98 are successful, operation 99 fails, and a screen shot (image data No. 100) is taken at the time of the error, image data is temporarily stored in image file 106d (image data storage area) for each shot, but the image data from the previous shot is discarded and only the most recent data (image data No. 99) is retained.

[0102] The number of steps recorded immediately before an error in the parameter master 106b may be set to the number of steps recorded immediately before an error. For example, if the number of steps recorded immediately before an error is 3, three pieces of image data of screenshots immediately before an error occurs are recorded, and in the example shown in Fig. 14, image data of three screenshots No. 97 to 99 are recorded.

[0103] FIG. 15 is a diagram for explaining how screenshots are stored in memory. In FI / FO format, when a new screenshot is taken, the data of the oldest screenshot is discarded. The number of screenshots to be recorded can be changed depending on the number of steps recorded immediately before an error in the parameter master 106b. In the example shown in FIG. 15, the number of steps recorded immediately before an error is 3.

[0104] If step 3 is successful, screenshots (1), (2), and (3) are saved as shown in Figure 15. If step 4 is successful, the oldest screenshot (1) is discarded. If an error occurs in step 5, the screenshot immediately after the error occurs, as well as screenshots (2), (3), and (4), are saved.

[0105] (3-3. Processing of Screen Display Control Unit 102a2) The display processing of the screen display control unit 102a2 will be described in detail with reference to Figs. 16 to 20. Figs. 16 to 20 show images of reference screens. The screen display control unit 102a2 displays an RPA log list screen based on the logs recorded in the log file 106c on the monitor 114, and displays the screen shots before and after the error recorded in the image file 106d on a screen shot confirmation screen in response to an operator's operation on the RPA log list screen. The screen display control unit 102a2 displays a list of thumbnails of the screen shots before and after the error on the screen shot confirmation screen, and may also highlight the difference between the screen shots before and after the error.

[0106] Figure 16(A) is a diagram showing an example of the RPA log list screen. The RPA log list screen displays the SEQ, the result of the automatic operation (success or failure), the target item name, the processing content of the automatic operation, and a screenshot confirmation button linked to the error log when the result of the automatic operation is "failure." Pressing the screenshot confirmation button launches a screenshot confirmation screen like the one shown in Figure 16(B).

[0107] 16(B) is a diagram showing an example of the screenshot confirmation screen. The screenshot confirmation screen has a thumbnail list area where a list of captured screenshots is displayed as thumbnails, and an image display area where the captured screenshots are displayed.

[0108] The thumbnail list area allows you to scroll through the thumbnails, and the image display area displays the screenshot selected in the thumbnail list area, allowing you to switch between screenshots using the buttons to switch between previous and next images.

[0109] Fig. 17 is a diagram for explaining an example of operation on the screenshot confirmation screen. Fig. 18 is a diagram showing an example of comparative image 1 (image No. 99: immediately before the error). Fig. 19 is a diagram showing an example of comparative image 2 (image No. 100: immediately after the error). Comparative image 2 (image No. 100: immediately after the error) in Fig. 19 is an error that occurred when an invalid code was entered when entering the XX code.

[0110] In Figure 17, when multiple images (thumbnails) are selected in the thumbnail list area and the image comparison button is pressed, a comparison image of the selected multiple images is displayed in the image display area. In the image display area, multiple images are compared pixel by pixel, and an enhanced comparison image is displayed with several pixels surrounding points of difference highlighted. This makes it possible to provide information that enables more intuitive cause identification.

[0111] Two or more images can be selected as comparison images. For example, if two images, Image 1 and Image 2, are selected, a comparison image is displayed in which the differences between Image 1 and Image 2 are highlighted. If three images, Image 1, Image 2, and Image 3, are selected, a comparison image is displayed in which the differences between adjacent images are highlighted, and a comparison image is displayed in which the differences between Image 1 and Image 2 and the differences between Image 2 and Image 3 are highlighted. If four images, Image 1, Image 2, Image 3, and Image 4, are selected, a comparison image is displayed in which the differences between Image 1 and Image 2, the differences between Image 2 and Image 3, and the differences between Image 3 and Image 4 are highlighted. Each difference may be displayed in a different highlight color.

[0112] In the example shown in Figure 17, image No. 99 (just before the error) and image No. 100 (just after the error) are selected in the thumbnail list area and the image comparison button is pressed, and a comparison image is displayed in the image display area, with several pixels around the difference between image No. 99 (just before the error) and image No. 100 (just after the error) highlighted in highlight color. In the example shown in Figure 17, the difference between the two images, "XXX This code cannot be used because it is classified as B," is highlighted in highlight color.

[0113] Figure 20 shows an example of the display when three images are selected on the screenshot confirmation screen. When three or more screenshots are selected from the thumbnail list display area and compared, the gradual differences in the images are visualized by using multiple highlight colors to indicate the differences.

[0114] In the example shown in Figure 20, three images, image No. 98 (before the error), image No. 99 (just before the error), and image No. 100 (just after the error), are selected in the thumbnail list area and the image comparison button is pressed, and comparison images are displayed in the image display area, highlighting in different highlight colors the differences between image No. 98 (before the error) and image No. 99 (just before the error) and the differences between image No. 99 (just before the error) and image No. 100 (just after the error).

[0115] As described above, according to this embodiment, there are provided screenshot unit 102a1 that records in a memory area screenshots taken before and after an error has occurred during automatic operation of an application, and screen display control unit 102a2 that displays the screenshots taken before and after the error recorded in the memory area on a confirmation screen. Therefore, when an error occurs during automatic operation, it becomes possible to visually and intuitively understand the cause of the error.

[0116] [4. Contribution to the United Nations-led Sustainable Development Goals (SDGs)] This embodiment can contribute to improving business efficiency and promoting appropriate management decisions by companies, thereby contributing to the achievement of SDGs Goals 8 and 9.

[0117] Furthermore, this embodiment can contribute to reducing waste and promoting paperless and electronic systems, thereby contributing to the achievement of SDGs Goals 12, 13, and 15.

[0118] Furthermore, this embodiment can contribute to strengthening control and governance, which can contribute to the achievement of Goal 16 of the SDGs.

[0119] 5. Other Embodiments The present invention may be implemented in various different embodiments other than those described above within the scope of the technical concept set forth in the claims.

[0120] For example, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods.

[0121] Furthermore, the processing procedures, control procedures, specific names, information including parameters such as registered data and search conditions for each process, screen examples, and database configurations shown in this specification and drawings can be changed as desired unless otherwise specified.

[0122] Furthermore, with regard to the information processing device 100, the components shown in the figures are functional concepts, and do not necessarily have to be physically configured as shown in the figures.

[0123] For example, all or any part of the processing functions of the information processing device 100, particularly the processing functions performed by the control unit 102, may be implemented by a CPU and a program interpreted and executed by the CPU, or may be implemented as hardware using wired logic. The program is recorded on a non-transitory computer-readable recording medium containing programmed instructions for causing the information processing device to execute the processes described in this embodiment, and is mechanically read by the information processing device 100 as needed. That is, a computer program for providing instructions to the CPU in cooperation with the OS and performing various processes is recorded in the storage unit 106, such as a ROM or HDD (Hard Disk Drive). The computer program is executed by being loaded into RAM, and cooperates with the CPU to configure the control unit 102.

[0124] This computer program may also be stored in an application program server connected to the information processing device 100 via any network, and all or part of it may be downloaded as needed.

[0125] Furthermore, the program for executing the processes described in this embodiment may be stored in a non-transitory computer-readable recording medium or configured as a program product. Here, the term "recording medium" includes any "portable physical medium" such as a memory card, a Universal Serial Bus (USB) memory, a Secure Digital (SD) card, a flexible disk, a magneto-optical disk, a ROM, an Erasable Programmable Read Only Memory (EPROM), an Electrically Erasable and Programmable Read Only Memory (EEPROM (registered trademark)), a Compact Disk Read Only Memory (CD-ROM), a Magneto-Optical disk (MO), a Digital Versatile Disk (DVD), and a Blu-ray (registered trademark) disc.

[0126] Furthermore, a "program" is a data processing method written in any language or description method, regardless of the format, such as source code or binary code. Note that a "program" is not necessarily limited to a single structure, but also includes a structure that is distributed as multiple modules or libraries, or a structure that achieves its function by cooperating with a separate program, such as an OS. Note that the specific configuration and reading procedure for reading a recording medium in each device shown in the embodiments, as well as the installation procedure after reading, can use well-known configurations and procedures.

[0127] The various databases stored in the memory unit 106 are storage means such as memory devices such as RAM and ROM, fixed disk devices such as hard disks, flexible disks, and optical disks, and store various programs, tables, databases, and web page files used for various processes and providing websites.

[0128] The information processing device 100 may be configured as a known information processing device such as a personal computer or a workstation, or may be configured as the information processing device to which any peripheral device is connected. The information processing device 100 may be realized by installing software (including programs, data, etc.) that causes the information processing device to perform the processes described in this embodiment.

[0129] Furthermore, the specific form of distribution and integration of the devices is not limited to that shown in the drawings, and all or part of them can be configured by functionally or physically distributing and integrating them in any unit depending on various additions or function additions. In other words, the above-described embodiments can be implemented in any combination, or embodiments can be implemented selectively. [Explanation of symbols]

[0130] 100 Information processing device 102 Control section 102a Automatic operation unit (RPA system) 102a1 Screenshot section 102a2 Screen display control unit 102b Application 104 Communication interface unit 106 Storage section 106a definition file 106b Parameter Master 106c Log File 106d image file 108 Input / Output Interface Section 112 Input Device 114 Output Device 200 servers 300 Network

Claims

[Claim 1] An information processing device that automatically operates an application in accordance with a definition that prescribes a series of operations, a screen capture means for recording screen shots taken before and after an error occurs during the automatic operation of the application in a storage area; a screen display means for displaying a confirmation screen for confirming the screen shots before and after the error recorded in the storage area; Equipped with The information processing device is characterized in that the screenshot means takes a screenshot every time just before automatically operating a screen item, and for screenshots before an error, only the screenshots of the number of steps recorded just before the error that are set are stored in the memory area.

Images