Screenshot capture device, screenshot capture method and program

Abstract

To provide a screen shot photographing device, a screen shot photographing method and a program which can photograph a screen shot including a hidden area in which operation target screens overlap with each other.SOLUTION: A screen shot photographing device acquires an execution ID of an operation target program; acquires screen IDs of all screens provided by the operation target program identified by the acquired execution ID; photographs screen shots of the screens identified by each of all acquired screen IDs; and saves all photographed screen shots in a storage unit.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a screen shot photographing device, a screen shot photographing method, and a program. [Background technology]

[0002] As companies are increasingly adopting the automated operation of business systems (application programs) using RPA (Robotic Process Automation) tools, there is an increasing need to understand the execution status of the automated operations on the screens provided by the business systems (in other words, various scenes during the execution of the RPA tool) for the purpose of collecting evidence. However, because the execution speed of the automated operations is fast, it is difficult for a person to visually check the execution status. Moreover, if a person who checks the execution status must be present when the automated operations are being executed, there is a risk that the significance of adopting the RPA tool in the first place will be lost. In addition, although it is possible for a person to understand the execution status by looking at and checking the text log in which the execution status is described, it is difficult for a person to intuitively understand the execution status from the text log alone. Therefore, in order to support a person's intuitive understanding of the execution status, screenshots of the screens that are the target of the automated operations are taken.

[0003] Patent document 1 describes a screenshot method for a terminal in which a screenshot is taken once based on the image change status of the screenshot reference area each time the screen scrolls a specified distance, and a final screenshot image is obtained based on the images screenshotted multiple times and an initial screenshot image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2019-533251 Summary of the Invention [Problem to be solved by the invention]

[0005] However, up until now, the screen shot of the screen to be automatically operated (the operation target screen) was taken without considering the state of the operation target screen. Therefore, when multiple application programs were running, the screen shot was taken in a state where the operation target screens were overlapping each other, which resulted in a problem that the hidden screen items could not be confirmed.

[0006] The present invention has been made in consideration of the above, and aims to provide a screen shot taking device, a screen shot taking method, and a program that are capable of taking screen shots including areas where screens to be operated overlap and are hidden. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, the screen shot photographing device of the present invention is a screen shot photographing device having a control unit that photographs screen shots of all of a plurality of screens provided by a target program which is an application program to be operated, the control unit comprising: execution identifier acquisition means for acquiring an execution identifier of the target program being operated; screen identifier acquisition means for acquiring screen identifiers of all of the screens provided by the target program identified by the execution identifier acquired by the execution identifier acquisition means; photographing means for photographing screen shots of the screens identified by each of all of the screen identifiers acquired by the screen identifier acquisition means; and saving and executing means for saving all of the screen shots photographed by the photographing means in a memory unit; the save / execute means saves the screen shots by assigning numbers indicating the order in which the screen shots were taken so that the user can later check the screen context in a state where the error occurrence state can be understood and the automatic operation result can be kept as a record, the execution identifier is a process of the target program, and the screen identifier is a window handle of the screen, The photographing means is Specify the window handle of the screen to be shot, The feature of this method is that screen shots are taken for each layer, starting from the bottom layer, without overlapping with other screens.

[0008] In the screen shot photographing device according to the present invention, the number of screens provided by the target program is preferably multiple.

[0009] In addition, the screen shot capturing device of the present invention is capable of accessing master data in which a screen identifier is linked to one or more control identifiers for vertical or horizontal scrolling and / or a control identifier for page movement, and in the screen shot capturing device of the present invention, the capturing means executes the following processes (1) to (3) for all screen identifiers acquired by the screen identifier acquisition means. (1) Obtain the control identifier associated with the screen identifier from the master data. (2) If the control identifier cannot be obtained, a screenshot of the screen identified by the screen identifier is taken. (3) If the control identifier is obtained, (31) obtains scrolling information or page movement information from each control identifier, (32) sets the scrolling and / or page movement state of the screen identified by the screen identifier to an initial state according to each piece of obtained information, and takes screenshots, and (33) performs scrolling and / or page movement processing according to each piece of obtained information on the screen identified by the screen identifier, continuing to take screenshots until the processing can no longer be performed.

[0010] In the screen shot photographing device according to the present invention, the photographing means photographs the screen shot while executing the processes of scrolling and page moving in the process of (33) in that order.

[0011] In the screen shot photographing device according to the present invention, the photographing means, in the process of (33), executes the scrolling process in the order of vertical scrolling and horizontal scrolling.

[0012] In the screen shot photographing device according to the present invention, the scroll process uses the size of the scroll box as a unit of movement of the scroll box.

[0013] In addition, in the screen shot capturing device of the present invention, the saving execution means is characterized in that it saves the screen shots arranged on one sheet based on the scrolling information and / or page movement information saved when the screen shots are captured.

[0014] In the screen shot photographing device according to the present invention, the photographing means photographs the screen shots in order from the bottom screen.

[0015] In the screen shot photographing device according to the present invention, the save execution means saves the screen shots by assigning numbers to the screen shots indicating the order in which they were photographed.

[0016] In the screen shot photographing device according to the present invention, the application program is a business application program.

[0017] In the screen shot photographing device according to the present invention, the operation is an automatic operation by an RPA (Robotic Process Automation) tool or a user operation related to taking a screen shot.

[0018] In addition, in the screen shot capturing device of the present invention, the execution identifier is a process of the target program, the screen identifier is a window handle of the screen, and the capturing means specifies the window handle of the screen to be captured and captures screen shots in order from the bottom screen.

[0019] A screen shot capturing method according to the present invention is a screen shot capturing method for capturing screen shots of all of a plurality of screens provided by a target program, which is an application program to be operated and executed by an information processing device having a control unit, comprising the steps of: acquiring an execution identifier of the target program being operated and executed by the control unit; acquiring screen identifiers of all of the screens provided by the target program identified by the acquired execution identifier; and capturing screen shots of the screens identified by each of all of the acquired screen identifiers. By shooting means A step of taking a photograph; In order to allow people to check the error situation in a way that allows them to understand it after the fact in the context of the screen, All images captured by the imaging means Save the screenshot to the memory to keep a record of the automated operation results. and a storing step, In the saving step, the screen shots are saved by assigning numbers indicating the order of capture to the screen shots, the execution identifier is a process of the target program, and the screen identifier is a window handle of the screen; In the photographing step, Specify the window handle of the screen to be shot, The feature of this method is that screen shots are taken for each layer, starting from the bottom layer, without overlapping with other screens.

[0020] A program according to the present invention is a program for taking screen shots of all of a plurality of screens provided by a target program, which is an application program to be operated, to be executed by an information processing device having a control unit, the program including a step of acquiring an execution identifier of the target program being operated to be executed by the control unit, a step of acquiring screen identifiers of all of the screens provided by the target program identified by the acquired execution identifier, and a step of taking screen shots of the screens identified by each of all of the acquired screen identifiers. By shooting means A step of taking a photograph; In order to allow people to check the error situation in a way that allows them to understand it after the fact in the context of the screen, All images captured by the imaging means Save the screenshot to the memory to keep a record of the automated operation results. and a storing step, In the saving step, the screen shots are saved by assigning numbers indicating the order of capture to the screen shots, the execution identifier is a process of the target program, and the screen identifier is a window handle of the screen; In the photographing step, Specify the window handle of the screen to be shot, The feature of this method is that screen shots are taken for each layer, starting from the bottom layer, without overlapping with other screens. Effect of the Invention

[0021] According to the present invention, it is possible to take a screen shot including an area where the operation target screens overlap and are hidden. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a diagram showing an example of a problem that occurs when taking a screen shot of a screen to be operated. [Diagram 2] FIG. 2 is a diagram showing a mechanism for dividing and photographing only the screen related to the automatic operation target. [Diagram 3] FIG. 3 is a diagram showing a mechanism for capturing an image of the entire screen to be operated, taking into account vertical and horizontal scrolling. [Figure 4] FIG. 4 is a diagram showing a mechanism for capturing images of the entire screen to be operated, taking into account scrolling and page movement. [Diagram 5] FIG. 5 is a diagram showing an example of the configuration of the screen shot photographing device 100. As shown in FIG. [Figure 6] FIG. 6 is a diagram showing an example of a flowchart relating to the screen shot photographing process executed by the screen shot photographing device 100 when only the screen related to the automatic operation target is divided and photographed in its entirety. [Figure 7] FIG. 7 is a diagram showing an example of a flowchart relating to a screen shot photographing process executed by a screen shot photographing device 100 whose OS is Windows 10 (registered trademark) when only the screen related to an automatic operation target is divided and photographed in its entirety. [Figure 8] FIG. 8 is a diagram showing an example of a flowchart relating to the screen shot photographing process executed by the screen shot photographing device 100 when the screen is divided and photographed in its entirety, taking into consideration screen scrolling and page movement. [Figure 9] FIG. 9 is a diagram showing an example of a screen shot storage image when the screen is divided and all of the screen is photographed, taking into consideration screen scrolling and page movement. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an embodiment of a screen shot photographing device, a screen shot photographing method, and a program according to the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiment.

[0024] [1. Overview] Here, an overview of this embodiment will be described with reference to FIG. 1 to FIG. 4 and so on.

[0025] During RPA execution, there are various situations where you may want to take a screenshot. Examples of these situations include "after inputting important items in the operation screen" (Scene Example 1), "immediately before confirmation processing after inputting all items in the operation screen" (Scene Example 2), "immediately after confirmation processing after inputting all items in the operation screen" (Scene Example 3), "after transition of the operation screen" (Scene Example 4), and "immediately before automatic operation stops due to an error" (Scene Example 5). Note that the purpose of taking a screenshot in Scene Example 1 is, for example, to check whether values ​​are entered into important items. Also, the purpose of taking a screenshot in Scene Example 2 is, for example, to check whether all input information is correct. Also, the purpose of taking a screenshot in Scene Example 3 is, for example, to check whether confirmation processing has been performed appropriately. Also, the purpose of taking a screenshot in Scene Example 4 is, for example, to check whether the screen has switched appropriately. Also, the purpose of taking a screenshot in Scene Example 5 is, for example, to identify the cause of the operation stopping.

[0026] However, up until now, screenshots of the target screen have been taken without considering the state of the target screen. As a result, as shown in Fig. 1, for example, problems have arisen such as "message dialogs or other screens overlap, so the input items you want to check cannot be confirmed from the screenshot" (Problem Example 1) and "Screen items may be hidden by scrolling or moving pages within the screen, so the input items you want to check cannot be confirmed from the screenshot" (Problem Example 2). In other words, there were times when the screen items you wanted to check could not be confirmed from the screenshot. Fig. 1 is a diagram showing an example of a problem that occurs when taking a screenshot of the target screen.

[0027] Therefore, as shown in Figure 2, we developed a mechanism to identify only the screen to be operated and its related screens, split them up one by one, and take screenshots. With this mechanism, screens that are not related to the screen to be operated are not captured, so no unnecessary screenshots are taken. Also, with this mechanism, screenshots are taken in order starting from the bottommost overlapping screen, so the chronological relationship of the screens can be understood. Figure 2 shows the mechanism to split up and capture only the screens related to the automatic operation target.

[0028] Based on the above mechanism, we have also developed a mechanism for splitting the target screen into individual shots while scrolling the screen by recognizing the vertical and horizontal scroll bars, as shown in Figure 3. With this mechanism, the screen shots are rearranged and displayed taking into account the vertical and horizontal scrolls, so the relative positions of each screen shot can be understood at a glance. Figure 3 shows the mechanism for splitting the target screen and taking all shots while taking into account the vertical and horizontal scrolls.

[0029] Furthermore, based on the above mechanism, we have also developed a mechanism that recognizes the scroll bars and page navigation buttons on the target screen, and automatically changes the screen while splitting it into individual shots and taking screenshots, as shown in Figure 4. With this mechanism, screenshots are rearranged and displayed taking into account scrolling and page navigation, so item information can be understood at a glance. Figure 4 is a diagram showing the mechanism that splits the target screen and takes all screenshots while taking into account scrolling and page navigation.

[0030] [2. Configuration] Here, an example of the configuration of the screen shot photographing device 100 according to this embodiment will be described with reference to FIG.

[0031] 5 is a block diagram showing an example of the configuration of the screen shot photographing device 100. The screen shot photographing device 100 photographs all screens (the screens provided by the target program may be one or more) provided by a target program, which is an application program (e.g., a business application program) that is the target of an operation (e.g., an automatic operation by an RPA tool or an operation related to taking a screen shot by a user), and is, for example, a business application program that is the target of an automatic operation by an RPA tool and an RPA tool that has been introduced.

[0032] The screenshot photographing device 100 is a commercially available desktop personal computer. The screenshot photographing device 100 is not limited to a stationary information processing device such as a desktop personal computer, but may be a portable information processing device such as a commercially available notebook personal computer, a PDA (Personal Digital Assistant), a smartphone, or a tablet personal computer.

[0033] The screen shot photographing device 100 includes a control unit 102, a communication interface unit 104, a storage unit 106, and an input / output interface unit 108. Each unit included in the screen shot photographing device 100 is connected to each other so as to be able to communicate with each other via an arbitrary communication path.

[0034] The communication interface unit 104 communicably connects the information processing device 100 to a network 300 via a communication device such as a router and a wired or wireless communication line such as a dedicated line. The communication interface unit 104 has a function of communicating data with other devices via the communication line. Here, the network 300 has a function of connecting the screen shot photographing device 100 and the server 200 so that they can communicate with each other, and is, for example, the Internet or a LAN (Local Area Network). Note that the data stored in the storage unit 106 may be stored in the server device 200, for example.

[0035] An input device 112 and an output device 114 are connected to the input / output interface unit 108. The output device 114 may be a monitor (including a home television), a speaker, or a printer. The input device 112 may be a keyboard, a mouse, a microphone, or a monitor that cooperates with a mouse to realize a pointing device function. In the following, the output device 114 may be referred to as the monitor 114, and the input device 112 may be referred to as the keyboard 112 or the mouse 112.

[0036] Various databases, tables, files, etc. are stored in the storage unit 106. Computer programs for cooperating with an OS (Operating System) to give instructions to a CPU (Central Processing Unit) to perform various processes are recorded in the storage unit 106. For example, memory devices such as RAM (Random Access Memory) and ROM (Read Only Memory), fixed disk devices such as hard disks, flexible disks, and optical disks can be used as the storage unit 106.

[0037] The storage unit 106 includes, for example, master data 106a, a screen shot storage unit 106b in which captured screen shots are stored, etc. The master data 106a stores a screen identifier (e.g., a screen ID) and one or more control identifiers (e.g., control IDs) for vertical or horizontal scrolling and / or a control identifier (e.g., control ID) for page movement in association with each other.

[0038] Returning to Fig. 5, the control unit 102 is a CPU or the like that performs overall control of the screen shot photographing device 100. The control unit 102 has an internal memory for storing a control program for an OS (e.g., Windows (registered trademark), macOS (registered trademark), UNIX (registered trademark) (including Linux (registered trademark), etc.), Android (registered trademark), iOS (registered trademark), etc.), a program that defines various processing procedures, etc., required data, etc., and executes various information processing based on these stored programs.

[0039] The control unit 102 conceptually includes, in terms of functions, an execution identifier acquisition unit 102a, a screen identifier acquisition unit 102b, a shooting unit 102c, and a storage execution unit 102d.

[0040] The execution identifier acquisition unit 102a acquires an execution identifier (e.g., execution ID) of a target program during operation execution. For example, when the OS is Windows 10 (registered trademark), the execution identifier acquisition unit 102a acquires a process (an ID that can uniquely identify the execution of a program) of the target program during operation execution.

[0041] The screen identifier acquisition unit 102b acquires screen identifiers (e.g., screen IDs) of all screens provided by the target program identified by the execution identifier acquired by the execution identifier acquisition unit 102a. For example, when the OS is Windows 10 (registered trademark), the screen identifier acquisition unit 102b acquires window handles (IDs that can uniquely identify a screen to recognize a screen or a control) of all screens provided by the target program identified by the process acquired by the execution identifier acquisition unit 102a.

[0042] The image capturing unit 102c captures screen shots of the screens identified by all the screen identifiers acquired by the screen identifier acquisition unit 102b. The image capturing unit 102c may capture screen shots in order from the bottom screen. For example, when the OS is Windows 10 (registered trademark), the window handle of the screen to be captured is specified, and screen shots are captured in order from the bottom screen.

[0043] Furthermore, the photographing unit 102c may execute the following processes (1) to (3) for all the screen identifiers acquired by the screen identifier acquisition unit 102b. (1) A control identifier (eg, control ID) associated with the screen identifier is obtained from the master data 106a. (2) If the control identifier cannot be obtained, a screenshot of the screen identified by the screen identifier is taken. (3) If the control identifier is obtained, (31) obtains scrolling information or page movement information from each control identifier, (32) sets the scrolling and / or page movement state of the screen identified by the screen identifier to an initial state according to each piece of obtained information, and takes screenshots, and (33) performs scrolling and / or page movement processing according to each piece of obtained information on the screen identified by the screen identifier, continuing to take screenshots until the processing can no longer be performed.

[0044] Also, the photographing unit 102c may take a screen shot while performing the scroll process and the page movement process in this order in the process of (33). Also, the photographing unit 102c may perform the scroll process in the order of vertical scroll and horizontal scroll in the process of (33). Also, the scroll process may use the size of the scroll box as the unit of movement of the scroll box.

[0045] The save execution unit 102d saves all the screen shots taken by the photographing unit 102c in the screen shot storage unit 106b.

[0046] The save execution unit 102d may save the screen shots by assigning numbers to the screen shots indicating the order in which they were taken.

[0047] Furthermore, the save execution unit 102d may arrange and save the screen shots on one sheet based on the scroll information and / or page movement information saved when the screen shots were taken.

[0048] [3. Specific examples of processing] Here, a specific example of the process executed by the screen shot photographing device 100 will be described with reference to Figs. 6 to 9 and the like.

[0049] [3-1. Specific example 1] FIG. 6 is a diagram showing an example of a flowchart relating to the screen shot photographing process executed by the screen shot photographing device 100 when only the screen related to the automatic operation target is divided and photographed in its entirety.

[0050] First, the execution identifier acquisition unit 102a acquires the execution ID of the operation target program (step SA1).

[0051] Next, the screen identifier acquisition unit 102b acquires all screen IDs of the program identified by the execution ID acquired in step SA1 (step SA2).

[0052] Next, the photographing unit 102c photographs screen shots of screens identified by the screen IDs in ascending order of the screen IDs based on all the screen IDs acquired in step SA2 (step SA3).

[0053] Next, the save execution unit 102d assigns the order of capture to the file names of all the screen shots taken in step SA3, and saves the assigned screen shots in the screen shot storage unit 106b (step SA4).

[0054] [3-2. Specific example 2] FIG. 7 is a diagram showing an example of a flowchart relating to a screen shot photographing process executed by a screen shot photographing device 100 whose OS is Windows 10 (registered trademark) when only the screen related to an automatic operation target is divided and photographed in its entirety.

[0055] First, the execution identifier acquisition unit 102a acquires the process of the operation target program (step SB1).

[0056] Next, screen identifier acquisition unit 102b acquires all window handles of the operation target screens of the program identified by the process acquired in step SB1 (step SB2).

[0057] Next, the image capturing unit 102c designates the window handle of the screen to be captured from among the window handles acquired in step SB2 (step SB3).

[0058] Next, the image capturing unit 102c captures screen shots of the screens to be captured, in order from the bottom layer, based on the window handle designated in step SB3 (step SB4).

[0059] Next, the save execution unit 102d assigns the order of capture to the file names of all the screen shots taken in step SB4, and saves the assigned screen shots in the screen shot storage unit 106b (step SB5).

[0060] [3-3. Example 3] Fig. 8 is a diagram showing an example of a flowchart relating to the screenshot photographing process executed by the screenshot photographing device 100 when the screen is divided and photographed in its entirety, taking into consideration scrolling of the screen and page movement. Note that this flowchart is a specific example of step SA3 in the flowchart shown in Fig. 6, and therefore a description of the processes corresponding to steps SA1 and SA2 shown in Fig. 6 will be omitted.

[0061] First, the photographing unit 102c acquires a control ID associated with the acquired screen ID from the master data 106a (step SC1).

[0062] Next, the image capturing unit 102c acquires information on scrolling and page movement of the operation target screen from the control ID acquired in step SC1 (step SC2).

[0063] Next, the photographing unit 102c scrolls and moves the page to the initial position, and photographs a screen shot after the movement (step SC3).

[0064] Next, the photographing unit 102c determines whether or not page movement is possible on the operation target screen (step SC4).

[0065] Next, if page movement is possible, the image capturing unit 102c determines whether horizontal scrolling is possible for the operation target screen (step SC5). Specifically, the image capturing unit 102c determines whether horizontal scrolling is possible from the size of the scroll area and the size of the scroll box.

[0066] Next, if horizontal scrolling is possible, the image capturing unit 102c determines whether vertical scrolling is possible for the operation target screen (step SC6). Specifically, the image capturing unit 102c determines whether vertical scrolling is possible from the size of the scroll area and the size of the scroll box.

[0067] Next, if vertical scrolling is possible, the photographing unit 102c executes vertical scrolling processing and takes a screen shot of the operation target screen (step SC7). Specifically, the photographing unit 102c moves the scroll position by the size of the vertical scroll box and takes a screen shot of the operation target screen. After taking the shot, the process returns to step SC6.

[0068] Steps SC6 and SC7 are repeated until vertical scrolling is no longer possible. Specifically, they are repeated until the scroll position of the vertical scroll box moves to the final position. This allows for screen shots taken while fixing the page and horizontal scrolling and only performing vertical scrolling (specifically, each screen shot (z=0,1,...) corresponding to (0,0,z) shown in Figure 9).

[0069] Next, the photographing unit 102c executes a horizontal scroll process and takes a screen shot of the operation target screen (step SC8). Specifically, the photographing unit 102c moves the scroll position by the size of the horizontal scroll box and takes a screen shot of the operation target screen. After taking the shot, the process returns to step SC5.

[0070] Steps SC5 to SC8 are repeated until horizontal scrolling is no longer possible. Specifically, they are repeated until the scroll position of the horizontal scroll box moves to the final position. This allows for the acquisition of screen shots taken while the page is fixed and horizontal scrolling is further performed (specifically, each screen shot corresponding to (0, y, z) shown in FIG. 9 (y=0, 1, 2, . . .; z=0, 1, . . .)). Steps SC5 to SC8 are also performed the same number of times as the number of scroll bars.

[0071] Next, the photographing unit 102c executes a page moving process and takes a screen shot of the operation target screen (step SC9). Specifically, the photographing unit 102c moves the page by one page and takes a screen shot of the operation target screen. After taking the shot, the process returns to step SC4.

[0072] Steps SC4 to SC9 are repeated until page navigation is no longer possible. Specifically, they are repeated until the page moves to the final page. This allows screen shots taken while page navigation is also being performed (specifically, each screen shot corresponding to (x, y, z) shown in FIG. 9 (x=0,1,...; y=0,1,2,...; z=0,1,...) to be obtained.

[0073] Then, the save execution unit 102d arranges the screen shots on one sheet as shown in Fig. 9 based on the information on scrolling and page movement stored when the screen shots were captured. When page movement and horizontal scrolling are applied, the save execution unit 102d arranges the screen shots in order horizontally as shown in Fig. 9. When vertical scrolling is applied, the save execution unit 102d arranges the screen shots in order vertically as shown in Fig. 9. The save execution unit 102d then saves this sheet in the screen shot storage unit 106b. Fig. 9 is a diagram showing an example of a screen shot storage image when the screen is divided and all the screen shots are captured, taking into account screen scrolling and page movement.

[0074] [4. Summary of the present embodiment] As described above, according to this embodiment, when taking a screen shot, it is possible to take a picture including an area that would normally be hidden. Specifically, it is possible to take a picture of an item that is hidden by overlapping other screens, or to take a picture of an item outside the scroll display area. This makes it possible to grasp all items on the screen, and makes the screen shot more meaningful as a record of the results of automatic operations. Furthermore, the screen shot makes it possible to more efficiently investigate the cause of an error.

[0075] In the past, it was desirable to grasp the automatic operation status by taking a screen shot during RPA operation, but in some cases, it was not possible to capture all items in a normal screen shot, and it was not possible to confirm the items that one wanted to check. However, according to this embodiment, it is possible to capture all items related to the automatic operation target without omission, so that it is possible to reliably confirm the items that one wants to check from the screen shot. In addition, according to this embodiment, it can be used not only during RPA operation, but also when a user takes a screen shot of an application, so that it is possible to capture all screen items without omission.

[0076] 5. Other embodiments The present invention may be embodied in various different embodiments other than those described above within the scope of the technical concept set forth in the claims.

[0077] For example, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods.

[0078] In addition, the processing procedures, control procedures, specific names, registered data for each process, information including parameters such as search conditions, screen examples, and database configurations shown in this specification and drawings may be changed as desired unless otherwise specified.

[0079] Moreover, with regard to the screenshot photographing device 100, the components shown in the figures are merely conceptual functional elements, and the devices do not necessarily have to be physically configured as shown in the figures.

[0080] For example, the processing functions of the screen shot photographing device 100, particularly the processing functions performed by the control unit, may be realized in whole or in part by a CPU and a program interpreted and executed by the CPU, or may be realized as hardware using wired logic. The program is recorded in a non-transient computer-readable recording medium including programmed instructions for causing the information processing device to execute the processes described in this embodiment, and is mechanically read by the screen shot photographing device 100 as necessary. That is, a computer program for giving instructions to the CPU in cooperation with the OS and performing various processes is recorded in a storage unit such as a ROM or HDD (Hard Disk Drive). This computer program is executed by being loaded into a RAM, and cooperates with the CPU to form a control unit.

[0081] In addition, this computer program may be stored in an application program server connected to the screen shot photographing device 100 via any network, and it is also possible to download all or part of it as necessary.

[0082] In addition, the program for executing the process described in this embodiment may be stored in a non-transient computer-readable recording medium, or may be configured as a program product. Here, the "recording medium" includes any "portable physical medium" such as a memory card, a Universal Serial Bus (USB) memory, a Secure Digital (SD) card, a flexible disk, a magneto-optical disk, a ROM, an Erasable Programmable Read Only Memory (EPROM), an Electrically Erasable and Programmable Read Only Memory (EEPROM (registered trademark)), a Compact Disk Read Only Memory (CD-ROM), a Magneto-Optical disk (MO), a Digital Versatile Disk (DVD), and a Blu-ray (registered trademark) Disc.

[0083] Moreover, a "program" is a data processing method written in any language or description method, and may be in any form, such as source code or binary code. Note that a "program" is not necessarily limited to a single configuration, but also includes a distributed configuration as multiple modules or libraries, and a program that works with a separate program, such as an OS, to achieve its function. Note that the specific configuration and reading procedure for reading a recording medium in each device shown in the embodiments, as well as the installation procedure after reading, may use well-known configurations and procedures.

[0084] The various databases etc. stored in the memory unit are storage means such as memory devices such as RAM and ROM, fixed disk devices such as hard disks, flexible disks, and optical disks, and store various programs, tables, databases, and web page files etc. used for various processes and providing websites.

[0085] The screen shot photographing device 100 may be configured as an information processing device such as a known personal computer or a workstation, or may be configured as the information processing device to which any peripheral device is connected. The screen shot photographing device 100 may be realized by installing software (including programs or data, etc.) that causes the device to realize the processing described in this embodiment.

[0086] Furthermore, the specific form of distribution and integration of the devices is not limited to that shown in the drawings, and all or part of them can be functionally or physically distributed and integrated in any unit according to various additions or functional loads. In other words, the above-mentioned embodiments can be implemented in any combination, or the embodiments can be implemented selectively. [Industrial Applicability]

[0087] The present invention is particularly useful in companies that have adopted automated operations of their business systems using RPA tools. [Explanation of symbols]

[0088] 100 Screenshot camera 102 Control section 102a Execution identifier acquisition unit 102b Screen identifier acquisition unit 102c Photography Department 102d Storage Execution Department 104 Communication interface section 106 Storage section 106a Master Data 106b Screenshot storage section 108 Input / Output Interface Section 112 Input Devices 114 Output Device 200 Servers 300 Network

Claims

1. A screen shot photographing device including a control unit that photographs all of a plurality of screens provided by a target program that is an application program to be operated, the screen shot photographing device comprising: The control unit is an execution identifier acquisition means for acquiring an execution identifier of a target program being operated; a screen identifier acquiring means for acquiring screen identifiers of all screens provided by a target program identified by the execution identifier acquired by the execution identifier acquiring means; a photographing means for photographing a screen shot of a screen identified by each of all the screen identifiers acquired by the screen identifier acquiring means; a storage and execution means for storing all screen shots taken by the image taking means in a storage unit so that a person can later check the error occurrence state in a state where the error occurrence state can be understood in the context of the screen, and for leaving a record of the automatic operation result; The storage execution means stores the screen shots by assigning numbers to the screen shots indicating the order in which the screen shots were taken; the execution identifier is a process of the target program, the screen identifier is a window handle for the screen; said photographing means designates a window handle of a screen to be photographed, and photographs screen shots of each layer, starting from the bottommost screen, without overlapping with other screens; A screen shot taking device comprising:

2. Master data in which a screen identifier and one or more control identifiers for vertical or horizontal scrolling and / or a control identifier for page movement are associated with each other and stored is accessible; the imaging means executes the following processes (1) to (3) for all screen identifiers acquired by the screen identifier acquisition means; (1) From the master data, obtain the control identifier associated with the screen identifier: (2) If the control identifier cannot be obtained, take a screenshot of the screen identified by the screen identifier: (3) If the control identifiers are obtained, (31) obtain scroll information or page movement information from each control identifier, (32) set the scroll and / or page movement state of the screen identified by the screen identifier to an initial state according to each piece of obtained information, and take screen shots, and (33) take screen shots while performing a scroll and / or page movement process according to each piece of obtained information on the screen identified by the screen identifier, until the process can no longer be performed.

2. The screen shot photographing device according to claim 1,

3. The photographing means photographs a screen shot while performing each process in the process of (33) in the order of a scroll process and a page move process; 3. The screen shot photographing device according to claim 2,

4. The photographing means, in the process of (33), executes the scroll process in the order of vertical scroll and horizontal scroll; 4. The screen shot photographing device according to claim 3,

5. The scroll process uses the size of the scroll box as a unit of movement of the scroll box; 5. The screen shot photographing device according to claim 2, wherein:

6. the save execution means saves the screen shots arranged on one sheet based on scroll information and / or page movement information saved when the screen shots were taken; 6. The screen shot photographing device according to claim 2, wherein:

7. The application program is a business application program; 7. The screen shot photographing device according to claim 1,

8. The operation is an automatic operation by a Robotic Process Automation (RPA) tool or an operation by a user to take a screen shot; 8. The screen shot photographing device according to claim 1, wherein:

9. A screen shot capturing method for capturing screen shots of all of a plurality of screens provided by a target program, which is an application program to be operated and executed in an information processing device having a control unit, comprising: Executed by the control unit, obtaining an execution identifier of a target program during which an operation is being performed; acquiring screen identifiers of all screens provided by the target program identified by the acquired execution identifier; taking, by an image taking means, screen shots of the screens identified by each of all the acquired screen identifiers; a storage step of storing all the screen shots taken by the photographing means in a storage unit so that a person can later confirm the error occurrence situation in a state where the error occurrence situation can be understood in the context of the screen, and keeping a record of the automatic operation result; In the saving step, the screen shots are saved by assigning numbers to the screen shots indicating the order in which the screen shots were taken; the execution identifier is a process of the target program, the screen identifier is a window handle for the screen; In the step of taking the images, a window handle of a screen to be photographed is designated, and a screen shot of each layer that does not overlap with other screens is taken in order from the bottom screen; A method for taking a screenshot, comprising:

10. A program for taking screenshots of all of a plurality of screens provided by a target program, which is an application program to be operated, to be executed by an information processing device having a control unit, To cause the control unit to execute obtaining an execution identifier of a target program during which an operation is being performed; acquiring screen identifiers of all screens provided by the target program identified by the acquired execution identifier; taking, by an image taking means, screen shots of the screens identified by each of all the acquired screen identifiers; a storage step of storing all the screen shots taken by the photographing means in a storage unit so that a person can later confirm the error occurrence situation in a state where the error occurrence situation can be understood in the context of the screen, and keeping a record of the automatic operation result; In the saving step, the screen shots are saved by assigning numbers to the screen shots indicating the order in which the screen shots were taken; the execution identifier is a process of the target program, the screen identifier is a window handle for the screen; In the step of taking the images, a window handle of a screen to be photographed is designated, and a screen shot of each layer that does not overlap with other screens is taken in order from the bottom screen; A program characterized by.

Images