Environment construction device and program

Abstract

An environment construction device (130) according to an embodiment comprises: a screen extraction unit (131) that extracts a common region from captured images for each operation using an operation log acquired from an external device, and deletes the extracted regions from all of the captured images to generate a system screen for each operation; a screen generation unit (132) that combines two system screens using, as a reference, a common region of consecutive system screens among the system screens for each operation including the initial system screen in the initial state before operation, to thereby generate an overall system screen; an element extraction unit (133) that uses the operation location coordinates on the system screen for one operation to compare at least two system screens for which operations are consecutively performed, among the system screens for each operation including the initial system screen, to thereby determine if there is an input element or a temporary display element depending on whether information remains at the operation location coordinates; an element setting unit (134) that sets a reference by calculating the position coordinates of the system screen for one operation in an overall system screen, and calculates the operation location coordinates on the overall system screen; and an environment construction unit (135) that combines an input element displayed at the operation location with the overall system screen on the basis of the operation location coordinates on the overall system screen.

Description

Environment construction equipment and programs

[0001] The present invention relates to an environment construction device and a program.

[0002] In order to reproduce operations on a business system using machine learning, it is necessary to build a GUI stub as a simulation environment. Conventional techniques include creating a GUI stub based on the operations performed by a worker.

[0003] This is a method of automatically creating a GUI stub that reproduces the screen transitions in a business system by recording, as information representing the operation content, GUI structural information, screen image information, and operation content as an operation log when an operator operates a terminal, since structural information based on the hierarchical structure of GUI components (buttons, etc.) that the software has can be obtained from various system libraries, and by creating a GUI state transition graph from the accumulated operation log.

[0004] Hideki Koya, Hajime Nakajima, Hiroki Nakanishi, and Ikuya Takahashi, "A Study on Automatic GUI Stub Creation Method Using Operation Logs to Support the Creation of Control Programs for Automatic Software Operation," B-14. Information and Communication Management, General Session, Proceedings of the Society Conference of the Institute of Electronics, Information and Communication Engineers, 2013

[0005] Japanese Patent No. 6052892

[0006] However, in conventional technology, since information is acquired using an API, it is difficult to create a simulation environment in an environment such as a thin client environment where only screen captures and mouse / keyboard operation information can be acquired.

[0007] The present invention has been made in consideration of the above circumstances, and has as its object to provide an environment construction device and a program that are capable of creating a simulation environment in an environment where only operation information can be obtained.

[0008] An environment construction device according to a first aspect of the present invention includes a screen extraction unit that extracts common areas from capture images for each operation using an operation log acquired from an external device and generates a system screen for each operation by deleting the extracted areas from all of the capture images; a screen generation unit that generates an entire system screen by combining two system screens on which operations are performed successively in system screens for each operation, including an initial system screen in an initial state before the operation, using a common area of ​​system screens on which operations are performed successively as a reference; an element extraction unit that uses position coordinates of an operation point on a system screen for one operation to compare at least two system screens on which operations are performed successively in the system screen for each operation, including the initial system screen, and determines that the operation point is an input element or a temporary display element depending on whether information remains at the position coordinates of the operation point; an element setting unit that sets a reference by calculating the position coordinates of the system screen for one operation on the entire system screen and calculates the position coordinates of the operation point on the entire system screen; and an environment construction unit that combines the input element displayed at the operation point into the entire system screen based on the position coordinates of the operation point on the entire system screen.

[0009] A program according to a second aspect of the present invention is a program that causes a computer to function as the environment construction device according to the first aspect.

[0010] According to the present invention, it is possible to provide an environment construction device and a program that are capable of creating a simulation environment in an environment where only operation information can be obtained.

[0011] FIG. 1 is a block diagram illustrating an example of a configuration of an environment creation system according to an embodiment. FIG. 2 is a block diagram illustrating an example of a functional configuration of an environment creation device according to an embodiment. FIG. 3 is a diagram illustrating an example of a captured image included in an operation log. FIG. 4 is a diagram illustrating another example of a captured image included in an operation log. FIG. 5 is a diagram illustrating another example of a captured image included in an operation log. FIG. 6 is a diagram illustrating an example of an entire captured image included in an operation log. FIG. 7 is a diagram illustrating an example of processing operations by a system screen extraction unit and an entire system screen generation unit according to an embodiment. FIG. 8 is a diagram illustrating an example of processing operations by an operation element extraction unit, an operation element setting unit, and an environment creation unit according to an embodiment. FIG. 9 is a flowchart illustrating an example of system screen extraction processing by a system screen extraction unit according to an embodiment. FIG. 10 is a flowchart illustrating an example of entire system screen generation processing by an entire system screen generation unit according to an embodiment. FIG. 11 is a flowchart illustrating an example of operation element extraction processing by an operation element extraction unit according to an embodiment. FIG. 12 is a flowchart illustrating an example of operation element setting processing by an operation element setting unit according to an embodiment. FIG. 13 is a block diagram illustrating an example of a hardware configuration of a business system simulation environment creation device according to an embodiment.

[0012] Each embodiment will be described below with reference to the drawings. Each embodiment illustrates an apparatus or method for embodying the technical idea of ​​the invention. The drawings are schematic or conceptual. In the following, the same reference numerals are used to designate components having substantially the same functions and configurations. The numbers following the letters that make up the reference numerals are used to distinguish between elements that are referred to by the reference numerals containing the same letters and that have similar configurations. When there is no need to distinguish between elements indicated by reference numerals containing the same letters or numbers, these elements will be referred to by reference numerals containing only letters or numbers.

[0013] 1 is a block diagram illustrating a schematic configuration example of an environment creation system according to an embodiment. The environment creation system 10 includes a business system 100, an information terminal 110, an operation log acquisition unit 120, a business system simulation environment creation device 130, and an output unit 140. The operation log acquisition unit 120 and the output unit 140 may be included in the business system simulation environment creation device 130. The business system simulation environment illustrated in FIG. 1 is the name of the environment created by the business system simulation environment creation device 130, and is output (displayed) via the output unit 140.

[0014] The business system 100 is provided as, for example, system software that can be commonly used by multiple users, and at least one of the business systems 100 exists in the environment construction system 10. At least one of the business systems 100 is installed in the information terminal 110, and is used by the operator of the information terminal 110.

[0015] The information terminal 110 may be an electronic device such as a computer, and may be, for example, a television set (including an Internet television), a personal computer (PC), a mobile terminal (for example, a tablet, a smartphone, a laptop, a feature phone, a digital music player, an e-book reader, a smartwatch, etc.), a game device (a home game console, a portable game console), a virtual reality (VR) terminal, an augmented reality (AR) terminal, etc., but is not limited to these. Note that the number of information terminals 110 using the business system 100 may be zero, or may increase or decrease at any time.

[0016] The information terminal 110 and the business system simulation environment creation device 130 may be communicably connected to an external server or the like via a network, for example. In this case, multiple information terminals 110 and business system simulation environment creation devices 130 are communicably connected to the external server via the network, enabling functions such as sharing of necessary information. The external server may be a storage device or the like.

[0017] The operation log acquisition unit 120 acquires operation information when the operator of the information terminal 110 operates the screen. This information includes history information for each operation unit corresponding to the user's terminal operation, such as mouse and keyboard operations, operations on the active window, and operations on GUI components on the screen, as the operation log 111. The operation log acquisition unit 120 may also acquire the operation log 111 as an operation process organized into specific units for each task. The timing at which the operation log acquisition unit 120 acquires the operation log 111 from the information terminal 110 may be, for example, by the information terminal 110 transmitting the operation log 111 to the operation log acquisition unit 120 based on a request signal from the operation log acquisition unit 120, or by the information terminal 110 transmitting the operation log 111 at predetermined time intervals. The operation log acquisition unit 120 may also continuously acquire the operation log 111 while the operator is logged in to the information terminal 110 or while the business system 100 is running.

[0018] The output unit 140 may include, for example, a display means such as a monitor that allows the business system simulation environment, which is the processing result of the business system simulation environment creation device 130, to be visually recognized, or an audio output means that allows the business system simulation environment, which is the processing result of the business system simulation environment creation device 130, to be audibly recognized using a speaker or the like.

[0019] The business system simulation environment creation device 130 includes a system screen extraction unit 131, an entire system screen generation unit 132, an operation element extraction unit 133, an operation element setting unit 134, an environment creation unit 135, and an operation location identification unit 136. As described above, the business system simulation environment creation device 130 may also include an operation log acquisition unit 120 and an output unit 140. The business system simulation environment creation device 130 is an example of an environment creation unit.

[0020] The system screen extraction unit 131 has a function of using multiple capture images, obtained by taking screenshots or the like of the screen of the information terminal 110 included in the operation log 111 for each operation acquired by the operation log acquisition unit 120, to determine from the capture images areas that are not related to the operation of the business system 100 (hereinafter referred to as system operation), and extract only the parts related to the system operation (hereinafter referred to as system screen). Note that the system screen extraction unit 131 is an example of a screen extraction unit.

[0021] Here, the captured images included in the operation log 111 will be described. Fig. 3 is a diagram showing an example of a captured image included in the operation log. Fig. 4 is a diagram showing another example of a captured image included in the operation log. Fig. 5 is a diagram showing another example of a captured image included in the operation log. Fig. 6 is a diagram showing an example of an entire captured image included in the operation log.

[0022] The entire system screen generating unit 132 has a function of identifying the matching parts of the extracted multiple system screens, synthesizing the multiple system screens based on the matching parts, and generating a screen image (hereinafter referred to as the entire system screen) that represents the entire area related to system operation. The entire system screen generating unit 132 is an example of a screen generating unit.

[0023] 3, 4, and 5, when the entire screen required for the system is larger than the window size that can be displayed on the output device of the information terminal 110, the operation log for each operation includes a capture image 200, a capture image 300, and a capture image 400. If the capture image 200 is the initial screen, the capture image 300 is an image obtained by scrolling a certain area from the capture image 200, and the capture image 400 is an image obtained by scrolling a certain area from the capture image 300.

[0024] 6, when the window size that can be displayed on the output device of information terminal 110 is equal to or smaller than the entire screen required for the system, the operation log includes a capture image 500. Note that when the capture image included in the operation log is capture image 500, the process of combining multiple system screens by entire system screen generation unit 132 does not need to be executed.

[0025] 7 is a diagram showing an example of processing operations by the system screen extraction unit and the entire system screen generation unit according to the embodiment. As shown in Fig. 7, the system screen extraction unit 131 generates system screens 201, 301, and 401 by extracting areas other than the same area from captured images 200, 300, and 400 included in the operation log 111 acquired from the operation log acquisition unit 120. Note that the system screens 201, 301, and 401 are examples of initial system screens.

[0026] The entire system screen generation unit 132 identifies the matching parts between the system screen 201 and the system screen 301, and combines them by superimposing the matching parts of the system screen 201 and the system screen 301. It also identifies the matching parts between the screen obtained by combining the system screen 201 and the system screen 301 and the system screen 401, and generates the entire system screen 501 by superimposing the matching parts of the screen obtained by combining the system screen 201 and the system screen 301 and the system screen 401.

[0027] The operation location identification unit 136 has a function of identifying an operation location (GUI component) such as a text box or a pull-down, the operation content, and the position coordinates on the captured image corresponding to the operation location, based on the information included in the operation log 111. For example, the operation location identification unit 136 compares a plurality of captured images to identify an area where a difference occurs in the captured images and the operation location and the operation content using the difference characteristics of the area.

[0028] Furthermore, the operation location identification unit 136 identifies position coordinates on the captured image corresponding to the operation location using information on cursor coordinates at the time of the operation. The operation location identification unit 136 identifies the operation location corresponding to the captured image in response to a request from the operation element extraction unit 133, and transmits the identified information to the operation element extraction unit 133. The method of identifying the operation location and position information by the operation location identification unit 136 is not limited to this, and for example, the operation location may be identified using a preset reference position, reference image, or the like. Note that the operation location identification unit 136 is an example of an identification unit.

[0029] The operation element extraction unit 133 has a function of comparing a captured image before an operation with a captured image after the operation or a system screen before an operation with a system screen after the operation, and determining whether an element is an input element or a temporary display element depending on whether any value has been input into the captured image after the operation or the system screen after the operation. Note that the operation element extraction unit 133 is an example of an element extraction unit.

[0030] A temporary display element is a display of annotations required when the operator of the information terminal 110 performs an operation, or an option displayed by a pull-down menu. For example, if the operation location operated by the operator of the information terminal 110 is a text box, the input value is generally displayed in the operation location after the operation, and the operation element extraction unit 133 determines this value to be the input element. Also, if the operation location is a pull-down menu, the pull-down menu disappears after the operation, and only the value selected by the operator from the pull-down menu is displayed. In this case, the pull-down portion is determined to be the temporary display element, and the portion displaying the selected value is determined to be the input element.

[0031] The operation element setting unit 134 has a function of comparing the entire system screen generated by the entire system screen generation unit 132 with a captured image, calculating which part of the entire system screen the captured image corresponds to, thereby setting a reference, and calculating position information of the operation point on the entire system screen from position information of the operation point on the captured image. Note that the operation element setting unit 134 may also set a reference by comparing the entire system screen with the system screen, calculating which part of the entire system screen the system screen corresponds to, and calculating position information of the operation point on the entire system screen from position information of the operation point on the system screen. Note that the operation element setting unit 134 is an example of an element setting unit.

[0032] The environment construction unit 135 has a function of synthesizing input elements displayed at operation locations on the entire system screen based on the position information of the operation locations calculated by the operation element setting unit 134. The environment construction unit 135 repeatedly executes synthesizing of input elements displayed at operation locations on the entire system screen based on the position information of the operation locations calculated by the operation element setting unit 134, while maintaining the display of the synthesized input elements. Furthermore, the environment construction unit 135 switches between displaying and hiding temporary display elements such as pull-down menus depending on the operation status.

[0033] 8 is a diagram showing an example of processing operations by the operation element extraction unit, operation element setting unit, and environment construction unit according to the embodiment. The operation element extraction unit 133 acquires a capture image included in an operation log corresponding to any one operation, and requests the operation location identification unit 136 to identify an operation location in the capture image. The operation location identification unit 136 identifies an operation location (GUI component) such as a text box or pull-down, the operation content, and the position coordinates on the capture image corresponding to the operation location for the requested capture image, and feeds this information back to the operation element extraction unit 133.

[0034] The operation element extraction unit 133 compares a capture image corresponding to any one operation with a capture image corresponding to the operation immediately before the arbitrary operation to determine whether it is an input element or a temporary display element, and transmits information including the determination result to the operation element setting unit 134.

[0035] The operation element setting unit 134 acquires a capture image included in the operation log corresponding to any one operation and acquires position information of the capture image using the entire system screen. That is, the operation element setting unit 134 identifies the position of the bold frame (operation location) in the capture image shown in FIG. 8 on the entire system screen. The operation element setting unit 134 calculates position information of the bold frame on the entire system screen by identifying which part of the entire system screen the bold frame corresponds to. The environment construction unit 135 combines input elements within the bold frame based on the position information of the bold frame, as shown in FIG. 8.

[0036] Next, an example of the processing operation of the system screen extraction unit in this embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of system screen extraction processing by the system screen extraction unit according to this embodiment.

[0037] In this embodiment, for example, the business system simulation environment creation device 130 is started when the operator of the information terminal 110 starts the business system 100, and the operation log acquisition unit 120 acquires the operation log 111 from the information terminal 110, causing the system screen extraction unit 131 to start processing operations.

[0038] The system screen extraction unit 131 acquires the operation log 111 from the operation log acquisition unit 120 (step S10). The system screen extraction unit 131 acquires multiple operation logs 111 related to one operation acquired by the operation log acquisition unit 120. The system screen extraction unit 131 reads a capture image related to one operation from the operation log 111 (step S11). The system screen extraction unit 131 reads each of the capture images included in the multiple acquired operation logs.

[0039] The system screen extraction unit 131 extracts areas that are displayed in common across the multiple captured images that have been read (step S12). The system screen extraction unit 131 determines whether areas that are displayed in common across all operations have been extracted (step S13). The system screen extraction unit 131 repeats the process of step S12 until areas that are displayed in common across all operations, that is, across all captured images included in the acquired operation log 111, have been extracted. If the system screen extraction unit 131 determines that areas that are displayed in common across all captured images included in the acquired operation log 111 have been extracted (step S13, YES), it proceeds to the next process.

[0040] The system screen extraction unit 131 deletes from all captured images any extracted areas that are equal to or greater than a set value (operation content similarity threshold) (step S14). The system screen extraction unit 131 generates a system screen by deleting from the captured images any areas that are equal to or greater than the operation content similarity threshold. The operation content similarity threshold, which is a set value, is set in advance by the operator of the business system simulation environment creation device 130 and can be changed whenever necessary.

[0041] The system screen extraction unit 131 updates the position coordinates of the operation location from the deleted area (step S15), and ends the process. That is, the system screen extraction unit 131 updates the position coordinates of the operation location by changing the reference coordinates that determine the position coordinates of the operation location.

[0042] Next, an example of the processing operation of the overall system screen generating unit in this embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the overall system screen generating process by the overall system screen generating unit according to this embodiment.

[0043] In this embodiment, for example, the entire system screen generation unit 132 starts a processing operation when the system screen extraction unit 131 generates a system screen. The entire system screen generation unit 132 acquires the operation log 111 including the system screen from the system screen extraction unit 131 (step S20).

[0044] The overall system screen generating unit 132 reads a system screen relating to the first operation in a series of operations and sets it as the overall system screen (step S21). The overall system screen generating unit 132 reads a system screen relating to the next operation from the operation log 111 (step S22). The overall system screen generating unit 132 compares the system screen read in S22 with the overall system screen set in S21 and combines the system screen and the overall system screen based on the common parts (step S23).

[0045] The overall system screen generating unit 132 determines whether images have been synthesized based on the common parts for all operations (step S24). The overall system screen generating unit 132 compares all system screens included in the acquired operation log 111 with the overall system screen and repeatedly executes a process of synthesizing the system screen and the overall system screen based on the common parts. When the overall system screen generating unit 132 determines that images have been synthesized based on the common parts for all system screens included in the acquired operation log 111 (step S24, YES), the process ends.

[0046] Next, an example of the processing operation of the operation element extraction unit in this embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of operation element extraction processing by the operation element extraction unit according to this embodiment.

[0047] In this embodiment, for example, the generation of the entire system screen by the entire system screen generation unit 132 is used as a trigger to start the processing operation of the operation element extraction unit 133. The operation element extraction unit 133 acquires the system screen and the operation log 111 including the entire system screen from the entire system screen generation unit 132 (step S30).

[0048] The operation element extraction unit 133 reads a system screen related to the first operation in a series of operations and sets it as the first system screen (step S31). The operation element extraction unit 133 reads a system screen related to the next operation from the operation log 111 and sets it as the second system screen (step S32). The operation element extraction unit 133 compares the first system screen with the second system screen (step S33).

[0049] The operation element extraction unit 133 compares the first system screen with the second system screen to determine whether a value or the like remains on the second system screen at a corresponding operation location on the first system screen (step S34). If the operation element extraction unit 133 determines that a value or the like remains on the second system screen at a corresponding operation location on the first system screen (step S34, YES), it determines that the area of ​​the operation location on the first system screen is an input element (step S35). If the operation element extraction unit 133 determines that a value or the like does not remain on the second system screen at a corresponding operation location on the first system screen (step S34, NO), it determines that the area of ​​the operation location on the first system screen is a temporarily displayed element (step S36).

[0050] The operation element extraction unit 133 reads a system screen related to an operation subsequent to the operation read as the second system screen from the operation log 111 and sets it as the second system screen, thereby repeatedly executing the processes from step S33 to step S37. When the operation element extraction unit 133 determines that it has identified the input elements or temporary display elements at the operation locations for all system screens included in the operation log 111 (YES in step S37), it ends the process.

[0051] Next, an example of the processing operation of the operation element setting unit in this embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of operation element setting processing by the operation element setting unit according to this embodiment.

[0052] In this embodiment, the operation element setting unit 134 starts processing, for example, when the operation element extraction unit 133 determines an input element or a temporary display element corresponding to an area of ​​an operation location on an arbitrary system screen. The operation element setting unit 134 acquires the operation log 111 including the system screen and the overall system screen from the operation element extraction unit 133 (step S40).

[0053] The operation element setting unit 134 reads a system screen related to one operation and operation content (information related to the operation location) associated with the system screen from the operation log 111 (step S41). The operation element setting unit 134 compares the read system screen with the entire system screen (step S42). The operation element setting unit 134 calculates the area (position coordinates) of the system screen on the entire system screen (step S43).

[0054] The operation element setting unit 134 calculates the position coordinates of the operation location on the entire system screen using the position coordinates of the system screen on the entire system screen and the position coordinates of the operation location on the system screen (step S44). The operation element setting unit 134 reads the system screen and operation content related to the next operation from the operation log 111 acquired from the operation element extraction unit 133, and repeatedly executes the processes from step S42 to step S45. When the operation element setting unit 134 determines that it has calculated the position information of the operation location on the entire system screen for all system screens included in the operation log 111 (step S45, YES), it ends the process.

[0055] Next, an example of the hardware configuration of the business system simulation environment creation device 130 according to this embodiment will be described with reference to Fig. 13. Fig. 13 is a block diagram showing an example of the hardware configuration of the business system simulation environment creation device according to this embodiment.

[0056] 13, the business system simulation environment creation device 130 includes, for example, a processor 1301, a data memory 1302, a program memory 1303, a communication interface 1304, and an input / output interface 1305. The processor 1301, the data memory 1302, the program memory 1303, the communication interface 1304, and the input / output interface 1305 are connected to each other via a bus communication line, and can transmit and receive data between them. Note that the processor 1301, the data memory 1302, the program memory 1303, the communication interface 1304, and the input / output interface 1305 may also be configured to transmit and receive data between them via wireless communication or the like.

[0057] The processor 1301 includes at least one processor such as a CPU (Central Process Unit), MPU (micro processing unit), GPU (Graphics Processing Unit), or FPGA (field-programmable gate array), and can realize various functions of the business system simulation environment construction device 130 by executing programs such as system software, application software, or firmware stored in the data memory 1302 or the program memory 1303.

[0058] The data memory 1302 may include, for example, a read-only memory (ROM) and a random-access memory (RAM). The ROM is a non-volatile memory used exclusively for reading data, and can store data and various setting values ​​used by the processor 1301 when performing various processes. The RAM can be used as a so-called work area for temporarily storing data when the processor 1301 performs various processes.

[0059] The program memory 1303 is a non-transitory computer-readable storage medium of a computer centered around the processor 1301. The auxiliary storage unit is, for example, an EEPROM (registered trademark) (electric erasable programmable read-only memory), a hard disk drive (HDD), or a solid state drive (SSD). The program memory 1303 can store data used by the processor 1301 (described later) when performing various processes, data generated by the processes in the processor 1301, various setting values, and the like.

[0060] The communication interface 1304 can transmit information received from the outside to the components within the business system simulation environment creation device 130, and can also transmit information received from the components within the business system simulation environment creation device 130 to the outside.

[0061] The input / output interface 1305 is installed in the business system simulation environment creation device 130, and is connected to an input device and an output device (e.g., the output unit 140) used by the operator of the business system simulation environment creation device 130. The input device is operated by the operator of the business system simulation environment creation device 130, and may include, for example, a user interface such as a mouse or keyboard for inputting various information (e.g., information for changing the numerical values ​​of various settings of the business system simulation environment creation device 130), and various sensors such as a microphone or a touch panel.

[0062] The business system simulation environment creation device 130 according to the embodiment collects screen captures that appear in the daily work of workers and identifies the positions of components on the system screen based on that information, making it possible to create a simulation environment that reproduces the entire business system using only information that can be obtained regardless of the application. Furthermore, by learning system operations using machine learning, it is possible to support business automation, etc. Furthermore, the business system simulation environment creation device 130 according to the embodiment reproduces the entire system screen, making it possible to refer to the entire system by scrolling and select GUI components to operate in any order. This reduces the gap between the simulation environment and the system environment in which the worker operates, making it possible to bring the simulation environment closer to the system environment. This makes it possible to use machine learning to learn system operations similar to those performed by the worker.

[0063] That is, according to this embodiment, it is possible to provide an environment construction device and a program that can create a simulation environment in an environment where only operation information can be acquired.

[0064] In the above embodiment, the system screen extraction unit 131, the entire system screen generation unit 132, the operation element extraction unit 133, the operation element setting unit 134, the environment construction unit 135, and the operation location identification unit 136 may be mounted on different devices. In this case, the multiple devices included in the business system simulation environment construction device 130 can share various information stored in the data memory 1302 and the program memory 1303.

[0065] The program according to the embodiment may be transferred in a state where it is stored in an electronic device, or may be transferred in a state where it is not stored in an electronic device. In the latter case, the program may be transferred via a network, or may be transferred in a state where it is stored in a storage medium. The storage medium is a non-transitory tangible medium. The storage medium is a computer-readable medium. The storage medium may be in any form, such as a CD-ROM or a memory card, as long as it is capable of storing the program and is computer-readable.

[0066] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0067] DESCRIPTION OF SYMBOLS 10... ENVIRONMENT CONSTRUCTION SYSTEM 100... BUSINESS SYSTEM 110... INFORMATION TERMINAL 111... OPERATION LOG 120... OPERATION LOG ACQUISITION UNIT 130... BUSINESS SYSTEM SIMULATION ENVIRONMENT CONSTRUCTION DEVICE 131... SYSTEM SCREEN EXTRACTION UNIT 132... ENTIRE SYSTEM SCREEN GENERATION UNIT 133... OPERATION ELEMENT EXTRACTION UNIT 134... OPERATION ELEMENT SETTING UNIT 135... ENVIRONMENT CONSTRUCTION UNIT 136... OPERATION POINT IDENTIFICATION UNIT 1301... PROCESSOR 1302... DATA MEMORY 1303... PROGRAM MEMORY 1304... COMMUNICATION INTERFACE 1305... I / O INTERFACE 140... OUTPUT UNIT

Claims

1. An environment construction device comprising: a screen extraction unit that extracts common areas from capture images for each operation using an operation log obtained from an external device and generates a system screen for each operation by deleting the extracted areas from all of the capture images; a screen generation unit that generates an entire system screen by combining two system screens by using the common areas of consecutive system screens in the system screen for each operation, including an initial system screen in the initial state before the operation, as a reference; an element extraction unit that uses the position coordinates of an operation point on the system screen for each operation to compare at least two system screens in which operations are consecutive in the system screen for each operation, including the initial system screen, and determines that the operation point is an input element or a temporary display element depending on whether information remains at the position coordinates of the operation point; an element setting unit that sets a reference by calculating the position coordinates of the system screen for each operation on the entire system screen and calculates the position coordinates of the operation point on the entire system screen; and an environment construction unit that combines input elements displayed at the operation point into the entire system screen based on the position coordinates of the operation point on the entire system screen.

2. The environment construction device according to claim 1, wherein the screen extraction unit deletes any extracted area that is equal to or larger than a set value from the captured image for each operation.

3. The environment construction device according to claim 1, further comprising an identification unit that identifies the operation location, operation content, and the operation location coordinates on the captured image corresponding to the operation location from the operation log.

4. A program that causes a computer to function as the environment construction device according to any one of claims 1 to 3.

Images