Information processing device and method

Abstract

An information processing device according to one embodiment of the present invention includes: a generation unit configured so that when information is input to a region in a display screen including the region which is for receiving information input, a sentence describing information to be inputted to the region is generated on the basis of the inputted information; and a display processing unit that acquires coordinates of the region in which the information is inputted on the display screen, displays the generated sentence on the display screen, and causes the display screen to display information indicating the relation between the region related to the acquired coordinates and the generated sentence.

Description

Information processing device and method

[0001] FIELD Embodiments of the present invention relate to an information processing apparatus and method.

[0002] A typical procedure manual or operation manual for an operation screen represented by a GUI (Graphical User Interface) component often consists of explanatory text that explains each operation step and captured images of the actual operation screen. Therefore, creating such a manual manually requires a lot of effort.

[0003] Also, for example, Non-Patent Document 1 discloses an example of a manual creation support tool.

[0004] NotePM, <https: / / notepm.jp>

[0005] However, the position of the GUI component to be operated in the captured image and the explanatory text for the operation of the GUI component must be manually associated by a person, that is, a user.

[0006] This invention has been made in light of the above circumstances, and its purpose is to provide an information processing device and method that can appropriately associate an operation location on a screen with an explanatory text related to that operation location.

[0007] An information processing device according to one aspect of the present invention comprises a generation unit that, when information is input into an area of ​​a display screen that includes an area for accepting information input, generates a sentence describing the information that is to be input into the area based on the input information, and a display processing unit that acquires the coordinates of the area on the display screen into which the information was input, displays the generated sentence on the display screen, and displays information on the display screen indicating the relationship between the area related to the acquired coordinates and the generated sentence.

[0008] An information processing method according to one aspect of the present invention is a method performed by an information processing device, and includes the steps of: generating, by a generation unit of the information processing device, a sentence describing the information to be input into an area on a display screen including an area that accepts information input, based on the input information when the information is input into the area; and acquiring, by a display processing unit of the information processing device, the coordinates of the area on the display screen where the information was input, displaying the generated sentence on the display screen, and displaying information on the display screen indicating the relationship between the area related to the acquired coordinates and the generated sentence.

[0009] According to the present invention, an operation location on a screen can be appropriately associated with an explanatory text relating to this operation location.

[0010] Fig. 1 is a diagram showing an application example of an information processing device according to an embodiment of the present invention. Fig. 2 is a diagram showing an example of association between an operation location in a captured image and explanatory text. Fig. 3 is a diagram showing an example of correction of coordinates accompanying scrolling of a screen. Fig. 4 is a flowchart showing an example of a procedure of processing operations by an information processing device according to an embodiment of the present invention. Fig. 5 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention.

[0011] An embodiment of the present invention will be described below. FIG. 1 is a diagram showing an application example of an information processing device according to an embodiment of the present invention. As shown in FIG. 1, the information processing device 100 according to this embodiment includes an operation log storage unit 10, an operation log acquisition unit 20, a capture image acquisition unit 30, an explanatory text generation unit 40, a capture / operation location association unit 50, and a screen generation / display unit 60. The operation log storage unit 10 has a storage device such as a non-volatile memory. The processing content of each unit will be described later.

[0012] The business system shown in Figure 1 is provided to multiple users as one or more systems that can be used in common via user terminals. Operation logs are historical information on operation units that are generated in chronological order and can be acquired for each operation terminal. A chronological operation log acquired for a specific unit, such as for each task, is called an operation process.

[0013] This embodiment relates to a method for displaying an automatically generated business manual, and in particular to associating operation locations on a captured screen with explanatory text for each operation. This embodiment can be realized by an application program, and may be realized as an apparatus or a computer program. The terminal on which this is realized may be an operating terminal on an operator's computer, a server on a cloud, or the like, and is not particularly limited.

[0014] FIG. 2 is a diagram showing an example of the association between operation locations in a captured image and explanatory text. The information processing device 100 according to this embodiment automatically generates a format in which a captured image of an operation screen and explanatory text for each operation are arranged on the same screen, similar to a manually created business manual, and displays the format as shown in FIG. 2. The information processing device 100 also automatically associates each explanatory text with the location of the operation in the captured image. In the example shown in FIG. 2, a numbered speech bubble is attached to the position of the GUI component to be operated on the left side of the screen, and the text associated with the same numbered speech bubble on the right side of the screen is explanatory text related to the operation of the corresponding GUI component.

[0015] To generate the above format, it is necessary to associate the XY coordinates of the corresponding operation with the XY coordinates of the capture screen, and this association is realized by recording the coordinates of the GUI parts when recording the operation log.

[0016] 3 is a diagram showing an example of coordinate correction associated with screen scrolling. When screen scrolling or the like requires conversion between the coordinates of a target GUI component in a captured image and the coordinates recorded in the operation log, the screen scroll state for each operation, for example, "scroll 0 px" before scrolling and "scroll 120 px" after scrolling as shown in FIG. 3, is recorded, and the conversion is performed by correcting the coordinates in the operation log based on this recorded state. The symbols a and b in FIG. 3 indicate GUI components whose positions have been identified by the correction.

[0017] Regarding the arrangement of the captured image and explanatory text, for example, the explanatory text and the captured image may be arranged on the left and right as in the example of FIG. 2, or the explanatory text may be overlaid on the captured image.

[0018] The information processing device 100 generates explanatory text from the coordinates of the GUI components and the operation log, such as the input value, label value, type attribute value, and captured image, acquired when the operation log was acquired. This explanatory text can be generated using, for example, the method disclosed in "Proposal of a Method for Adding Annotations to RPA Scenarios" (The Institute of Electronics, Information and Communication Engineers).

[0019] In this embodiment, by using the operation log mechanically recorded by the operation log acquisition tool, a document can be automatically generated that makes it easy for the user to understand the operation content recorded in the operation log, such as a procedure manual or business manual, without the user having to perform any special operations.

[0020] Furthermore, in this embodiment, the operation position in the captured image of the target GUI component is automatically associated with an explanatory text for the operation of the target GUI component, which can reduce the amount of manual creation work, for example.

[0021] 4 is a flowchart showing an example of a processing operation procedure by an information processing device according to an embodiment of the present invention. The operation contents of the operation screen by the user terminal and a captured image of the operation screen are stored as an operation log in the operation log storage unit 10. The operation contents of the operation screen include the coordinate position of the operated GUI component, the scroll position, etc.

[0022] The operation log acquisition unit 20 acquires the operation log stored in the operation log storage unit 10 as information necessary for generating the explanatory text, and records the captured image, coordinate positions of GUI components, scroll position, etc. included in the operation log in an internal memory (S10). The captured image of the operation screen in the operation log is acquired from the operation log by the captured image acquisition unit 30 and returned to the operation log acquisition unit 20 (S20).

[0023] The explanatory text generation unit 40 generates explanatory text using the acquired operation log results and passes it to the capture / operation location association unit 50 (S30). The capture / operation location association unit 50 associates the captured image with the operation location on the operation screen, and passes association information indicating this association to the screen generation / display unit 60 (S40).

[0024] The screen generation / display unit 60 displays the capture image and explanatory text, as well as a user interface that associates them, based on the capture image acquired by the capture image acquisition unit 30, the explanatory text generated by the explanatory text generation unit 40, and the association information passed from the capture / operation location association unit 50 (S50).

[0025] 5 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention. In the example shown in FIG. 5, the information processing device 100 according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0026] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0027] An input device 200 and an output device 300 attached to the information processing device 100 and used by a user or the like are connected to the input / output interface 113. The input / output interface 113 can take in operation data input by a user or the like through the input device 200 such as a keyboard, touch panel, touchpad, or mouse, and can output output data to an output device 300 including a display device using a liquid crystal or organic electroluminescence (EL) display, etc., for display. The input device 200 and the output device 300 may be devices built into the information processing device 100, or may be input devices and output devices of other information terminals that can communicate with the information processing device 100 via a network.

[0028] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and can store programs necessary to execute various control processes, etc., according to one embodiment.

[0029] The data memory 112 is a tangible storage medium that is, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM (Random Access Memory), and can be used to store various data or information acquired and created during various processes.

[0030] An information processing apparatus 100 according to an embodiment of the present invention can be configured as a data processing apparatus having the units shown in FIG. 1 as software-based processing function units.

[0031] The information storage unit used as a work memory or the like by each unit of the information processing device 100 can be configured by using the data memory 112 shown in Fig. 5. However, these configured storage areas are not essential components within the information processing device 100, and may be areas provided in, for example, an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0032] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0033] The methods described in each embodiment can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), or semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables and data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes storage media such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0034] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0035] REFERENCE SIGNS LIST 100... Information processing device 10... Operation log storage unit 20... Operation log acquisition unit 30... Capture image acquisition unit 40... Explanation sentence generation unit 50... Capture / operation location association unit 60... Screen generation / display unit

Claims

1. An information processing device comprising: a generation unit that generates a sentence explaining the information to be input into a display screen area that includes an area for accepting information input, based on the input information, when information is input into the area; and a display processing unit that acquires the coordinates of the area on the display screen where the information has been input, displays the generated sentence on the display screen, and displays information on the display screen indicating the relationship between the area related to the acquired coordinates and the generated sentence.

2. The information processing device of claim 1, wherein the display screen is a screen on which images of graphical user interface components are displayed, the generation unit generates a sentence explaining the information to be input into the area based on the input information when an operation related to inputting information is performed on the display screen to a component that accepts the input of the information, and the display processing unit acquires the coordinates of the component on the display screen on which the operation related to inputting the information is performed, and displays information indicating the relationship between the component related to the acquired coordinates and the generated sentence at the location of the component related to the acquired coordinates on the display screen.

3. A method performed by an information processing device, comprising: a generation unit of the information processing device, when information is input into an area on a display screen including an area that accepts information input, generating a sentence explaining the information that is to be input into the area based on the input information; and a display processing unit of the information processing device, acquiring the coordinates of the area on the display screen into which the information was input, displaying the generated sentence on the display screen, and displaying information on the display screen indicating the relationship between the area related to the acquired coordinates and the generated sentence.

Images