Information processing device and method

Abstract

An information processing device according to one embodiment of the present invention comprises: a first identification unit that identifies, on an input screen of information including a plurality of regions associated by a hierarchical structure, different regions associated, along the hierarchical structure, with an operation target region related to input of the information, and that identifies, from the identified regions, different regions along the upper direction and the left direction on the input screen; and a second identification unit that, when each of the identified regions is a region in which a character string on the input screen is displayed, identifies the character string as a candidate for label information of information that can be inputted to the operation target region.

Description

Information processing device and method

[0001] FIELD Embodiments of the present invention relate to an information processing apparatus and method.

[0002] Input screens expressed using a GUI (Graphical User Interface) or the like often have text labels clearly written near the input fields so that people, i.e., users, can understand the information that should be entered into each input field.

[0003] On the other hand, the internal structure of the GUI is expressed as IDs specifying objects, coordinates, and a tree structure between objects, such as an object tree, but does not have information about the association between labels and input fields.

[0004] For example, Patent Document 1 discloses a method in which a GUI is regarded as one element of an electronic document, and nine relative placement patterns are extracted for each label using each input field as the axis, consisting of top left, top, top right, left, same position, right, bottom left, bottom, and bottom right.The method evaluates whether this relative placement pattern is a pre-set, acceptable relative placement pattern, and if it is evaluated as an acceptable relative placement pattern, associates this input field with the label.

[0005] Japanese Patent Application Publication No. 2020-140410

[0006] In order to determine the association between a label and an input field, it is necessary to calculate all relative placement patterns with other GUI components for each input field, which significantly widens the search range and requires a lot of search time. For example, this method is not suitable for dynamic application programs where associations are made in real time in response to screen operations by a user and saved in a log.

[0007] The present invention has been made in light of the above circumstances, and its purpose is to provide an information processing device and method that can appropriately identify a label corresponding to an area on a display screen that is to be operated.

[0008] An information processing device according to one aspect of the present invention includes a first identification unit that identifies different areas associated with an area to be operated for inputting information in accordance with the hierarchical structure on an information input screen including a plurality of areas associated with the area in a hierarchical structure, and identifies different areas along an upward and leftward direction on the input screen from the identified areas, and a second identification unit that, when the identified area is an area on the input screen that displays a character string, identifies the character string as a candidate for label information of information that can be input into the area to be operated.

[0009] An information processing method according to one aspect of the present invention is a method performed by an information processing device, and includes: identifying, by a first identification unit of the information processing device, different areas associated in accordance with the hierarchical structure with respect to an area to be operated for inputting information on an information input screen including a plurality of areas associated by the hierarchical structure; identifying, from the identified areas, different areas along an upward and leftward direction on the input screen; and, by a second identification unit of the information processing device, when the identified area is an area to display a character string on the input screen, identifying the character string as a candidate for label information of information that can be input into the area to be operated.

[0010] According to the present invention, it is possible to appropriately identify a label corresponding to an area to be operated on the display screen.

[0011] Fig. 1 is a diagram showing an application example of an information processing device according to an embodiment of the present invention. Fig. 2 is a diagram showing an example of an input screen configured with GUI components. Fig. 3 is a diagram showing an example of a search range for labels of GUI components. Fig. 4 is a flowchart showing an example of a procedure for processing operations by an information processing device according to an embodiment of the present invention. Fig. 5 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention.

[0012] An embodiment of the present invention will be described below. Fig. 1 is a diagram showing an application example of an information processing device according to an embodiment of the present invention. As shown in Fig. 1, the information processing device 100 according to the embodiment of the present invention has an operation event detection unit 10, a label information search unit 20, and an operation log storage unit 30. The operation log storage unit 30 has a storage device such as a non-volatile memory. The content of the processing of each unit will be described later.

[0013] 1 is provided to multiple users as one or more systems that can be used in common via user terminals. In this embodiment, the information processing device 100 has a function for acquiring the key value (key of key-value) of the GUI component to be operated based on the object structure and operation screen of the business system. In other words, the information processing device 100 utilizes information on the object tree that constitutes the GUI screen to efficiently limit the GUI components that are label candidates to be searched for and associate the key with the value.

[0014] An operation log including this key value is then recorded in the operation log storage unit 30, and the user can refer to this recorded operation log to understand the operation content.

[0015] The present embodiment can be realized by an application program, and may be realized as an apparatus or a computer program. The terminal on which the present embodiment is realized may be an operation terminal on an operator's computer or a server on a cloud, and is not particularly limited.

[0016] 2 is a diagram showing an example of an input screen configured with GUI components. The example shown in FIG. 2 is an example of processing when searching for the character string "business trip destination" or "basic information" when, for example, the "local" checkbox is operated. Note that, in general, the label of a GUI component may refer to the character string "local" itself, but since this can often be linked relatively easily, it is not included in the search target in this embodiment.

[0017] In this embodiment, the label search range is defined by taking into account both the parent-child relationships in the object tree and the relative positions on the display screen. This embodiment targets a form in which the object tree is defined so that GUI components are arranged in a roughly tabular format, as shown in Figure 2. This includes, for example, the HTML DOM (Document Object Model) tree. By focusing on this, it is possible to logically search the table structure from top to bottom.

[0018] 3 is a diagram showing an example of a search range for a label of a GUI component. As an example of the search range for a label of a GUI component, as shown in area r in FIG. 3, the search range is changed step by step in the direction of the parent in the object tree, starting from the GUI component to be operated.

[0019] Examining all GUI component elements within a form requires a relatively high search cost, i.e., a lot of processing time, but by limiting the search range based on the object tree in this way, it is possible to reduce processing time.

[0020] In this embodiment, the label search range is limited based on the object tree as described above, and then the label search range is further narrowed down based on the relative position on the display screen.

[0021] Specifically, as shown in area b in Figure 3, the search range for the label is the intersection of the range extending upward and leftward from the search range on the object tree at the k-1th step to the end of the Form and the search range on the object tree at the kth step, i.e., area p in Figure 3.

[0022] The reason why the range extended upward and leftward as described above is set as the search range for labels on the display screen is that keys are generally often located above and to the left of values.

[0023] In this way, in this embodiment, by combining the object tree and relative position, it is possible to speed up label searches, and it is also possible to infer structures that are not expressed in the object tree structure from the apparent relative position.

[0024] Furthermore, in this embodiment, the attributes of the strings that match within the above search range, such as the type attribute value in HTML, are referenced, and direct labels such as check boxes, for example, "Local" shown in Figure 2, or values ​​in pull-down menus, which cannot be used as keys, are excluded from the label search range, which is expected to further improve accuracy.

[0025] As described above, in this embodiment, by limiting the search range of a label on the display screen and searching for that label, it is possible to significantly reduce the time required for this search. This makes it possible to obtain the key corresponding to a given value in real time.

[0026] One example of such a format is to record user operations including key-values ​​as an operation log, so that when the user later refers to the operation log, sufficient information is obtained to understand the operation details. The operation log envisioned here is recorded based on user operations, and values ​​including keys are acquired in real time. The reason for using such a format is that object trees, such as the DOM tree in HTML described above, can change dynamically, and it is reasonable to acquire values ​​including keys simultaneously with operations.

[0027] 4 is a flowchart showing an example of a procedure for processing operations by an information processing apparatus according to an embodiment of the present invention. First, when a user operates a GUI component and this operation is detected by the operation event detection unit 11 (S11), the label information search unit 20 sets k to an initial value of 2 (S12).

[0028] If the search range for the label of the operated GUI component is not the edge of the Form on the display screen (No in S13), the label information search unit 20 acquires the element k elements (corresponding to symbol r in FIG. 3) on the parent side of the operated GUI component on the object tree as candidates for the label search range (S14). The edge of the Form corresponds to the body element of the DOM tree.

[0029] Next, the label information search unit 20 identifies elements (corresponding to symbol b in Figure 3) in a range extending upward and leftward on the display screen from the area of ​​the parent element by k-1 elements from the GUI component to be operated as candidates for the label search range (S15).

[0030] Then, the label information search unit 20 identifies the common part (corresponding to symbol p in FIG. 3) between the element identified in S14 and the element identified in S15 as the final search range for the label, and determines whether or not there is a character string in this part that is a candidate for the key (S16).

[0031] If it is determined that there is no character string (No in S16), the label information searching unit 20 updates k by adding 1 (S19) and returns to S13.

[0032] If the above determination results in a character string being present (Yes in S16), the label information search unit 20 determines whether or not the element that is this character string is an object to be excluded from matching as a key (S17). Objects to be excluded from matching as a key include elements that are inappropriate as keys, such as character strings that are direct labels for check boxes, radio buttons, etc., text boxes, and character strings in pull-down menus.

[0033] If the determination in S17 is Yes, i.e., if the corresponding element is not an exclusion target for matching as a key, the label information search unit 20 writes an operation log with the corresponding matched element as a key to the operation log storage unit 30 (S18), and the series of processes ends. On the other hand, if the determination in S17 is No, i.e., if the corresponding element is an exclusion target for matching as a key, the process proceeds to S19 above. In other words, if the type of area related to a candidate for matching as a key is not appropriate as the type of area related to label information, the label information search unit 20 performs exclusion processing to exclude this area from candidates for label information.

[0034] Furthermore, when the search range for the label of the operated GUI component is at the edge of the Form on the display screen, i.e., when a matching key is not found (Yes in S13), the label information search unit 20 writes an operation log indicating that there is no key to the operation log storage unit 30 (S20), and the series of processes ends.

[0035] 5 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention. In the example shown in FIG. 5, the information processing device 100 according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0036] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0037] An input device 200 and an output device 300 attached to the information processing device 100 and used by a user or the like are connected to the input / output interface 113. The input / output interface 113 can take in operation data input by a user or the like through the input device 200 such as a keyboard, touch panel, touchpad, or mouse, and can output output data to an output device 300 including a display device using a liquid crystal or organic electroluminescence (EL) display, etc., for display. The input device 200 and the output device 300 may be devices built into the information processing device 100, or may be input devices and output devices of other information terminals that can communicate with the information processing device 100 via a network.

[0038] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and can store programs necessary to execute various control processes, etc., according to one embodiment.

[0039] The data memory 112 is a tangible storage medium that is, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM (Random Access Memory), and can be used to store various data or information acquired and created during various processes.

[0040] An information processing apparatus 100 according to an embodiment of the present invention can be configured as a data processing apparatus having the units shown in FIG. 1 as software-based processing function units.

[0041] The information storage unit used as a work memory or the like by each unit of the information processing device 100 can be configured by using the data memory 112 shown in Fig. 5. However, these configured storage areas are not essential components within the information processing device 100, and may be areas provided in, for example, an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0042] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0043] The methods described in each embodiment can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), or semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables and data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes storage media such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0044] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0045] REFERENCE SIGNS LIST 100... information processing device 10... operation event detection unit 20... label information search unit 30... operation log storage unit

Claims

1. An information processing device comprising: a first identification unit that identifies, on an information input screen including a plurality of areas associated by a hierarchical structure, different areas associated in accordance with the hierarchical structure with respect to an area to be operated for inputting the information, and identifies, from the identified areas, different areas along the upper and left directions on the input screen; and a second identification unit that, when the identified area is an area to display a character string on the input screen, identifies the character string as a candidate for label information of information that can be input into the area to be operated.

2. The information processing device according to claim 1, further comprising an exclusion processing unit that excludes an area from candidates for the label information when the type of area related to the candidate identified by the second identification unit is not appropriate as the type of area related to the label information.

3. A method performed by an information processing device, comprising: identifying, by a first identification unit of the information processing device, a different area associated in accordance with the hierarchical structure with respect to an area to be operated for inputting information on an information input screen including a plurality of areas associated by the hierarchical structure, and identifying, from the identified area, different areas along the upper and left directions on the input screen; and identifying, by a second identification unit of the information processing device, when the identified area is an area to display a character string on the input screen, the character string as a candidate for label information of information that can be input into the area to be operated.

Images