Display device, display method, and display program

Abstract

A display device (10) receives the designation of an object included in a first display screen relating to work. The display device (10) then identifies an object which corresponds to the object for which the designation was received and which is included in a second display screen having a display form different from that of the first display screen. The display device (10) subsequently displays the object for which the designation was received and the identified object in a prescribed form.

Description

Technical Field

[0001] The present invention relates to a display device, a display method, and a display program.

Background Art

[0002] In order to effectively improve operations, analysts need to accurately understand the actual state of operations. Conventionally, a method has been proposed in which an operation log of a terminal is acquired and visualized to eliminate personal attributes and enable an efficient, wide-ranging, and fine-grained understanding of the actual state of operations.

[0003] As an example of using visualization in business analysis, visualization in a timeline format (hereinafter referred to as timeline display) and visualization in a node-link type graph format (hereinafter referred to as node-link type display) are generally known (see, for example, Non-Patent Document 1). The timeline display is a visualization method in which time is assigned to the X-axis (or Y-axis), elements are arranged on the Y-axis (or X-axis), and the duration of each element is represented by a rectangle, and is suitable for an intuitive understanding of the sequential and co-occurrence relationships between elements. For example, based on the log of the transition between the active and inactive states of a window, a timeline display of the window usage state on a terminal is realized.

[0004] The node-link type display is, for example, a visualization method in which an operation or a task is represented as one node and the transition between nodes is represented as a link, and is suitable for understanding in what order the operations or tasks have transitioned. For an operation log including information at multiple granularities such as an application, a window, and operation details, the node-link type display is realized by using operations, etc. as nodes and aggregating and displaying the same operations.

[0005] For business analysts to identify problem areas in business processes, it is effective to combine multiple visualization representations depending on the analytical perspective. For example, if the purpose of the analysis is to find areas where RPA (Robotic Process Automation) tools can be applied, the analyst might examine the flow of operations using a node-link type display to check the standard operation flow and the presence or absence of branching in a particular task, while simultaneously checking the frequency and timing of the occurrence of that operation using a timeline display to consider the applicability of the tool. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] "Business visualization technology that contributes to the promotion of DX," [Accessed May 31, 2022], Internet<https: / / journal.ntt.co.jp / article / 880> [Overview of the project] [Problems that the invention aims to solve]

[0007] However, conventional technologies have presented challenges in efficiently analyzing data because it is difficult to establish correspondences between elements in different visualization representations. In other words, with conventional technologies, each visualization function exists independently, making it difficult to establish correspondences between elements and perform efficient analysis unless the analyst has a mental map of the meaning and data structure of each visualization representation.

[0008] Each visualization function reads operation logs and reflects the items in the visualization display according to rules defined for each visualization type. For example, the timeline is often used to understand the usage status of applications and windows, displaying the usage time of applications and windows as a bar chart. On the other hand, process visualization is often used to identify the user's operation flow, displaying each user operation as a node and connecting them with links that indicate the direction of the operation flow to represent the flow of operations.

[0009] In other words, each visualization function displays different objects, and the objects displayed do not have a one-to-one correspondence. Therefore, a mechanism is needed to map these different elements in a way that does not disrupt the user's mental map.

[0010] The present invention has been made in view of the above, and aims to provide a display device, a display method, and a display program that enable correspondence between objects in different visualization representations and enable effective business analysis. [Means for solving the problem]

[0011] To solve the above-mentioned problems and achieve the objective, the present invention is characterized by comprising: a reception unit that receives the designation of an object to be included in a first display screen relating to business operations; an identification unit that uses an operation log to identify an object corresponding to the object designated by the reception unit, which is included in a second display screen that has a different display mode from the first display screen; and a display unit that displays the object designated by the reception unit and the object identified by the identification unit in a predetermined manner. [Effects of the Invention]

[0012] According to the present invention, it is possible to establish correspondences between objects in different visualization representations, thereby enabling effective business analysis. [Brief explanation of the drawing]

[0013] [Figure 1] FIG. 1 is a block diagram illustrating the configuration of the display device according to the present embodiment. [Figure 2] FIG. 2 is a diagram showing an example of an operation log. [Figure 3] FIG. 3 is a diagram showing an example of a timeline display. [Figure 4] FIG. 4 is a diagram showing an example of a timeline display. [Figure 5] FIG. 5 is a diagram showing an example of a process visualization display. [Figure 6] FIG. 6 is a diagram for explaining specific processing by a specific unit. [Figure 7] FIG. 7 is a diagram illustrating an example of object cooperation. [Figure 8] FIG. 8 is a diagram illustrating an example of object cooperation. [Figure 9] FIG. 9 is a diagram illustrating an example of object cooperation. [Figure 10] FIG. 10 is a diagram illustrating an example of object cooperation. [Figure 11] FIG. 11 is a diagram illustrating an example of object cooperation. [Figure 12] FIG. 12 is a diagram illustrating an example of object cooperation. [Figure 13] FIG. 13 is a flowchart showing an example of a processing procedure by the display device according to the embodiment. [Figure 14] FIG. 14 is a diagram showing a computer that executes a program.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of the display device, display method, and display program according to the present application will be described in detail based on the drawings. Further, the present invention is not limited to the embodiments described below.

[0015] [Configuration of Specific Device] FIG. 1 is a block diagram illustrating the configuration of the specific device of the present embodiment. As shown in FIG. 1, the display device 10 of the present embodiment is connected to a terminal device 20 via a network (not shown). Note that the connection between devices is not limited to being made via a network, and any connection form may be used, whether wired or wireless. Further, in FIG. 1, the case where the display device 10 is connected to one terminal device 20 is illustrated, but it may be connected to a plurality of terminal devices 20.

[0016] The display device 10 monitors the operations performed on each terminal device 20 using an operation interface related to operation cooperation on the visualization screen displayed on the terminal device 20, and when detecting a user operation related to operation cooperation, identifies the object to be cooperated. Then, the display device 10 notifies the terminal device 20 of the information of the object, and based on the notified information, identifies the target object and highlights the identified object, thereby enabling the association between elements (objects) in different visualization expressions. Note that the display device 10 may be any type of information processing device including a server.

[0017] The terminal device 20 is an information processing device used by a user. The user is, for example, a person in charge of business. The person in charge of business uses various software such as a business system and general-purpose applications on the terminal device 20, for example. Note that the terminal device 20 may be any type of information processing device including a client device such as a smartphone, a desktop PC, a notebook PC, or a tablet PC. Further, in the example of FIG. 1, the display device 10 and the terminal device 20 are shown as separate devices, but the terminal device 20 may have some or all of the functions of the display device 10.

[0018] The terminal device 20 acquires operation logs, including operation time, location, and position, at the timing of operation events such as mouse and keyboard operations performed by the user. Here, the terminal device 20, for example, records the operation content on the terminal device 20 as a log, by detecting events that occur in conjunction with the operation from the OS and GUI framework, and by acquiring information on GUI components, thereby generating operation logs that serve as the basis for analysis, including information on operation type and location, which are the smallest units of an RPA scenario. The terminal device 20 then sends the operation log to the display device 10.

[0019] The display device 10 receives the operation log acquired by the terminal device 20 and stores it in the operation log storage unit 13a of the storage unit 13. The timing of the transmission and reception of the operation log between the display device 10 and the terminal device 20 can be any timing; for example, transmission and reception may be performed at predetermined time intervals, or transmission and reception may be performed each time the terminal device 20 acquires an operation log.

[0020] The display device 10 includes a communication processing unit 11, a control unit 12, and a storage unit 13. The processing of each part of the display device 10 is described below.

[0021] The communication processing unit 11 is implemented using a NIC (Network Interface Card) or the like, and controls communication between the control unit 12 and external devices via telecommunication lines such as a LAN (Local Area Network) or the Internet. For example, the communication processing unit 11 receives operation logs from the terminal device 20.

[0022] The storage unit 13 stores data and programs necessary for various processes performed by the control unit 12 and has an operation log storage unit 13a. For example, the storage unit 13 may be a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disc.

[0023] The operation log storage unit 13a stores operation logs acquired from the terminal device 20. For example, as illustrated in Figure 2, the operation log storage unit 13a stores a table containing columns such as "operation time" (the time when the operation event occurred), "user" (indicating the user who performed the operation), "application name" (indicating the application being operated on), "window title" (indicating the window title of the window being operated on), "event type" (indicating the type of operation event), and "operation location" (indicating the location of the operation). Note that the information included in the operation log is merely an example and is not limited to this.

[0024] The control unit 12 has an internal memory for storing programs that define various processing procedures and required data, and executes various processes using these. For example, the control unit 12 has a reception unit 12a, a specification unit 12b, and a display unit 12c. Here, the control unit 12 is an electronic circuit such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0025] The reception unit 12a accepts the specification of objects included in the display screen related to the business (the first display screen). For example, the reception unit 12a receives information from the terminal device 20 about objects specified by the user from among the objects included in the display screen related to the business displayed on the terminal device 20. In the following, the screen on which the specification of the object to be linked is accepted will be referred to as the first display screen, and a display screen with a different display format from the first display screen will be referred to as the second display screen as appropriate. The reception unit 12a may also monitor operations performed on the first display screen and detect user operations to specify objects included in the first display screen, and accept the specification of the object.

[0026] Here, using Figures 3 to 5, examples of display screens related to business operations displayed on the terminal device 20 will be explained. Figures 3 and 4 show examples of timeline displays. Figure 5 shows an example of process visualization displays. The timeline will be displayed in the order of user, application, and window hierarchy. In the example in Figure 3, a screen visualizing the timeline at the application level for user A's operations is shown. In the example in Figure 4, a screen visualizing the timeline at the window title level for user A's operations is shown. In the example in Figure 5, a screen visualizing the process of user A's operations is shown, with event types and operation locations linked as node labels.

[0027] When such a screen is displayed on terminal device 20, the user performs an operation to select the object to be linked. For example, in the example in Figure 3, the user may select the row for "iexplore.exe" on the timeline as the object to be linked. Also, for example in Figure 4, the user may select the row for "Web Page 1" on the timeline as the object to be linked. Also, for example in Figure 5, the user may select the node for Web Page 1 + Button a as the object to be linked.

[0028] The identification unit 12b uses the operation log to identify an object that corresponds to the object specified by the reception unit 12a and is included in a display screen (second display screen) that has a different display mode from the display screen related to the business (first display screen). For example, the identification unit 12b identifies an object on the second display screen that corresponds to an operation log that matches the operation log corresponding to the object specified by the reception unit 12a.

[0029] For example, the identification unit 12b determines the operation log item corresponding to the specified object, according to the type of the first display screen on which the object specification was received by the reception unit 12a. The identification unit 12b then obtains the information of the determined operation log item and identifies the object on the second display screen that matches the obtained information from among the objects on the second display screen.

[0030] Here, the identification process by the identification unit 12b will be explained using Figure 6. Figure 6 is a diagram illustrating the identification process by the identification unit. As illustrated in Figure 6, the "visualization type," which is the type of the first display screen, the "method for selecting the object to be linked," which indicates how to select the object to be linked, and the "linked object and linking information for each selection method," which indicates the items for identifying the linked object of the operation log corresponding to the object that has been specified, are predetermined.

[0031] For example, if the type of the first display screen that received the object specification from the reception unit 12a is "timeline" and objects are selected individually, the identification unit 12b obtains the "application" and "window title" of the selected objects from the operation log stored in the operation log storage unit 13a. Then, the identification unit 12b identifies the object among the objects on the second display screen whose obtained "application" and "window title" match. Note that the "operation type" exemplified in Figure 6 includes "event type" and "operation location".

[0032] The display unit 12c displays the object specified by the reception unit 12a and the object specified by the identification unit 12b in a predetermined manner. For example, the display unit 12c highlights the object specified by the reception unit 12a and the object specified by the identification unit 12b by surrounding them with a thick, colored border. Furthermore, the display unit 12c is not limited to displaying the linked objects in this manner. For example, the display unit 12c distinguishes the linked objects from other objects by changing their visual variables. Examples of visual variables include size, shape, hue, lightness, saturation, intensity, pattern, texture, transparency, and opacity.

[0033] Here, we will explain the linking of objects in detail using Figures 7 to 12. Figures 7 to 12 are diagrams illustrating examples of object linking. For example, in the example in Figure 7, the reception unit 12a accepts the selection of the iexplore.exe row on the timeline. Then, the identification unit 12b obtains the "Application" and "Window Title" of the objects included in the iexplore.exe row on the first display screen (the screen on the left in Figure 7) from the operation log stored in the operation log storage unit 13a.

[0034] The identification unit 12b then refers to the operation log and identifies four objects on the second display screen (the screen on the right in Figure 7) that match the acquired "Application" and "Window Title": "Web Page 1 + Button a", "Web Page 1 + Button c", "Web Page 2 + Text Box d", and "Web Page 2 + Button f". The display unit 12c then highlights the specified object and the identified object by surrounding them with a thick, colored border, as illustrated in Figure 7.

[0035] Furthermore, for example, in the example shown in Figure 8, the reception unit 12a accepts the selection of a row of webpage 1 on the timeline. The identification unit 12b then retrieves the "application" and "window title" of the objects included in the row of webpage 1 on the first display screen (the screen on the left in Figure 8) from the operation log stored in the operation log storage unit 13a.

[0036] The identification unit 12b then identifies two objects, "Web Page 1 + Button a" and "Web Page 1 + Button c," on the second display screen (the screen on the right in Figure 8) as objects that match the acquired "Application" and "Window Title." The display unit 12c then highlights the specified object and the identified object by surrounding them with a thick, colored border, as illustrated in Figure 8.

[0037] Furthermore, for example, in the example shown in Figure 9, the reception unit 12a accepts the selection of a row for webpage 2 and a row for file 1 on the timeline. The identification unit 12b then retrieves the "application" and "window title" of the objects contained in the row for webpage 2 and the row for file 1 on the first display screen (the screen on the left in Figure 9) from the operation log stored in the operation log storage unit 13a.

[0038] The identification unit 12b then identifies four objects on the second display screen (the screen on the right in Figure 9) that match the acquired "application" and "window title": "webpage 2 + text box d", "webpage 2 + button f", "file 1 + cell B4", and "file 1 + cell B6". The display unit 12c then highlights the specified object and the identified object by surrounding them with a thick, colored border, as illustrated in Figure 9.

[0039] Furthermore, for example, in the example shown in Figure 10, the reception unit 12a accepts the selection of two objects on the timeline. The identification unit 12b then retrieves the "application" and "window title" of the two objects on the first display screen (the screen on the left in Figure 10) from the operation log stored in the operation log storage unit 13a.

[0040] The identification unit 12b then identifies four objects on the second display screen (the screen on the right in Figure 10) that match the acquired "application" and "window title": "Web page 1 + button a", "Web page 1 + button c", "File 1 + cell B4", and "File 1 + cell B6". The display unit 12c then highlights the specified object and the identified object by surrounding them with a thick, colored border, as illustrated in Figure 10.

[0041] Furthermore, for example, in the example shown in Figure 11, the reception unit 12a accepts the selection of the web page 1 + button a node on the process visualization screen. The identification unit 12b then retrieves the "application," "window title," "event type," and "operation location" of the selected node on the first display screen (the screen on the left in Figure 11) from the operation log stored in the operation log storage unit 13a.

[0042] The identification unit 12b then identifies objects on the second display screen (the screen on the right in Figure 11) at the application level and window level, respectively, as objects that match the acquired "application," "window title," "event type," and "operation location." The display unit 12c then highlights the specified object and the identified object by surrounding them with a thick, colored border, as illustrated in Figure 11.

[0043] Furthermore, for example, in the example shown in Figure 12, the reception unit 12a accepts the selection of the Web Page 1 + Button a node and the File 1 + Cell B6 node on the process visualization screen. The identification unit 12b then retrieves the "Application," "Window Title," "Event Type," and "Operation Location" of the selected node on the first display screen (the screen on the left in Figure 12) from the operation log stored in the operation log storage unit 13a.

[0044] The identification unit 12b then identifies objects on the second display screen (the screen on the right in Figure 12) at the application level and window level, respectively, as objects that match the acquired "application," "window title," "event type," and "operation location." The display unit 12c then highlights the specified object and the identified object by surrounding them with a thick, colored border, as illustrated in Figure 12.

[0045] [Processing procedure for display device 10] Next, an example of the processing procedure performed by the display device 10 will be described using Figure 13. Figure 13 is a flowchart showing an example of the processing procedure by the display device of the embodiment.

[0046] As illustrated in Figure 13, the reception unit 12a of the display device 10 monitors operations performed on each display unit (not shown) of each terminal device 20 (step S101) and detects user operations related to operation coordination (step S102).

[0047] Then, the identification unit 12b identifies the object to be linked (step S103). For example, using the operation log, the identification unit 12b identifies an object that corresponds to the object specified by the reception unit 12a and is included in a display screen (second display screen) that has a different display mode from the display screen related to the business (first display screen). Subsequently, the display unit 13c highlights the identified object (step S104).

[0048] [Effects of the embodiment] As described above, the display device 10 according to this embodiment receives a designation of an object included in a first display screen related to a business operation, uses an operation log to identify an object corresponding to the designated object that is included in a second display screen with a different display mode from the first display screen, and displays the designated object and the identified object in a predetermined manner. This enables the display device 10 to associate objects in different visualization representations and to perform effective business analysis.

[0049] In other words, the display device 10 identifies the objects to be linked and highlights the identified objects, allowing users to easily observe areas of interest from multiple perspectives. Furthermore, the display device 10 enables effective business analysis by allowing correspondence between different visualization elements (objects).

[0050] [System configuration, etc.] Each component of the illustrated device according to the above embodiment is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Furthermore, each processing function performed by each device can be implemented in whole or in any part by a CPU and a program that is analyzed and executed by the CPU, or by hardware using wired logic.

[0051] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified.

[0052] 〔program〕 Furthermore, it is also possible to create a program in which the processing performed by the display device described in the above embodiment is written in a computer-executable language. In this case, the same effects as in the above embodiment can be obtained by having the computer execute the program. Moreover, the same processing as in the above embodiment may be achieved by recording such a program on a computer-readable recording medium and having the computer read and execute the program recorded on this recording medium.

[0053] Figure 14 shows a computer running a program. As illustrated in Figure 14, computer 1000 includes, for example, memory 1010, CPU 1020, hard disk drive interface 1030, disk drive interface 1040, serial port interface 1050, video adapter 1060, and network interface 1070, all of which are connected by bus 1080.

[0054] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM 1012, as illustrated in Figure 14. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090, as illustrated in Figure 14. The disk drive interface 1040 is connected to the disk drive 1100, as illustrated in Figure 14. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1051 and a keyboard 1052, as illustrated in Figure 14. The video adapter 1060 is connected to, for example, a display 1061, as illustrated in Figure 14.

[0055] Here, as illustrated in Figure 14, the hard disk drive 1090 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. That is, the above program is stored in the hard disk drive 1090 as a program module in which instructions to be executed by the computer 1000 are described.

[0056] Furthermore, the various data described in the above embodiment are stored as program data in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes various processing procedures.

[0057] Furthermore, the program module 1093 and program data 1094 related to the program are not limited to being stored on the hard disk drive 1090, but may also be stored on a removable storage medium, for example, and read by the CPU 1020 via a disk drive or the like. Alternatively, the program module 1093 and program data 1094 related to the program may be stored on another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.) and read by the CPU 1020 via the network interface 1070.

[0058] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in this embodiment. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on this embodiment are included in the scope of the present invention. [Explanation of Symbols]

[0059] 10 Display device 11 Communication Processing Unit 12 Control Unit 12a Reception Desk 12b Specific part 12c Display section 13 Storage section 13a Operation log storage unit

Claims

1. A reception unit that accepts the specification of objects included in the first display screen related to the business, An identification unit identifies an object that corresponds to an object specified by the reception unit, and which is included in a second display screen that has a different display mode from the first display screen, using the operation log. An object whose designation is received by the reception unit, and a display unit that displays the object identified by the identification unit in a predetermined manner. It has, The reception unit monitors operations performed on the first display screen, detects user operations that specify objects included in the first display screen, and accepts the specification of the object. The display device is characterized in that the identification unit determines an operation log item corresponding to the specified object according to the type of first display screen on which the object specification was received by the reception unit, and identifies the object on the second display screen that matches the information of the determined operation log item.

2. The display device according to claim 1, characterized in that the identifying unit identifies an object on the second display screen that matches an operation log that matches an operation log corresponding to an object specified by the receiving unit.

3. The display device according to claim 1, characterized in that the display unit displays the object specified by the reception unit and the object specified by the identification unit by enclosing them in a thick, colored border.

4. A display method performed by a display device, A reception process that accepts the specification of objects included in the first display screen related to the business, A selection step that uses the operation log to identify an object corresponding to the object specified in the reception step, which is included in a second display screen that has a different display mode from the first display screen, A display step that displays an object specified by the aforementioned reception step and an object specified by the aforementioned identification step in a predetermined manner. Includes, The reception process monitors operations performed on the first display screen, detects user operations that specify objects included in the first display screen, and accepts the specification of the object. The aforementioned identification step is characterized by determining an operation log item corresponding to the specified object according to the type of first display screen that received the object specification in the reception step, and identifying an object on a second display screen that matches the information of the determined operation log item.

5. A reception step that accepts the specification of objects to be included in the first display screen related to the business, A selection step that uses the operation log to identify an object corresponding to the object specified in the reception step, which is included in a second display screen that has a different display mode from the first display screen, The object specified in the aforementioned reception step and the display step which displays the object specified in the aforementioned identification step in a predetermined manner. Have the computer run it, The reception step monitors operations performed on the first display screen, detects user operations that specify objects included in the first display screen, and accepts the specification of the object. The display program is characterized in that the aforementioned identification step determines an operation log item corresponding to the specified object according to the type of first display screen that received the object specification in the reception step, and identifies the object on the second display screen that matches the information of the determined operation log item.

Images