Information processing apparatus, ui control method, and program

Abstract

To improve user experience when using another device having an input function as an input device.SOLUTION: The information processing apparatus 10 is configured to be able to communicate with an input device 20 having a touch panel display. The information processing device 10 includes a display, an information acquisition unit 31 that acquires input operation information including information on an application as an operation target displayed on the display, a menu specifying unit 33 that specifies menu information to be displayed on a display screen of the input device 20 on the basis of the acquired input operation information, and a display instruction unit 34 that transmits instruction information for displaying operation screen information including the specified menu information on the display screen of the input device 20 to the input device 20.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a UI control method, and a program. [Background technology]

[0002] In recent years, a technology has been proposed that allows a smartphone to be used as an input device (controller) for information processing devices such as laptops by installing a specific application on the smartphone. One example is the "FMV Controller" (https: / / www.fmworld.net / fmv / soft / fmv-controller.html) provided by Fujitsu Client Computing Ltd. This FMV controller has, for example, (a) touchpad mode, (b) presentation remote control mode, and (c) video remote control mode, as shown in Figure 8, and allows the user to use the smartphone as the desired input device by switching the smartphone mode depending on the application to be operated.

[0003] Furthermore, Patent Document 1 discloses a technique for connecting a smartphone and a display device via wireless communication and using the smartphone as a remote controller for the display device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6378487 Summary of the Invention [Problem to be solved by the invention]

[0005] In conventional input devices using smartphones, the application to be operated and the operation screen displayed on the smartphone are not linked. Therefore, if the application to be operated on a laptop (e.g., a video playback application) is in a different mode from the default mode (e.g., touchpad mode), the user must manually switch to the corresponding input mode by operating the smartphone.

[0006] The present invention has been made in consideration of the above circumstances, and aims to provide an information processing device, a UI control method, and a program that can improve the user experience when using other devices with input functions as input devices. [Means for solving the problem]

[0007] One aspect of the present invention is an information processing device configured to be able to communicate with an input device having a touch panel display, and comprising: a display; an information acquisition unit that acquires input operation information including information about an application to be operated and displayed on the display; a menu identification unit that identifies menu information to be displayed on a display screen of the input device based on the acquired input operation information; and a display instruction unit that transmits instruction information to the input device for displaying operation screen information including the identified menu information on the display screen of the input device.

[0008] One aspect of the present invention is a UI control method for displaying a user interface for operating an information processing device on a touch panel display of an input device, the UI control method being performed by a computer through the steps of: acquiring input operation information including information on an application to be operated that is displayed on the display of the information processing device; identifying menu information to be displayed on a display screen of the input device based on the acquired input operation information; and transmitting instruction information to the input device for displaying operation screen information including the identified menu information on the display screen of the input device.

[0009] One aspect of the present invention is a program for causing a computer to function as the information processing device. [Effects of the Invention]

[0010] According to the present invention, it is possible to improve the user experience when using another device having an input function as an input device. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a system configuration diagram schematically showing the overall configuration of an information processing system according to a first embodiment of the present invention. [Figure 2] 1 is a schematic configuration diagram showing an example of a hardware configuration of an information processing device according to a first embodiment of the present invention. [Figure 3] 1 is a functional configuration diagram showing an example of functions provided in a UI control system according to a first embodiment of the present invention. [Figure 4] 10 is a diagram showing an example of a plurality of pieces of button information (menu information) belonging to a "graphics drawing" group according to the first embodiment of the present invention. FIG. [Figure 5] FIG. 2 is a diagram showing an example of an operation screen on which menu information relating to "figure drawing" is displayed according to the first embodiment of the present invention. [Figure 6] 3 is a flowchart showing an example of a processing procedure of a UI control method according to the first embodiment of the present invention. [Figure 7] FIG. 10 is a functional configuration diagram showing an example of functions provided in a UI control system according to a second embodiment of the present invention. [Figure 8] FIG. 1 is a diagram illustrating a conventional technique. DETAILED DESCRIPTION OF THE INVENTION

[0012] [First embodiment] An information processing device, a UI control method, and a program according to a first embodiment of the present invention will be described below with reference to the drawings.

[0013] FIG. 1 is a system configuration diagram that schematically shows the overall configuration of an information processing system 1 according to a first embodiment of the present invention.

[0014] 1, the information processing system 1 includes an information processing device 10 and an input device 20. In the information processing system 1, the information processing device 10 and the input device 20 are configured to be capable of two-way communication. Examples of the information processing device 10 include a notebook PC, a desktop PC, a tablet terminal, and an all-in-one PC in which the PC main body and a display are integrated into a single housing. The input device 20 is a device that has a touch panel display and is configured to be able to communicate with the information processing device 10. Examples of the input device 20 include a smartphone and a tablet terminal.

[0015] 2 is a schematic diagram showing an example of the hardware configuration of an information processing device 10 according to the first embodiment of the present invention. The information processing device 10 includes, for example, a CPU (Central Processing Unit: processor) 11, a main memory 12, a secondary storage (memory) 13, an external interface 16, a communication interface 17, a display 18, and an input device 19. These components are connected to each other, for example, via a bus.

[0016] The CPU 11 controls the entire information processing device using, for example, an OS (Operating System) stored in a secondary storage device 13 connected via a bus, and executes various processes by executing various programs stored in the secondary storage device 13. One or more CPUs 11 may be provided, and they may work together to realize processes.

[0017] The main memory 12 is composed of writable memory such as cache memory or RAM (Random Access Memory), and is used as a working area for reading out programs executed by the CPU 11 and writing data processed by the programs.

[0018] The secondary storage device 13 is a non-transitory computer-readable storage medium. Examples of the secondary storage device 13 include a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, and a semiconductor memory. Examples of the secondary storage device 13 include a read-only memory (ROM), a hard disk drive (HDD), a solid-state drive (SSD), and a flash memory. The secondary storage device 13 stores, for example, an operating system (OS) for controlling the entire information processing device, such as iOS (registered trademark), Android (registered trademark), or Windows (registered trademark), a basic input / output system (BIOS), various device drivers for operating peripheral devices, various application software, and various data and files. The secondary storage device 13 also stores programs for implementing various processes and various data required for implementing the various processes. A plurality of secondary storage devices 13 may be provided, and the programs and data described above may be stored separately in each secondary storage device 13.

[0019] The external interface 16 is an interface for connecting to an external device. Examples of external devices include an input device 20, an external monitor, a USB memory, an external HDD, etc. Although only one external interface is shown in the example shown in FIG. 2, multiple external interfaces may be provided.

[0020] The communication interface 17 functions as an interface for connecting to a network to communicate with other devices and transmitting and receiving information. For example, the communication interface 17 communicates with other devices via a wired or wireless connection. Examples of wireless communication include communication via lines such as Bluetooth (registered trademark), Wi-Fi, mobile communication systems (3G, 4G, 5G, 6G, LTE, etc.), and wireless LAN. An example of wired communication is communication via lines such as a wired LAN (Local Area Network).

[0021] The display 18 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, etc. Furthermore, the display 18 may be a touch panel display on which a touch panel is superimposed. The input device 19 is a user interface, such as a keyboard, a mouse, or a touchpad, that allows the user to input instructions to the information processing device 10.

[0022] FIG. 3 is a functional configuration diagram showing an example of functions provided in a UI control system (UI control function) 30 of the information processing device 10 according to this embodiment. A series of processes for realizing the functions of the UI control system 30 described below is stored in the secondary storage device 13 or the like in the form of a program, for example, and the CPU (processor) 11 reads this program into the main memory 12 and executes information processing and arithmetic processing to realize various functions. Note that the program may be pre-installed in the secondary storage device 13, provided in a state stored in another non-transitory computer-readable storage medium, or distributed via wired or wireless communication means. Examples of non-transitory computer-readable storage media include magnetic disks, magneto-optical disks, CD-ROMs, DVD-ROMs, and semiconductor memories.

[0023] As shown in FIG. 3, the UI control system 30 includes an information acquisition unit 31, a menu information acquisition unit 32, a menu specification unit 33, a display instruction unit , and a UI information creation unit .

[0024] The information acquisition unit 31 acquires input operation information including information on the application to be operated that is displayed on the display 18. For example, the information acquiring unit 31 acquires information on the foreground (active window) application as information on the application to be operated. Furthermore, the information acquiring unit 31 may acquire information on not only the foreground application but also the background application.

[0025] Furthermore, the information acquisition unit 31 may acquire, as input operation information, information about an object that is focused in the application to be operated. Here, an object is information that defines the type of control. For example, in the case of an application for creating presentation materials, examples of objects include "edit," "image," and "text box." Objects are, for example, information managed by the OS. As an example, when the OS is "Microsoft Windows," the information acquisition unit 31 acquires at least one piece of information, "Localized Control Type" or "Name," from the OS as object information.

[0026] The menu information acquisition unit 32 acquires a plurality of pieces of menu information for the application to be operated. The menu information includes, for example, information on a plurality of buttons displayed on a toolbar.

[0027] Here, the plurality of menu information may be grouped by function and acquired. For example, if the application to be operated is a presentation application, the menu information acquisition unit 32 acquires, as menu information, information on a plurality of buttons belonging to groups such as "clipboard," "slide," "font," "paragraph," "graphics drawing," and "edit." For example, Fig. 4 illustrates information on a plurality of buttons (menu information) belonging to the "graphics drawing" group.

[0028] The menu specifying unit 33 specifies menu information to be displayed on the display screen of the input device 20 based on the input operation information acquired by the information acquiring unit 31 . For example, the menu specifying unit 33 specifies, from among the plurality of pieces of menu information obtained by the menu information obtaining unit 32, the menu information corresponding to the object information obtained by the information obtaining unit 31.

[0029] For example, if the application to be operated is an unknown application, in other words, if it is an application for which input operation information has been acquired for the first time as the application to be operated, the menu identification unit 33 identifies menu information near the mouse cursor. The menu identification unit 33 may also identify menu information based on the movement history of the mouse cursor over a predetermined period of time in the past. Since it is possible to infer what operation the user is about to perform from the movement history of the mouse cursor over the predetermined period of time in the past, it becomes possible to identify menu information that matches the user's intention.

[0030] Furthermore, in the case of a known application, in other words, an application for which input operation information has been acquired as an application to be operated at least once in the past, UI information based on a past operation history is registered by the UI information creation unit 35, which will be described later. Specifically, the UI information is information in which application identification information, object information, and menu information are associated and registered. Note that for commonly used applications, default UI information may be registered in advance, which makes it possible to identify menu information for commonly used applications using the UI information.

[0031] In this way, when UI information corresponding to the application to be operated is registered, the menu identification unit 33 uses the UI information to identify menu information corresponding to the input operation information acquired by the information acquisition unit 31. Specifically, menu information corresponding to the application information and object information to be operated acquired as the input operation information is acquired from the UI information, and the acquired menu information is identified as menu information to be displayed on the input device 20.

[0032] The display instruction unit 34 transmits to the input device 20 instruction information for displaying operation screen information including the identified menu information (input buttons) on the display screen of the input device 20.

[0033] For example, the instruction information may be transmitted to the input device 20 as file information in XML format in which button IDs for identifying various menu buttons constituting the operation screen are written. That is, the instruction information transmitted from the information processing device 10 to the input device 20 varies depending on the type of information held by the input device 20. The instruction information may be determined appropriately depending on the operation, etc., and is not particularly limited as long as it is information that can display the specified operation screen on the display screen of the input device 20.

[0034] When the input device 20 receives the instruction information, it displays an operation screen based on the instruction information on the touch panel display 21. As a result, for example, in each application, menus (for example, various input buttons displayed as a toolbar) through which the user performs input operations are selected and displayed on the display screen of the input device 20. Fig. 5 shows an example of an operation screen on which menu information related to graphic drawing is displayed.

[0035] When a user presses a desired input button from among various input buttons displayed on the input device 20, operation information corresponding to the pressed input button is transmitted from the input device 20 to the information processing device 10. The information processing device 10 executes processing based on the operation information received from the input device 20. Note that this processing is a known technique, and therefore a detailed description thereof will be omitted here.

[0036] Furthermore, the operation information received from the input device 20 is input to the UI information creation unit 35. The UI information creation unit 35 creates or updates UI information based on the received operation information. Specifically, the UI information creation unit 35 creates or updates UI information based on an input operation history in which focused object information is associated with operation information input by the user at that time. The UI information may be updated every time, or may be updated by statistically processing past data. In other words, as long as the input operation history, which is the actual input operation record by the user, can be reflected in the UI information, the method for doing so is not particularly limited.

[0037] In this way, by reflecting the actual input operation history of the user in the UI information, it becomes possible to customize the UI information, and as a result, it becomes possible to display on the touch panel display 21 of the input device 20 a menu that is likely to be selected by the user.

[0038] Next, a UI control method by the information processing device 10 according to this embodiment will be described with reference to the drawings. Fig. 6 is a flowchart showing an example of the processing procedure of the UI control method according to this embodiment. Note that the series of processes shown below are repeatedly executed at predetermined time intervals.

[0039] First, the process determines whether the mouse cursor has been moved and the focused object has changed as a result of the user's input operation (SA1). If the object has not changed (SA1: NO), the process ends. On the other hand, if the focused object has changed (SA1: YES), the process obtains foreground application information and object information as input operation information (SA2). Next, menu information is obtained from the foreground application (SA3).

[0040] Next, the menu information corresponding to the input operation information is identified from the plurality of pieces of menu information obtained (SA4), and instruction information for displaying operation screen information including the identified menu information on the display screen of the input device 20 is transmitted to the input device 20 (SA5), and this process ends.

[0041] As a result, for example, as shown in FIG. 5, an operation screen corresponding to the instruction information is displayed on the input device 20.

[0042] As explained above, according to this embodiment, the information processing device 10 includes an information acquisition unit 31 that acquires input operation information including information on the application to be operated that is displayed on the display 18 of the information processing device 10, a menu identification unit 33 that identifies menu information to be displayed on the display screen of the input device 20 based on the acquired input operation information, and a display instruction unit 34 that transmits instruction information to the input device 20 for displaying operation screen information including the identified menu information on the display screen of the input device 20.

[0043] According to this configuration, an operation screen including menu information corresponding to an application running in the foreground on the information processing device 10 can be automatically displayed on the input device 20. This eliminates the need for a user to manually switch the display mode as in the past, and improves the user experience when using another device with an input function as the input device 20.

[0044] Furthermore, according to this embodiment, a menu information acquisition unit 32 is provided that acquires multiple pieces of menu information in the application to be operated (foreground application), and a menu identification unit 33 identifies the menu information to be displayed on the display screen of the input device from the multiple pieces of menu information acquired by the menu information acquisition unit 32.

[0045] In this way, since menu information is obtained from the foreground application, it is possible to generate operation screen information corresponding to the foreground application. As a result, even when an unknown application is launched, it is possible to dynamically create an operation screen corresponding to the application and display it on the input device 20.

[0046] It is possible to omit the menu information acquisition unit 32. That is, by registering the above-described UI information in advance, it is possible to omit the process of acquiring menu information from an application.

[0047] Furthermore, according to this embodiment, the input operation information includes object information focused on in the application to be operated, and the menu identification unit 33 uses the application identification information, the object information, UI information associated with menu information, and the input operation information to identify the menu information to be displayed on the display screen of the input device 20.

[0048] This allows menu information corresponding to object information in the application to be operated to be displayed on the input device 20. This allows the input buttons displayed on the display screen of the input device 20 to be limited to menus related to the current user operation. As a result, it is possible to provide the user with an operation screen that is easy to see and operate. For example, in the case of an application with many menus, such as a presentation application, it is difficult to display all of the menus on the touch panel display 21 of the input device 20. In such a case, operability can be improved by selecting a menu that is highly relevant to the user's current operation and displaying it on the input device 20.

[0049] Furthermore, according to this embodiment, the UI information creating unit 35 is provided, which creates or updates UI information based on an input operation history in which object information and information on input operations performed by a user are associated with each other.

[0050] This allows customization of UI information, and allows the input device 20 to display a menu that is likely to be selected by the user.

[0051] Furthermore, according to this embodiment, the menu specifying unit 33 can also specify, from the menu information acquired by the menu information acquiring unit 32, the menu information that corresponds to the position of the mouse cursor.

[0052] As a result, even when an unknown application is launched as an application to be operated, it is possible to display an operation screen corresponding to the application on the input device 20. That is, even when an application does not have the above-described UI information, it is possible to display an operation screen corresponding to the application on the input device 20. Even in such a case, the UI information based on the user's operation history is created by the UI information creation unit 35, and therefore, in the subsequent processing, it is possible to identify menu information based on this UI information.

[0053] Second Embodiment Next, a second embodiment of the present invention will be described. The information processing device 10a and UI control method and program according to this embodiment differ from the UI control system 30 according to the first embodiment in that it further includes a behavior prediction unit 36 ​​(see FIG. 7). Hereinafter, components common to the first embodiment will be assigned the same reference numerals and description thereof will be omitted, and differences will be mainly described.

[0054] 7 is a functional configuration diagram showing an example of functions provided in the UI control system 30a according to this embodiment. As shown in FIG. 7, the UI control system 30a according to this embodiment further includes a behavior prediction unit 36 ​​in addition to the UI control system 30 according to the first embodiment described above.

[0055] The behavior prediction unit 36 ​​predicts the user's next behavior based on input operation information and the user's past operation history. That is, the UI control system 30a according to this embodiment has a function of accumulating past operation information by the user for each application and predicting the user's next behavior, specifically, the information about the object to be focused next, based on this past operation information. Note that known machine learning techniques can be used for behavior prediction. For example, a hidden Markov model (HMM) can be used to predict behavior.

[0056] The menu specifying unit 33a specifies, from the menu information acquired by the menu information acquiring unit 32, menu information that corresponds to the next object information predicted by the behavior predicting unit .

[0057] As described above, according to this embodiment, the user's next action is predicted based on the user's past operation history, and a menu corresponding to the predicted action is displayed on the input device 20. This makes it possible to display a menu highly relevant to the operation intended by the user on the input device 20, thereby further improving the user experience.

[0058] Note that the UI control system 30a according to this embodiment performs machine learning and therefore requires a considerable amount of data. Therefore, until a certain level of machine learning accuracy is achieved, the operation of the behavior prediction unit 36 ​​may be stopped, i.e., the UI control system 30 according to the first embodiment may be used for operation. After a considerable amount of data has been accumulated, the behavior prediction unit 36 ​​may be activated, and the UI control system 30a according to this embodiment may be used for operation.

[0059] In this way, when the number of data in the user's input operation history is less than a predetermined value, the operation of the behavior prediction unit is stopped, and when the number of data in the user's input operation history is equal to or greater than a predetermined value, behavior prediction by the behavior prediction unit 36 ​​is started, thereby making it possible to display the operation screen on the input device 20 using an appropriate method according to the number of data.

[0060] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. Various changes or improvements can be made to the above embodiments without departing from the gist of the present invention, and such changes or improvements are also included in the technical scope of the present invention. [Explanation of symbols]

[0061] 1: Information processing system 10: Information processing device 10a: Information processing device 11: CPU 12: Main memory 13:Secondary storage device 16: External interface 17: Communication interface 18: Display 19: Input device 20: Input device 21: Touch panel display 30: UI control system 30a: UI control system 31: Information acquisition department 32: Menu information acquisition section 33: Menu specific section 33a: Menu specific section 34:Display instruction section 35:UI information creation department 36: Behavior prediction section

Claims

1. An information processing device configured to be able to communicate with an input device having a touch panel display, The display and an information acquisition unit that acquires input operation information including information on an application to be operated that is displayed on the display; a menu specifying unit that specifies menu information to be displayed on a display screen of the input device based on the acquired input operation information; a display instruction unit that transmits instruction information to the input device for displaying operation screen information including the specified menu information on a display screen of the input device; An information processing device comprising:

2. a menu information acquisition unit that acquires a plurality of pieces of menu information in the application to be operated, The information processing apparatus according to claim 1 , wherein the menu specifying unit specifies, from among the plurality of pieces of menu information, menu information to be displayed on the display screen of the input device.

3. the input operation information includes information about an object that is focused in an application that is the object of the operation; 3. The information processing device according to claim 2, wherein the menu specification unit specifies menu information to be displayed on the display screen of the input device using UI information in which application identification information, object information, and menu information are associated, and the input operation information.

4. The information processing apparatus according to claim 3 , further comprising a UI information creating unit that creates or updates the UI information based on an input operation history in which the object information and information on input operations by a user are associated with each other.

5. The information processing apparatus according to claim 2 , wherein the menu specifying unit specifies the menu information corresponding to the position of a mouse cursor from among the menu information acquired by the menu information acquiring unit.

6. the input operation information includes information about an object that is focused in the application that is the operation target; a behavior prediction unit that predicts the user's next behavior based on a behavior history in which the object information is associated with the user's past input operation history; The information processing apparatus according to claim 1 , wherein the menu specifying unit specifies menu information corresponding to the behavior predicted by the behavior predicting unit.

7. A UI control method for displaying a user interface for operating an information processing device on a touch panel display of an input device, comprising: acquiring input operation information including information on an application to be operated that is displayed on a display of the information processing device; a step of specifying menu information to be displayed on a display screen of the input device based on the acquired input operation information; transmitting instruction information to the input device for displaying operation screen information including the specified menu information on a display screen of the input device; A UI control method in which the above is executed by a computer.

8. A program for causing a computer to function as the information processing device according to any one of claims 1 to 6.

Images