Information processing device, information processing control method, and computer program

Abstract

To solve the problem in which, when entering a character string by remotely operating an operation screen of an image forming apparatus by a mobile terminal, it is difficult for a user to operate it if both a software keyboard of the mobile terminal and a software keyboard of an image processing apparatus are displayed.SOLUTION: When an input field is selected, an image processing apparatus determines whether a remote operation is being performed, and switches whether or not to display a software keyboard of the image processing apparatus according to its state.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] This relates to a method for displaying the operation panel on a remote client. [Background technology]

[0002] A method has been disclosed in which the operation screen of an information processing device can be operated from a mobile terminal or the like using VNC (Virtual Network Computing) using the RFB protocol to remotely operate the operation screen of the information processing device. Specifically, an information processing device having a VNC server function transmits screen information of a screen displayed on an operation panel of the information processing device to a terminal having a VNC client function. Based on the received screen information, the terminal displays on its operation panel a screen (remote operation screen) that is the same as the operation screen displayed on the operation panel of the information processing device. Then, based on operations on the screen of the operation panel of the terminal, the terminal transmits operation information to the information processing device and remotely operates the information processing device.

[0003] For example, Patent Document 1 discloses a method for inputting characters when remotely operating the screen of an information processing device using a mobile terminal. Patent Document 1 discloses a method for displaying a software keyboard (hereinafter referred to as a soft keyboard) of the mobile terminal on the remote operation screen when an instruction to input characters is received when the mobile terminal remotely operates the screen of the information processing device. This method allows characters to be input to the information processing device using the soft keyboard of the mobile terminal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-137671 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when the soft keyboard of the mobile terminal is displayed on the mobile terminal that remotely controls the screen of the information processing device as in the conventional method, there is a risk that usability will be reduced.

[0006] Consider a case where an information processing device that displays its own soft keyboard when it receives an instruction to input characters is remotely controlled from a mobile terminal. When a character input instruction is received on the remote control screen, the information processing device displays its own soft keyboard, and the mobile terminal also displays its own soft keyboard, resulting in two soft keyboards being displayed on the remote control screen. This results in poor visibility and the possibility that the display of the soft keyboard may hide other setting items, reducing operability. Patent Document 1 does not take such a case into consideration. Furthermore, since the soft keyboard of a mobile terminal is designed for inputting characters on the mobile terminal, it would be beneficial for the user if they could input characters using the soft keyboard of the mobile terminal when inputting characters on the mobile terminal even while remotely controlling the device.

[0007] In view of the above, the present invention aims to provide a terminal that remotely controls the screen of an information processing device that displays the device's own soft keyboard when receiving an instruction to input characters, without reducing the user's operability when inputting characters using the terminal's own soft keyboard. [Means for solving the problem]

[0008] An information processing device having a first display unit, wherein when a selection operation of a character input item is accepted on a screen displayed on the first display unit, a first software keyboard is displayed on the first display unit. Display control means; , the first display unit of When the screen is updated, the display device has a transmitting means for transmitting screen information corresponding to the updated screen to the terminal, and the screen information transmitted by the transmitting means is transmitted to the terminal. The aforementioned Based on the screen information, information indicating that a selection operation of a character input item has been accepted on the screen displayed on the second display unit of the terminal is received from the terminal, the transmitting means teeth, different from the first software keyboardtransmitting to the terminal an instruction to display a second software keyboard of the terminal on the second display unit; The display control means 10. An information processing apparatus comprising: a display unit configured to display a first software keyboard on the first display unit; [Effects of the Invention]

[0009] According to the present invention, a terminal that remotely controls the screen of an information processing device that displays the device's own soft keyboard when receiving an instruction to input characters does not reduce the user's operability when inputting characters using the terminal's own soft keyboard. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a communication system. [Figure 2] FIG. 2 is a diagram illustrating a hardware configuration of a mobile terminal 100. [Figure 3] FIG. 2 is a diagram showing the software configuration of the mobile terminal 100. [Figure 4] FIG. 2 is a diagram illustrating a hardware configuration of an image processing device 110. [Figure 5] FIG. 2 is a diagram showing the software configuration of the image processing device 110. [Figure 6] 10 is a flowchart showing the processing of the mobile terminal 100 of this embodiment. [Figure 7] 4 is a flowchart showing the processing of the image processing apparatus of the present embodiment. [Figure 8] 10A and 10B are diagrams illustrating a UI displayed on an operation panel of the mobile terminal 100 according to the present embodiment. [Figure 9] 10A and 10B are diagrams showing UIs displayed on an operation panel of an image processing device 110 according to the present embodiment. [Figure 10] 10A and 10B are diagrams illustrating a UI displayed on an operation panel of the mobile terminal 100 according to the present embodiment. [Figure 11] 10A and 10B are diagrams showing UIs displayed on an operation panel of an image processing device 110 according to the present embodiment. [Figure 12]10 is a flowchart showing the processing of the mobile terminal 100 of this embodiment. [Figure 13] 10A and 10B are diagrams illustrating a UI displayed on an operation panel of the mobile terminal 100 according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] The best mode for carrying out the present invention will be described below with reference to the drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0012] Example 1 <System Configuration in this Embodiment> First, the system configuration according to this embodiment will be described with reference to Fig. 1. Here, a mobile terminal 100 and an MFP (multifunction peripheral) 110 will be described as examples of information processing devices, but the devices included in the information processing system are not limited to these. The mobile terminal 100 and the MFP 110 are connected to a LAN (Local Area Network) and can exchange information with each other. The mobile terminal 100 is connected to the LAN via an access point 120.

[0013] Mobile terminal 100 has a VNC (Virtual Network Computing) client function for remotely operating MFP 110. Specifically, mobile terminal 100 has VNC client software that supports the RFB (Remote Frame Buffer) protocol. Mobile terminal 100 establishes a VNC connection to MFP 110, and displays the screen displayed on the operation panel of MFP 110 on mobile terminal 100. MFP 110 has a VNC server function, and when a VNC connection is established from mobile terminal 100, it transmits screen information to mobile terminal 100, which is a VNC client. Mobile terminal 100 draws a screen based on the transmitted screen information. Screens are linked so that the screens displayed on the display units of mobile terminal 100 and MFP 110 change in response to operations accepted on the drawn screen and operations accepted on MFP 110.

[0014] Next, the hardware configuration of the mobile terminal 100 will be described with reference to Fig. 2. Note that although the mobile terminal 100 of this embodiment is assumed to be a device such as a smartphone or a tablet PC, it may be any other device as long as it is an information processing device that can communicate with the MFP 110.

[0015] The CPU 201 reads out a control program stored in the ROM 202 and executes various processes for controlling the operation of the mobile terminal 100. The ROM 202 stores the control program. The RAM 203 is used as a temporary storage area such as the main memory and work area of ​​the CPU 201. The storage 204 stores various data such as photographs and electronic documents, for example, on an eMMC, SSD, or HDD. An operating system (OS) 350 and an MFP application 300, which will be described later, are also stored in the storage 204. A real time clock (RTC) 205 measures time.

[0016] In the case of the mobile terminal 100, one CPU 201 executes each process shown in the flowcharts described below, but other configurations are also possible. For example, multiple CPUs may cooperate to execute each process shown in the flowcharts described below.

[0017] The operation panel 206 has a touch panel function that can detect a user's touch operation, and displays various screens provided by the OS 350 and the MFP application 300. The user can input desired operation instructions to the mobile terminal 100 by inputting touch operations on the operation panel 206.

[0018] The speaker 207 and microphone 208 are used when the user makes a call, for example, with another mobile terminal or a landline phone. The camera 209 takes an image in response to a user's instruction. The photographs taken by the camera 209 are stored in a predetermined area of ​​the storage 204. The wireless communication unit 211 executes wireless communication such as a wireless LAN.

[0019] Next, the software configuration of the mobile terminal 100 will be described with reference to Fig. 3. Fig. 3 is a functional block diagram of software that is implemented when the CPU 201 reads out a control program stored in the ROM 202 or storage 204.

[0020] The OS 350 is software for controlling the overall operation of the mobile terminal 100. Various applications can be installed in the mobile terminal 100, including the MFP application 300 described below. The OS 350 exchanges information with these applications and changes the screen displayed on the operation panel 206 according to instructions received from the applications. The OS 350 also includes a group of device drivers for controlling various hardware, and provides applications running on the OS with APIs for using the various hardware. In this embodiment, the group of device drivers corresponds to, for example, a wireless LAN control unit 351. The wireless LAN control unit 351 is a device driver for controlling the wireless communication unit 211.

[0021] The MFP application 300 is an application installed on the mobile terminal 100. The MFP application can issue print instructions and scan instructions to the MFP 110. Furthermore, as described above, the MFP application also operates as a VNC client, displays the operation screen of the MFP 110, and has the function of remotely operating the MFP 110. Various applications other than the MFP application 300 may be installed on the mobile terminal 100, but a description thereof will be omitted here.

[0022] The configuration of the MFP application 300 will be described in more detail below. The screen control unit 301 controls the screen displayed on the operation panel 206 via the OS 350. The screen control unit 301 displays the MFP application screen on the operation panel 206. The screen control unit 301 also accepts operation instructions input by the user via the operation panel 206. The communication unit 302 controls wireless communication by the wireless communication unit 211 via the OS 350, and transmits and receives information to and from external devices such as the MFP 110.

[0023] The device search control unit 311 generates search data for searching for an external device to which the mobile terminal 100 is connected, and transmits the search data to a device on the LAN 10 via the wireless communication unit 211. The device search control unit 311 then receives a response from the external device that responds to the search. The search data may be broadcast to all external devices on the LAN 10, or may be transmitted to a single external device by specifying its address.

[0024] The print job generation unit 305 generates a print job. The print job generated by the print job generation unit 305 is transmitted to the MFP 110 by the wireless communication unit 211, and the MFP 110 executes printing in accordance with the received print job.

[0025] The scan job control unit 307 sends a scan instruction to the MFP 110 via the wireless communication unit 211, and the MFP 110 executes a scan based on the instruction and sends the data generated by the scan (scan data) to the mobile terminal 100. The mobile terminal 100 displays the received scan data. When the scan data is to be saved, it is stored in the storage unit 306. The stored scan data is managed by the document management unit 308.

[0026] The remote control unit 313 displays a screen based on screen information acquired from the MFP 110 connected via VNC, and accepts operations on the displayed screen. In other words, it operates as a VNC client. Specifically, it displays a screen based on screen information received from the VNC server, and transmits operation information including the position and type of operation accepted on the displayed screen to the MFP 110. Based on the received operation information, the MFP 110 changes the screen displayed on the MFP 110 and the screen displayed on the mobile terminal 100. The application storage unit 306 temporarily stores various information generated by the MFP application 300. The timer unit 304 measures time.

[0027] 4 shows an example of the hardware configuration of the MFP 110. The MFP 110 includes a CPU 3020 that executes programs stored in a ROM 3030 or a large-scale storage device 3110 such as a hard disk, and the CPU 3020 performs overall control of each device connected to a system bus 3010.

[0028] Reference numeral 3040 denotes a RAM, which functions as the main memory and work area of ​​the CPU 3020. Reference numeral 3060 denotes an external input controller (PANELC), which controls instruction input from various hard keys 3120 or a touch panel (PANEL) 3070 provided on the multifunction peripheral. Reference numeral 3080 denotes a display controller (DISPC), which controls the display of a display module (DISPLAY) 3090, which is composed of, for example, an LCD display. Reference numeral 3110 denotes a disk controller (DKC), which controls a large-scale storage device (HD) 3110.

[0029] The operation unit of the MFP 110 will be described using 900 in Fig. 9. A screen is displayed on a display 3090, and hard keys 3120 are arranged next to the display 3090. User operations can be accepted by both the touch panel 3070 on the display 3090 and the hard keys 3120. A logout key 901, which is one of the hard keys 3120, is a button that, when pressed while a user is logged in to the MFP 110, issues a logout processing instruction to the MFP 110.

[0030] Reference numeral 3050 denotes a network interface card (NIC), which exchanges data bidirectionally with external devices (e.g., the mobile terminal 100) or file servers via the network 10. Reference numeral 3400 denotes a wireless communication module, which can connect to an access point or operate in access point mode to operate the MFP 110 as an access point and establish a direct wireless connection with the mobile terminal 100. Reference numeral 3200 denotes a printer, which prints on paper using an electrophotographic method. The printing method does not necessarily have to be electrophotographic. Reference numeral 3300 denotes a scanner, which is an image reading unit for reading images printed on paper. In many cases, the image reading unit 3300 is equipped with an optional ADF (automatic document feeder) (not shown), which can automatically read multiple pages of a document. Note that the large-scale storage device 3110 may also be used as a temporary storage location for images in some cases.

[0031] 5 is a block diagram showing the software configuration of the MFP 110. This software is stored in the ROM 3030, for example, and executed by the CPU 3020.

[0032] The UI processing unit 510 analyzes input from the external input controller 3060 or operation information received from the operation information receiving unit 522 described below, and if a screen change is necessary, generates screen data and displays it on the DISPLAY 3090.

[0033] 9, various screens are displayed on the DISPLAY 3090. For example, a keyboard login screen 910, a menu screen 930 that is displayed during login, an error screen (not shown) that is displayed when an error occurs, a counter screen (not shown) that displays counter information, etc. When the MFP 110 is in sleep mode, the DISPLAY 3090 is not displayed.

[0034] The keyboard login screen 910 is a screen for logging in to the MFP 110 by entering user information using a keyboard. The login process is performed by using the hard keys 3120 or a soft keyboard (described later) to enter information in a user name input field 911 and a password input field 912, and pressing a login button 913. Each user is assigned a role, and the authority to use functions varies depending on the logged-in user. Because the MFP 110 is a multifunction peripheral with multiple functions including at least a print function, it is also possible to configure the functions available to the logged-in user to vary. Pressing the ID card login switch button 914 switches the display to an ID card login screen (not shown).

[0035] Menu screen 930 is a menu screen that is displayed after logging in to MFP 110 via keyboard login screen 910 or ID card login screen. Depending on the application, the user can use various functions of MFP 110 by pressing copy button 931, scan button 932, or print button 933. When the user presses logout button 934, logout processing is performed.

[0036] Remote operation processing unit 520 includes the modules of connection processing unit 523, operation information receiving unit 522, and screen transmission unit 521. Remote operation processing unit 520 is a VNC server module that communicates using the RFB protocol, and is a software module that communicates with mobile terminal 100 operating as a VNC client via a network. Connection processing unit 523 performs connection processing when it receives a connection request from MFP 110 or mobile terminal 100 that is a VNC client, and performs disconnection processing when it receives a disconnection request.

[0037] The operation information receiving unit 522 receives screen operation information from the mobile terminal 100 and the MFP 110, and notifies the UI processing unit 510. The screen sending unit 521 is a display control unit that reads the screen data generated by the UI processing unit 510 from the RAM 3040 and sends the data to the mobile terminal 100, which is a VNC client, thereby controlling the display of the screen.

[0038] The login processing unit 530 is a module that performs login processing for the MFP 110, performs user authentication based on user information etc. received from the UI processing unit, and logs in if the user authentication is successful. Also, if a logout instruction is received from the UI processing unit 510 etc., it performs logout processing. The network control unit 500 is a module for controlling the network I / F 3050. In addition, although not shown, there are also a printer module that controls the printer 3200 and a scanner module that controls the scanner 3300.

[0039] 8 is an example of a screen displayed on operation panel 206 of mobile terminal 100. When mobile terminal 100 is started up, home screen 810 is displayed. MFP application icon 811 is an icon for starting MFP application 300, and when MFP application icon 811 is operated, MFP application 300 is started up.

[0040] When the MFP application 300 is launched, an application home screen (no device selected) 820 is displayed. A selected device icon 821 displays information about the MFP with which the MFP application 300 is currently communicating and the MFP to which the job is to be sent (hereinafter referred to as the target MFP). If the target MFP has not been set, "No Printer" is displayed. A device search button 823 is a button for searching for the target MFP. A remote operation button 822 is a button for connecting to the target MFP via VNC and remotely operating the target MFP. In addition to the above, the application home screen 820 may also display a print button 824 for sending a print job to the target MFP and a button for issuing a scan instruction.

[0041] When device search button 823 is pressed, a search for MFPs present within the network is executed, and the search results are displayed on device search screen 830. A list of MFPs that responded to the search is displayed on device search screen 830. For example, the target MFP is selected by selecting search result 831, which is one of the search results.

[0042] When the target MFP is selected, an application home screen (device selected) 840 is displayed. Information about the MFP selected on the device search screen 830 is displayed on the selected device icon 821. When the target MFP is selected and the user taps the remote operation button 822, the remote operation function of the target MFP is started. Here, the following description will be given assuming that the target MFP is MFP 110.

[0043] When the remote operation function is started, a device connection screen 850 is displayed. When the VNC connection is completed and the remote operation function is established, the screen displayed on the DISPLAY 3090 of the MFP 110 is displayed on the operation panel 206. Here, a keyboard login screen 860, which is the same as the keyboard login screen 910, is displayed. The screen content is the same as the keyboard login screen 910 of the MFP 110, so a description thereof will be omitted. Furthermore, if the MFP 110 was displaying an ID card login screen, the ID card login screen is displayed.

[0044] When a login instruction is given on the keyboard login screen 860, a menu screen (hard keys not shown) 870 that is the same as the menu screen 930 is displayed (the menu screen 930 is also displayed on the DISPLAY 3090 in the same way).

[0045] When the hard key display button 851 is tapped, a menu display screen (hard key display) 875 is displayed. That is, a software keyboard is displayed to which processing for each key of the hard keys 3120 of the MFP 110 is assigned. The hard key UI 877 is a UI for operating the hard keys 3120 of the MFP 110. When a button displayed on the hard key UI 877 is tapped, information about the pressed key is sent to the MFP 110, and the MFP 110 performs processing based on the pressed key. When a logout key 879 corresponding to the logout key 901 of the MFP 110 is pressed, the user logs out of the MFP 110. Furthermore, when the end button 878 on the hard key UI 877 is tapped, the remote operation function is ended, that is, the VNC connection is ended. When the VNC connection is ended, the application home screen (device selected) 840 is displayed on the operation panel 206.

[0046] 10 and 11 will be used to provide an overview of this embodiment. When the keyboard login screen 910 is displayed on the MFP 110, touching a character input field (for example, a user name input field 911) on the DISPLAY 3090 displays the soft keyboard of the MFP 110 on the DISPLAY 3090. In other words, a text input screen 1110 is displayed.

[0047] Consider now the case where the same screen as that displayed on DISPLAY 3090 of MFP 110 is displayed on operation panel 206 of mobile terminal 100, and MFP 110 is remotely operated from mobile terminal 100. For example, when login screen 860 is displayed on mobile terminal 100 (at this time, MFP 110 is displaying keyboard login screen 910), if user name input field 911 is touched on operation panel 206, operation information is transmitted to MFP 110, and MFP 110 performs processing according to the operation information. Here, MFP 110 instructs mobile terminal 100 to display the soft keyboard of mobile terminal 100, and prevents the soft keyboard of MFP 110 from being displayed on DISPLAY 3090. Then, control is performed so that screen information of the screen on which the soft keyboard of MFP 110 is displayed is not transmitted to mobile terminal 100. As a result, the mobile terminal 100 displays a login initial screen 1000 that includes a soft keyboard 1002 of the mobile terminal 100 and a screen 1001 that is the same as the screen displayed on the DISPLAY 3090, while the MFP 110 continues to display the login screen 910. At this time, if the user name input field 911 on the MFP 110 is touched, causing the soft keyboard of the MFP 110 to be displayed on the DISPLAY 3090, two soft keyboards, the soft keyboard of the MFP 110 and the soft keyboard of the mobile terminal 100, are displayed on the operation panel 206 of the mobile terminal 100. This reduces visibility and operability. For example, if the screen displayed on the DISPLAY 3090 is switched from the keyboard login screen 910 to a soft keyboard, that is, to a text input screen 1110, the password input field 912 that was displayed on the keyboard login screen 910 is hidden on the text input screen 1110. In order to avoid reducing visibility and operability, when the MFP 110 is being remotely operated, the MFP 110 does not display the soft keyboard of the MFP 110 even if a character input field (for example, the user name input field 911) is selected, and the screen displaying the soft keyboard of the MFP 110 is not transmitted.Here, the soft keyboard 1002 of the mobile terminal 100 may be a soft keyboard that is commonly used by multiple applications running on the mobile terminal 100, or it may be a soft keyboard included in the operating system of the mobile terminal 100 or a soft keyboard specific to each application.

[0048] 6 is a flowchart showing the processing performed by mobile terminal 100 from establishing a VNC connection with MFP 110 to remotely controlling the MFP 110 until disconnecting the VNC communication. Each step shown in the flowchart in FIG. 6 is processed by CPU 201 loading a control program stored in a memory such as ROM 202 into RAM 203 and executing the program.

[0049] In step S601, when it is detected that icon 811 has been tapped on home screen 810, MFP application 300 is launched. In step S602, screen control unit 301 detects that device search button 823 has been tapped on application home screen 820 (device not selected), and executes an MFP search. In step S603, it is determined whether an MFP has been selected on device search screen 830, and if selected, the screen transitions to application home screen (device selected) 840, and the process proceeds to step S604. If not, the process returns to S603.

[0050] In step S604, it is determined whether or not the remote operation button 822 has been selected on the application home screen (selected) 840. If selected, the process proceeds to step S605; otherwise, the process returns to step S604.

[0051] In step S605, a connection request is made to the MFP selected in S604 using the RFB protocol. Then, in step S606, it is determined whether the connection request has been accepted, and if accepted, the process proceeds to step S607, and if not accepted, the process proceeds to step S619. Whether the connection can be accepted is determined based on the response data of the connection request. In step S619, the screen control unit displays a connection refusal error screen 880, and the process ends.

[0052] In step S607, initialization processing in the RFB protocol is performed. In the initialization processing, the remote control unit 313 sends VNC client information and receives VNC server information. The client information includes the terminal type. The terminal type is obtained from the OS 350 and stored in the RAM 204. The terminal type includes the OS name, OS version, terminal name, etc. In step S608, the remote control unit 313 receives the data sent from the VNC server, that is, the MFP 110.

[0053] In step S609, remote control unit 313 determines whether the data received in step S608 is screen data. If it is screen data, the process proceeds to step S610. If the received data is information instructing the display of the soft keyboard of mobile terminal 100, the process proceeds to step S611.

[0054] In step S610, the screen control unit 301 draws a screen on the operation panel 206 based on the screen data received in step S609. For example, if the screen displayed on the display 3090 is the login screen 910, the login screen 860 is displayed on the operation panel 206. Then, the process proceeds to S612.

[0055] In step S611, the screen control unit 301 displays the soft keyboard of the mobile terminal 100. For example, when the screen displayed on the display 3090 is the login screen 860, touching the user name input field 911 causes the login initial screen 1000 to be displayed on the operation panel 206. Then, the process proceeds to S612.

[0056] In step S612, the screen control unit 301 determines whether or not an input operation from the user has been accepted from the operation panel 206. If accepted, the process proceeds to step S613; if not, the process of S612 is repeated.

[0057] In step S613, the screen control unit 301 determines whether or not an input operation has been performed on the soft keyboard displayed in step S611. If an input operation has been performed on the soft keyboard, the process proceeds to step S614; if not, the process proceeds to step S615.

[0058] In step S614, the screen control unit 301 transmits character input information (ASCII information) corresponding to the characters input on the soft keyboard to the image processing device 110, and proceeds to step S608 each time a character is transmitted. For example, if "u" on the soft keyboard is pressed, "117" is transmitted to the image processing device 110, and if "Backspace" is pressed, "8" is transmitted to the image processing device 110. When "u," "s," "e," "r," or "1" is pressed on the soft keyboard on the login initial screen 1000, an input login screen 1020 is displayed on the operation panel 206. Here, an example has been shown in which character input information is transmitted to the MFP 110 each time a character is input. However, this is not limiting. When a character string is input and the "Confirm key" is selected, the input character input information may be transmitted all at once. In other words, instead of transmitting character information each time a character is input, multiple pieces of character information are transmitted when the "Confirm key" is pressed.

[0059] In step S615, remote control unit 313 determines whether or not connection end button 878 has been pressed. If connection end button 878 has been pressed, the process proceeds to step S618, and if not, the process proceeds to step S616.

[0060] In step S616, if the soft keyboard was displayed in step S611, the soft keyboard is hidden. When the user performs a touch operation on an area other than the soft keyboard and the end button on the login screen 1020 during input, a login screen (input completed) 1030 is displayed on the operation panel 206.

[0061] In step S617, the remote control unit 313 transmits the touch position information to the image processing device 110. Then, the process proceeds to S608.

[0062] In step S618, the remote control unit 313 performs VNC disconnection processing and ends the processing.

[0063] Through the processing of the above flowchart, mobile terminal 100 displays a screen based on the screen information transmitted from MFP 110, and can remotely control MFP 110. Furthermore, when inputting characters during remote operation of MFP 110, the soft keyboard of mobile terminal 100 is displayed, so that the user can input characters to MFP 110 using the soft keyboard that the user is familiar with on mobile terminal 100. Furthermore, in S616 and S617, if an operation is accepted while mobile terminal 100 is displaying the soft keyboard of mobile terminal 100, the soft keyboard is hidden and then touch position information is transmitted to MFP 110. However, if an operation is accepted while mobile terminal 100 is displaying the soft keyboard, the soft keyboard may simply be hidden without transmitting touch position information to MFP 110.

[0064] Next, the processing of MFP 110 when screen linking with mobile terminal 100 via VNC will be described with reference to Fig. 7. Each step shown in the flowchart of Fig. 7 is processed by CPU 3020 expanding a control program stored in a memory such as ROM 3030 into RAM 3040 and executing it.

[0065] In step S701, the connection determination unit 523 determines whether there is a VNC connection request from the mobile terminal 100, which is the VNC client, and if there is a connection request, proceeds to step S702. If there is no connection request, return to step S701. In step S702, the connection determination unit 523 performs negotiation processing to connect with the VNC client. In step S703, the connection determination unit 523 performs initialization processing, receiving client information and sending server information.

[0066] In step S704, the screen transmission unit 521 transmits screen information corresponding to the screen displayed on the DISPLAY 3090 to the mobile terminal 100, which is the VNC client. The processing of S704 is executed every time the screen displayed on the DISPLAY 3090 is updated by a user operation or the like.

[0067] In step S705, it is determined whether or not information has been received from the mobile terminal 100. If so, the process proceeds to step S706; if not, the process returns to step S704.

[0068] In step S706, the operation information receiving unit 522 determines whether the information received from the mobile terminal 100 in step S705 is character input information. If it is character input information, the process proceeds to step S714, and if not, the process proceeds to step S707.

[0069] In step S707, the operation information receiving unit 522 determines whether the touch position information received in step S705 is a touch operation on a character input field (for example, the user name input field 911). If it is determined to be a touch operation on a character input field, the process proceeds to step S708; otherwise, the process proceeds to step S712.

[0070] In step S708, the operation information receiving unit 522 stores information in the RAM 203 so that the operation information receiving unit 522 operates in a character string input mode that accepts character input from the mobile terminal 100.

[0071] In step S709, the MFP 110 transmits a notification to the mobile terminal 100, which is a VNC client, instructing the mobile terminal 100 to display and use its own soft keyboard.

[0072] In step S710, UI processing unit 510 performs control so that the soft keyboard is not displayed on DISPLAY 3090, and screen transmission unit 521 does not transmit screen information of the screen displaying the soft keyboard to mobile terminal 100. In other words, even if user name input field 911 is touched via a terminal connected via VNC, MFP 110 does not display its own software keyboard on DISPLAY 3090 and does not transmit the screen displaying the soft keyboard. Here, an example has been shown in which MFP 110 does not display its own software keyboard on DISPLAY 3090 and does not transmit the screen displaying the soft keyboard, but a configuration may also be used in which its own software keyboard is displayed on DISPLAY 3090 and screen information of the screen displaying the soft keyboard is not transmitted to mobile terminal 100.

[0073] A configuration that does not transmit screen information of a screen displaying a soft keyboard to the mobile terminal 100 may be, for example, a configuration in which even if the screen displayed on DISPLAY3090 is updated to a screen displaying a soft keyboard, the screen information of the updated screen in S704 is not transmitted to the mobile terminal 100, or a configuration in which screen information of a screen not displaying a soft keyboard with the cursor positioned over the selected character input item is transmitted to the mobile terminal 100 in S704.

[0074] In this example, because the MFP 110 is being remotely controlled, the MFP 110 does not display its own software keyboard, but if the user name input field 911 is touched when the MFP 110 is not being remotely controlled, the MFP 110 displays its own software keyboard on DISPLAY 3090. When "user1" is entered on text input screen 1110, DISPLAY 3090 displays a text input screen 1120 in progress. When an OK button 1121 is subsequently pressed, the soft keyboard is hidden and DISPLAY 3090 displays a login screen 1130 with input completed.

[0075] In step S712, the operation information receiving unit 522 executes processing based on the touch position information, operation information, etc. received in S705. The processing here is, for example, processing to transition the screen displayed on the DISPLAY 3090.

[0076] In step S713, the operation information receiving unit 522 disables the character string input mode if it is enabled, and saves the information in the RAM 203. Note that S707 / S712 / S713 are common processes that are executed not only when data is received from a client, but also when a user input is received from the touch panel (PANEL) 3070.

[0077] In step S714, the operation information receiving unit 522 determines whether it is operating in the input character string mode. If it is operating, the process proceeds to step S715, and if not, the process proceeds to step S716.

[0078] In step S715, the UI processing unit 510 inputs a character string into the text field based on the character input information received in S705. For example, if the character input information is "117," "u" is displayed in the text field. Then, a screen in which "u" is displayed in the text field in the screen transfer process of S704 is displayed on the mobile terminal 110. That is, characters input on the mobile terminal 110 are reflected in real time on the DISPLAY 3090, and the mobile terminal 110 displays the reflected screen. At this time, a soft keyboard is not displayed on the DISPLAY 3090 of the MFP 110, but characters input on the mobile terminal 110 are displayed on the DISPLAY 3090. Here, an example is shown in which a character string is input into the text field based on the character input information received in S705. However, even if character input information is received, the character string may not be reflected in the text field, or may be displayed in a display form that is indistinguishable to the user (for example, an * mark). This is because there is a security problem if a character string is displayed on the DISPLAY 3090 of the MFP 110 even when the MFP 110 is being remotely operated.

[0079] In step S716, the operation information receiving unit 522 determines whether or not there is a VNC disconnection request, and if there is a disconnection request, the process proceeds to step S717, and if not, the process returns to step S704.

[0080] In step S717, the connection processing unit 523 performs VNC disconnection processing and terminates the VNC connection.

[0081] The above flowcharts explain how the screen displayed on the mobile terminal changes based on an operation received on the MFP or mobile terminal. However, even if the screen displayed on the MFP changes without receiving an operation from the MFP or mobile terminal, the MFP similarly transmits the changed screen information to the mobile terminal, and the mobile terminal displays the changed screen. Through the processing of the above flowcharts, when a character input item is selected and the MFP is being remotely operated, the MFP can be controlled not to display the MFP's software keyboard and not to transmit screen information of the screen displaying the MFP's software keyboard to the mobile terminal. In other words, when a character input item is selected and the MFP receives the selection operation information, the MFP does not display the MFP's software keyboard. When entering characters using the terminal's own software keyboard on a VNC client terminal, the MFP's software keyboard is hidden, improving the visibility of the screen being remotely operated. If the MFP's software keyboard were displayed even though the terminal's own software keyboard was displayed during a VNC connection, other input items would be hidden. In particular, if a character input item is selected on the text input screen 1110 and the screen transitions to a screen dedicated to the software keyboard, the user will have to go through the trouble of returning to the previous screen.

[0082] In the above example, the software keyboard of the image forming device is hidden during a VNC connection. However, some VNC client devices may not be able to display their own software keyboard. Therefore, the image forming device may hide its own software keyboard only if the VNC client device can display its own software keyboard. For example, if an error occurs that prevents the VNC client device from displaying its own software keyboard, the VNC client terminal notifies the image forming device of this error. Furthermore, whether the VNC client device can display its own software keyboard may be determined from client information received from the VNC client device.

[0083] Example 2 In the first embodiment, a method was described in which an image processing device sends a soft keyboard display instruction to a mobile terminal, and the mobile terminal displays the soft keyboard of the mobile terminal to input characters. However, the method of inputting characters into a mobile terminal is not limited to soft keyboards; some mobile terminals also allow voice input.

[0084] Therefore, in this embodiment, an example is shown in which text is input to an MFP connected via VNC using the voice input function of a mobile terminal. The basic configuration is the same as in the first embodiment, so only the differences are shown.

[0085] 13 is a flowchart showing the processing in which the mobile terminal 100 in this embodiment connects to the image processing device 110 via VNC using the RFB protocol, displays a screen, accepts voice input, and terminates the VNC connection. Each step shown in the flowchart in FIG. 13 is processed by the CPU 201 loading a control program stored in a memory such as the ROM 202 into the RAM 203 and executing it. Furthermore, explanations of the same processing as in FIG. 6 will be omitted.

[0086] In step S1201, the remote control unit 313 waits for voice input. For example, if the screen displayed on the display 3090 is the login screen 910, the login initial screen 1300 is displayed on the operation panel 206. Here, voice input is a function that converts the user's voice input into the microphone (208) into a string of characters.

[0087] In step S1202, remote control unit 313 determines whether or not there has been a voice input. If there has been a voice input, the process proceeds to step S1203, and if not, the process proceeds to step S1215.

[0088] In step S1203, the remote control unit 313 transmits ASCII information corresponding to the characters input by voice to the image processing device 110. For example, if "u" is input, "117" is transmitted. When "u", "s", "e", "r", or "1" is input using the soft keyboard on the login initial screen 1300, an input login screen 1320 is displayed on the operation panel 206.

[0089] In step S1204, the acceptance of voice input is cancelled. When the user performs a touch operation on an area other than the end button on login screen 1320 during input, login screen (input completed) 1030 is displayed on operation panel 206.

[0090] By performing the processing of the above flowchart, it is possible to input characters to a device connected via VNC using the voice input function of the mobile terminal.

[0091] (Other Examples) In the above embodiment, an example has been shown in which the same screen as that displayed by the MFP via a VNC connection is displayed on the mobile terminal, but this is not limiting. For example, the above embodiment can also be applied to a case in which, instead of displaying the same screen as the MFP, an expanded screen of the MFP is displayed on the mobile terminal.

[0092] Furthermore, the present invention can also be realized by executing the following process: That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program. In this case, the computer program and the storage medium storing the computer program constitute the present invention.

Claims

1. an information processing device having a first display unit, a display control means for displaying a first software keyboard on the first display unit when a selection operation of a character input item is accepted on a screen displayed on the first display unit; a transmitting means for transmitting screen information corresponding to the updated screen to a terminal when the screen of the first display unit is updated; Based on the screen information transmitted by the transmission means, the transmission means receives from the terminal information indicating that a selection operation for a character input item has been accepted on a screen displayed on a second display unit of the terminal, and based on the information, the transmission means transmits to the terminal an instruction to display on the second display unit a second software keyboard of the terminal that is different from the first software keyboard, and the display control means performs control so that the first software keyboard is not displayed on the first display unit.

1. An information processing device comprising:

2. Upon receiving operation information corresponding to an operation accepted on a screen displayed on a second display unit of the terminal based on the screen information transmitted by the transmission means, the display control means performs processing corresponding to the operation information, the display control means updates the screen displayed on the first display unit, and the transmission means transmits screen information corresponding to the updated screen to the terminal.

2. The information processing apparatus according to claim 1, wherein:

3. The second software keyboard is a software keyboard used by multiple applications running on the terminal.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. When the terminal is capable of displaying the second software keyboard based on the reception of information from the terminal indicating that a selection operation for a character input item has been accepted on a screen displayed on a second display unit of the terminal based on the screen information transmitted by the transmission means, the transmission means transmits to the terminal screen information corresponding to a screen on which the first software keyboard is not displayed, and when the terminal is not capable of displaying the second software keyboard, the transmission means transmits to the terminal screen information corresponding to a screen on which the first software keyboard is displayed.

4. The information processing device according to claim 1, wherein the information processing device is a computer.

5. The terminal cannot display the second software keyboard when an error occurs in the terminal that prevents the software keyboard from being displayed.

5. The information processing apparatus according to claim 4,

6. The screen displayed on the second display unit of the terminal based on the screen information transmitted by the transmission means is the same as the screen displayed on the first display unit.

6. The information processing device according to claim 1, wherein the information processing device is a computer.

7. The character input item is an item for inputting user information for logging in to the information processing device.

7. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

8. Each time the terminal accepts input of a character via the second software keyboard, the information processing device receives character information indicating the input character and displays a screen on which the character is input on the first display unit.

8. The information processing device according to claim 1, wherein the information processing device is a computer.

9. The screen on which the characters are input is a screen on which the characters are input into a character input field in a state where the user cannot identify the characters.

9. The information processing apparatus according to claim 8,

10. Based on the reception of information indicating that the terminal has accepted character input via the second software keyboard, the information processing device controls so that the screen on which the character is input is not displayed on the first display unit.

8. The information processing device according to claim 1, wherein the information processing device is a computer.

11. The information processing device is an MFP (multifunction peripheral) having multiple functions including a print function.

11. The information processing device according to claim 1,

12. The information processing device operates as a VNC (Virtual Network Computing) server, and the terminal operates as a VNC client.

12. The information processing device according to claim 1, wherein the information processing device is a computer.

13. When the transmitting means receives information from the terminal indicating that a selection operation for a character input item has been accepted on a screen displayed on a second display unit of the terminal based on the screen information transmitted by the transmitting means, the transmitting means does not transmit screen information corresponding to a screen displaying the first software keyboard to the terminal.

13. The information processing device according to claim 1, wherein the information processing device is a computer.

14. A system including the information processing device according to claim 1 and the terminal, A system characterized in that, when the information processing device sends an instruction to the terminal to display the second software keyboard and controls the first display unit so that the first software keyboard is not displayed, the terminal displays a screen on the second display unit where the first software keyboard is not displayed and where the second software keyboard is displayed.

15. When the second software keyboard is displayed on the second display unit of the terminal, even if the terminal receives an operation on an area of ​​the screen that is not the second software keyboard on the second display unit based on the screen information transmitted by the transmission means, the terminal does not transmit information indicating the operation to the information processing device.

15. The system of claim 14.

16. A control method for an information processing device having a first display unit, comprising: a display control step of displaying a first software keyboard on the first display unit when a selection operation for a character input item is accepted on a screen displayed on the first display unit; a transmission step of transmitting screen information corresponding to the updated screen of the first display unit to a terminal when the screen of the first display unit is updated; Based on the reception of information from the terminal indicating that a selection operation for a character input item has been accepted on a screen displayed on a second display unit of the terminal based on the screen information transmitted in the transmission step, the transmission step transmits to the terminal an instruction to display on the second display unit a second software keyboard of the terminal that is different from the first software keyboard, and the display control step performs control so that the first software keyboard is not displayed on the first display unit. A control method comprising:

17. A program for causing a computer to function as each of the means according to any one of claims 1 to 13.

Images