Information processing device, information processing control method, and computer program

Abstract

To solve the problem that, when an image processing device and a terminal is displaying a screen in corporation, processing not intended by a user can be possibly performed if the terminal receives user manipulation while a magnified screen is displayed only by the terminal, and information representing the manipulation is transmitted to the image processing device as it is.SOLUTION: When a terminal receives user manipulation while a magnified screen is being displayed only by the terminal, the terminal does not transmit the manipulation information to an image processing device as it is.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] It relates to a remote control method. [Background technology]

[0002] A method is disclosed in which the operation screen of the image processing device can be operated from a PC or a mobile terminal by using VNC (Virtual Network Computing) using RFB (Remote Frame Buffer) protocol to remotely operate the operation screen of the image processing device. Specifically, an image processing device having a VNC server function transmits screen information of the operation screen to a terminal having a VNC client function. The terminal displays the same screen as the operation screen displayed on the operation panel of the image processing device based on the received screen information. When a user operates the screen displayed on the terminal, information indicating the operation is transmitted from the terminal to the image processing device, and the image processing device performs processing such as screen transition based on the transmitted information and transmits the changed screen information to the terminal. In this way, by performing operations on the screen displayed on the terminal, the user can log in on the operation screen provided by the image processing device or cause the image processing device to execute processing without operating the operation panel of the image processing device. Patent Document 1 describes that when the screens are displayed in cooperation between the terminal and the image processing device, if an operation that is acceptable on the terminal but cannot be accepted by the image processing device is accepted on the terminal, the terminal invalidates the information indicating the operation without transmitting it to the image processing device. Furthermore, it is disclosed that when the terminal receives an operation that can be received by either device, information indicating the operation is transmitted to the image processing device to validate the operation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-6668 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, even if the terminal accepts an operation that can be accepted by both the terminal and the image processing device, depending on the state in which the terminal displays the screen, a problem may occur if the terminal transmits information indicating the operation to the image processing device as is. For example, consider a case in which both the terminal and the image processing device can accept a drag operation and are controlled to perform a process of transitioning the screen when the drag operation is accepted. In a case in which the terminal and the image processing device are linked to each other and the screen is enlarged only on the terminal, when a drag operation is accepted on the terminal, the screen is enlarged on the terminal, so the user's drag operation is intended to move the enlarged display area. However, in this case, if the terminal accepts a drag operation on the terminal and transmits information indicating the drag to the image processing device as is, the screen will transition and a process different from the user's intention will occur. Therefore, an object of the present invention is to provide a method that can perform a process intended by the user based on an operation accepted from the user, taking into consideration the state of the screen displayed by the terminal, when the screen is linked between the terminal and the image processing device. [Means for solving the problem]

[0005] A program for causing a computer of an information processing device that communicates with an image processing device to execute a control method, the control method comprising: a display step of displaying, on the display unit of the information processing device, a screen corresponding to a screen displayed on a display unit of the image processing device and based on display control of the image processing device; The above When a predetermined user operation on the screen is received, a screen based on information corresponding to the predetermined user operation is displayed on the image processing device, and the screen displayed in the display step is The above When the predetermined user operation is accepted in a state where the display form of the screen has been changed, a screen based on information corresponding to the predetermined user operation is displayed on the image processing device. The state in which the display form of the screen displayed in the display step is changed means a state in which the magnification ratio of the screen displayed in the display step is changed. A program characterized by: Effect of the Invention

[0006] Taking into account the state of the screen displayed by the terminal, it becomes possible to carry out the processing intended by the user based on the operations received from the user. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating a system configuration. [Diagram 2] FIG. 2 is a diagram showing a hardware configuration of a mobile terminal 100. [Diagram 3] FIG. 2 is a diagram showing a software configuration of the mobile terminal 100. [Figure 4] FIG. 2 is a diagram illustrating a hardware configuration of an MFP 110. [Diagram 5] FIG. 2 is a diagram illustrating a software configuration of the MFP 110. [Figure 6] 4 is a flowchart showing the process of the mobile terminal 100 of the present embodiment. [Figure 7] 4 is a flowchart showing the process of the MFP 110 of the present embodiment. [Figure 8] 4 is a flowchart showing the process of the mobile terminal 100 of the present embodiment. [Figure 9] 1A and 1B are diagrams illustrating a UI displayed on an operation panel of a mobile terminal 100 according to the present embodiment. [Figure 10] 1A and 1B are diagrams showing UIs displayed on the operation panels of mobile terminal 100 and MFP 110 of this embodiment. [Figure 11] 1A and 1B are diagrams showing UIs displayed on the operation panels of mobile terminal 100 and MFP 110 of this embodiment. [Figure 12] 1A and 1B are diagrams showing UIs displayed on the operation panels of mobile terminal 100 and MFP 110 of this embodiment. [Figure 13] 1A and 1B are diagrams showing UIs displayed on the operation panels of mobile terminal 100 and MFP 110 of this embodiment. [Figure 14] 1A and 1B are diagrams showing UIs displayed on the operation panels of mobile terminal 100 and MFP 110 of this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The best mode for carrying out the present invention will be described below with reference to the drawings. Note that the following embodiment does not limit the invention according to the claims, and all of the combinations of features described in the embodiment are not necessarily essential to the solution of the invention.

[0009] Example 1 <System Configuration in this Embodiment> First, the system configuration according to the present embodiment will be described with reference to Fig. 1. Here, the description will be given taking a mobile terminal 100 and an MFP 110 as an example of an information processing device, but the devices included in the information processing system are not limited to this. The mobile terminal 100 and the MFP 110 are connected to a LAN (Local Area Network) and can exchange information with each other. The mobile terminal 100 is connected to the LAN via an access point 120.

[0010] Mobile terminal 100 has a VNC (Virtual Network Computing) client function for remotely operating MFP 110. Specifically, mobile terminal 100 has VNC client software compatible with RFB (Remote Frame Buffer) protocol. Mobile terminal 100 makes a VNC connection to MFP 110, and displays a screen displayed on an operation panel of MFP 110 on mobile terminal 100. MFP 110 has a VNC server function, and when a VNC connection is made from mobile terminal 100, it transmits screen information to mobile terminal 100, which is a VNC client. Mobile terminal 100 draws a screen based on the transmitted screen information. Screens are linked so that the screens displayed on the display units of mobile terminal 100 and MFP 110 are changed in accordance with operations accepted on the drawn screen and operations accepted on MFP 110.

[0011] Next, a hardware configuration of the mobile terminal 100 will be described with reference to Fig. 2. Note that, although the mobile terminal 100 of this embodiment is assumed to be a device such as a smartphone or a tablet PC, it may be any other device as long as it is an information processing device capable of communicating with the MFP 110.

[0012] The CPU 201 reads out a control program stored in the ROM 202 and executes various processes for controlling the operation of the mobile terminal 100. The ROM 202 stores the control program. The RAM 203 is used as a temporary storage area such as the main memory and work area of ​​the CPU 201. The storage 204 stores various data such as photos and electronic documents, for example, in an embedded multimedia card (eMMC), a solid state disk (SSD), or a hard disk drive (HDD). An operating system (OS) 350 and an MFP application 300, which will be described later, are also stored in the storage 204. A real time clock (RTC) 205 keeps track of time.

[0013] In the case of the mobile terminal 100, one CPU 201 executes each process shown in the flowcharts described later, but other configurations are also possible. For example, multiple CPUs may cooperate to execute each process shown in the flowcharts described later.

[0014] Operation panel 206 has a touch panel function capable of detecting a user's touch operation, and displays various screens provided by OS 350 and MFP application 300. The user can input a desired operation instruction to mobile terminal 100 by inputting a touch operation to operation panel 206.

[0015] The speaker 207 and microphone 208 are used when the user makes a call, for example, with another mobile terminal or a landline phone. The camera 209 captures images in response to an image capture instruction from the user. Photos captured by the camera 209 are stored in a predetermined area of ​​the storage 204. The wireless communication unit 211 executes wireless communication such as a wireless LAN.

[0016] Next, the software configuration of the mobile terminal 100 will be described with reference to Fig. 3. Fig. 3 is a functional block diagram of software that is realized when the CPU 201 reads out a control program stored in the ROM 202 or storage 204.

[0017] The OS 350 is software for controlling the overall operation of the mobile terminal 100. Various applications can be installed in the mobile terminal 100, including the MFP application 300 described later. The OS 350 exchanges information with these applications, and changes the screen displayed on the operation panel 206 according to instructions received from the applications. The OS 350 also includes a group of device drivers for controlling various hardware, and provides an API for utilizing various hardware to applications running on the OS. In this embodiment, the group of device drivers corresponds to, for example, a wireless LAN control unit 351. The wireless LAN control unit 351 is a device driver for controlling the wireless communication unit 211.

[0018] The MFP application 300 is an application installed in the mobile terminal 100. The MFP application can issue print and scan instructions to the MFP 110. Furthermore, as described above, the MFP application also operates as a VNC client, displays the operation screen of the MFP 110, and has the function of remotely operating the MFP 110. Various applications other than the MFP application 300 may be installed in the mobile terminal 100, but a description thereof will be omitted here.

[0019] The configuration of the MFP application 300 will be described in more detail. The screen control unit 301 controls the screen to be displayed on the operation panel 206 via the OS 350. The screen control unit 301 displays the screen of the MFP application on the operation panel 206. The screen control unit 301 also accepts operation instructions input by the user via the operation panel 206. The communication unit 302 controls wireless communication by the wireless communication unit 211 via the OS 350, and transmits and receives information to and from external devices such as the MFP 110.

[0020] The device search control unit 311 generates search data for searching for an external device to which the mobile terminal 100 is connected, and transmits the search data to a device on the LAN 10 by the wireless communication unit 211. Then, it receives a response from the external device that responds to the search. The search data may be broadcast to all external devices on the LAN 10, or may be transmitted to a single external device by specifying its address.

[0021] A print job generating unit 305 generates a print job. The print job generated by the print job generating unit 305 is transmitted to the MFP 110 by the wireless communication unit 211, and the MFP 110 executes printing according to the received print job.

[0022] The scan job control unit 307 transmits a scan instruction to the MFP 110 via the wireless communication unit 211, and the MFP 110 executes a scan based on the instruction and transmits data generated by the scan (scan data) to the mobile terminal 100. The mobile terminal 100 displays the received scan data. When the scan data is to be saved, it is stored in the storage unit 306. The stored scan data is managed by the document management unit 308.

[0023] The remote operation unit 313 displays a screen based on screen information acquired from the MFP 110 connected via VNC, and accepts operations on the displayed screen. In other words, it operates as a VNC client. Specifically, it displays a screen based on screen information received from the VNC server, and transmits operation information including the position of the operation accepted on the displayed screen and the type of operation to the MFP 110. Based on the received operation information, the MFP 110 changes the screen displayed on the MFP 110 and the screen displayed by the mobile terminal 100. The application storage unit 306 temporarily stores various information generated by the MFP application 300. The clock unit 304 counts time.

[0024] 4 shows an example of the hardware configuration of the MFP 110. The MFP 110 includes a CPU 3020 that executes programs stored in a ROM 3030 or a large-scale storage device 3110 such as a hard disk, and the CPU 3020 performs overall control of each device connected to a system bus 3010.

[0025] Reference numeral 3040 denotes a RAM which functions as the main memory, work area, etc. of the CPU 3020. Reference numeral 3060 denotes an external input controller (PANELC) which controls instruction input from various hard keys 3120 or a touch panel (PANEL) 3070 provided on the multifunction peripheral. Reference numeral 3080 denotes a display controller (DISPC) which controls the display of a display module (DISPLAY) 3090 which is constituted by, for example, an LCD display. Reference numeral 3110 denotes a disk controller (DKC) which controls a large-scale storage device (HD) 3110.

[0026] The operation unit of the MFP 110 will be described using 1000 in Fig. 10. A screen is displayed on a DISPLAY 3090, and hard keys 3120 are arranged next to the DISPLAY 3090. User operations can be accepted by both the touch panel 3070 on the DISPLAY 3090 and the hard keys 3120. A logout key 1001, which is one of the hard keys 3120, is a button for issuing an instruction for logout processing to the MFP 110 when the logout key 1001 is pressed while a user is logged in to the MFP 110.

[0027] Returning to the explanation of FIG. 4, 3050 is a network interface card (NIC) that exchanges data bidirectionally with an external device (e.g., the mobile terminal 100) or a file server via the network 10. 3400 is a wireless communication module that can connect to an access point or operate in an access point mode to operate the MFP 110 as an access point and directly connect to the mobile terminal 100 via wireless communication. 3200 is a printer that is a printing unit that is realized by an electrophotographic method. The printing method does not have to be the electrophotographic method. 3300 is a scanner that is an image reading unit that reads images printed on paper. In many cases, the image reading unit 3300 is equipped with an ADF (automatic document feeder) (not shown) as an option, and can automatically read multiple sheets of originals. In some cases, the large-scale storage device 3110 may also be used as a temporary storage location for images.

[0028] 5 is a block diagram showing the software configuration of the MFP 110. This software is stored in the ROM 3030, for example, and executed by the CPU 3020.

[0029] The UI processing unit 510 analyzes input from the external input controller 3060 or operation information received from an operation information receiving unit 522 (described later), and if a screen change is necessary, generates screen data and displays it on the DISPLAY 3090 .

[0030] 10, various screens are displayed on the DISPLAY 3090. For example, an ID card login screen 1020, a keyboard login screen 1010, a menu screen 1030 that is displayed during login, an error screen (not shown) that is displayed when an error occurs, a counter screen (not shown) that displays counter information, etc. When the MFP 110 is in sleep mode, the DISPLAY 3090 is in a non-display state.

[0031] The ID card login screen 1020 is a screen for logging in to the MFP 110 with an ID card associated with user information, and by holding the ID card over the ID card reader of the MFP 110 while this screen is displayed, the user information is read and login processing is performed. When a keyboard login screen switching button 1021 is pressed, the display switches to the keyboard login screen 1010.

[0032] The keyboard login screen 1010 is a screen for logging in to the MFP 110 by inputting user information using a keyboard. The login process is performed by using the hard keys 3120 or a soft keyboard (not shown) to input information in a user name input field 1011 and a password input field 1012, and pressing a login button 1013. Pressing an ID card login switch button 1014 switches the display to an ID card login screen 1020.

[0033] Menu screen 1030 is a menu screen that is displayed after logging in to MFP 110 via keyboard login screen 1010 or ID card login screen 1020. The user can use various functions of MFP 110 by pressing copy button 1031, scan button 1032, or print button 1033 depending on the purpose. It is assumed that the menu screen is made up of multiple pages. When the user presses logout button 1034, logout processing is performed. Next button 1036 is a button that transitions the menu screen to the next page. Previous button 1035 is a button that transitions the menu screen to the previous page. Page transitions are also possible by dragging, flicking, and the like.

[0034] Remote operation processing unit 520 includes the modules of connection processing unit 523, operation information receiving unit 522, and screen transmission unit 521. Remote operation processing unit 520 is a VNC server module that communicates by the RFB protocol, and is a software module that communicates with mobile terminal 100 operating as a VNC client via a network. Connection processing unit 523 performs connection processing when a connection request is received from MFP 110 or mobile terminal 100 that is a VNC client, and performs disconnection processing when a disconnection request is received.

[0035] An operation information receiving unit 522 receives screen operation information from the mobile terminal 100 and the MFP 110, and notifies the UI processing unit 510. The screen sending unit 521 is a display control unit that reads screen data generated by the UI processing unit 510 from the RAM 3040, and sends the data to the mobile terminal 100, which is a VNC client, thereby controlling the display of the screen.

[0036] The login processing unit 530 is a module that performs login processing of the MFP 110, performs user authentication based on user information etc. received from the UI processing unit, and performs login if the user authentication is successful. Also, when a logout instruction is received from the UI processing unit 510 etc., it performs logout processing. The network control unit 500 is a module for controlling the network I / F 3050. In addition, although not shown, there are also a printer module that controls the printer 3200 and a scanner module that controls the scanner 3300.

[0037] 9 is an example of a screen displayed on operation panel 206 of mobile terminal 100. When mobile terminal 100 is started up, a home screen 910 is displayed. MFP application icon 911 is an icon for starting MFP application 300, and when MFP application icon 911 is operated, MFP application 300 is started.

[0038] When the MFP application 300 is launched, an application home screen (device not selected) 920 is displayed. A selected device icon 921 displays information about the MFP with which the MFP application 300 is currently communicating and the MFP to which a job is to be sent (hereinafter, the target MFP). If the target MFP has not been set, "No Printer" is displayed. A device search button 923 is a button for searching for the target MFP. A remote operation button 922 is a button for connecting to the target MFP via VNC and remotely operating the target MFP. In addition to the above, a print button 924 for sending a print job to the target MFP and a button for issuing a scan instruction may also be displayed on the application home screen 820.

[0039] When the device search button 923 is pressed, a search for MFPs present in the network is executed, and the search results are displayed on a device search screen 930. A list of MFPs that responded to the search is displayed on the device search screen 930. For example, the target MFP is selected by selecting a search result 931, which is one of the search results.

[0040] When the target MFP is selected, an application home screen (device selected) 940 is displayed. Information about the MFP selected on device search screen 930 is displayed on selected device icon 921. When the user taps remote operation button 922 with the target MFP selected, the remote operation function of the target MFP is started. Here, the following description will be given assuming that the target MFP is MFP 110.

[0041] When the remote operation function is started, a device connection screen 950 is displayed. When a hard key display button 951 is pressed, a UI for operating the hard keys 3120 of the MFP 110 is displayed. Details will be described later.

[0042] When the VNC connection is completed and the remote operation function is established, the screen displayed on the DISPLAY 3090 of the MFP 110 is displayed on the operation panel 206. Here, a keyboard login screen 960, which is the same screen as the keyboard login screen 1010, is displayed. Other than the fact that a hard key display button 951 is displayed on the operation panel 206, the screen contents are the same as the keyboard login screen 1010 of the MFP 110, so a description thereof will be omitted. Furthermore, if the MFP 110 was displaying the ID card login screen 1020, an ID card login screen 965 is displayed.

[0043] When a login instruction is given on the keyboard login screen 960, a menu screen (without hard keys displayed) 970, which is the same as the menu screen 1030, is displayed (similarly, the menu screen 1030 is also displayed on the DISPLAY 3090).

[0044] When the hard key display button 951 is tapped, a menu display screen (hard key display) 975 is displayed. That is, a software keyboard to which the processing of each key of the hard key 3120 of the MFP 110 is assigned is displayed. The hard key UI 977 is a UI for operating the hard key 3120 of the MFP 110. When a button displayed on the hard key UI 977 is tapped, the pressed key information is transmitted to the MFP 110, and the MFP 110 performs processing based on the pressed key. When a logout key 979 corresponding to the logout key 1001 of the MFP 110 is pressed, the user logs out from the MFP 110. When an end button 978 on the hard key UI 977 is tapped, the remote operation function is ended, that is, the VNC connection is ended. When the VNC connection is ended, the application home screen (device selected) 940 is displayed on the operation panel 206. When the VNC connection is ended, the DISPLAY 3090 continues to display the screen that was displayed before the VNC connection was ended.

[0045] 11 shows an example of the screens of mobile terminal 100 and MFP 110 when a drag operation is performed by the user while menu screen 970 is displayed on mobile terminal 100. Menu display screen (before dragging) 1110 is a menu screen displayed on mobile terminal 100 based on screen information transmitted from MFP 110. Menu display screen (before dragging) 1110 is a default screen in which the entire screen is displayed based on screen information received from the MFP, that is, no enlarged display has been performed. This is a state (default screen state) in which the screen transmitted from the MFP is displayed as is, so as to fit the horizontal and vertical lengths of operation panel 206 of mobile terminal 100 according to the size of the panel.

[0046] When a drag operation 1111 is performed on the menu display screen (before dragging) 1110, operation information indicating the drag is transmitted from the mobile terminal 100 to the MFP 110, and the mobile terminal 100 displays the menu display screen (after dragging) 1120, which is the menu screen of the next page. At this time, the MFP 110 also displays the menu screen 1130 (after dragging), which is the next page, based on the operation information received from the mobile terminal 100. The menu display screen (after dragging) 1120 displays a fax button 1121, a box button 1122, or a scan and send button 1123, and when pressed, transitions to a setting screen for the corresponding process, etc. Similarly, when a drag operation is received on the MFP 110 while the menu screen 1030 is displayed on the MFP 110, the menu display screen 1120 (after dragging) is displayed on the mobile terminal 100, and the menu screen 1130 (after dragging) is displayed on the MFP 110. Here, a process of transitioning pages is assigned when a drag operation is received from a user while menu screen 103 is being displayed on MFP 110. Therefore, even when MFP 110 is being remotely operated on mobile terminal 100, a page transition can be performed when a drag operation is received on mobile terminal 100.

[0047] FIG. 12 shows the screens of mobile terminal 100 and MFP 110 when a drag operation is performed in a state where menu screen 970 is displayed in an enlarged manner on mobile terminal 100. In FIG.

[0048] Menu display screen (enlarged display, before dragging) 1210 is a screen that is displayed when a screen enlargement operation such as pinch out or double tap is received from the user on mobile terminal 100 displaying menu display screen (before dragging) 1110. Here, it is assumed that operation information for enlarging the screen is not transmitted from mobile terminal 100 to MFP 110. This is because the process of enlarging the screen while menu screen 103 is being displayed is not assigned on MFP 110. Therefore, when a screen enlargement operation is received from the user on mobile terminal 100 displaying menu display screen (before dragging) 1110, an enlarged menu screen screen (1210) is displayed on mobile terminal 100, and a non-enlarged screen (1030) is displayed on MFP 110.

[0049] When a drag operation 1211 is received from the user on the menu display screen (enlarged display, before dragging) 1210, the enlarged display area is moved on the mobile terminal 100, and a menu display screen (enlarged display, after dragging) 1220 after the movement is displayed. At this time, the mobile terminal 100 does not transmit operation information indicating the drag operation 1211 to the MFP 110. Therefore, the MFP 110 keeps displaying a menu screen (after dragging) 1230 similar to the menu screen 1030. When the user drags while the menu screen is enlarged on the mobile terminal 100, it means that the user wants to move the enlarged display area in the drag direction. Therefore, the mobile terminal 100 does not transmit operation information to the MFP 110, and performs a process to change the display area on its own device. If the user accepts a drag while the menu screen is enlarged on the mobile terminal 100 and transmits operation information indicating the drag to the MFP 110, the MFP 110 assigns a process to transition pages to the drag operation, and the page of the menu screen transitions, resulting in a process different from the user's intention. Therefore, even if an operation to change pages, such as dragging, is received when the screen is enlarged on mobile terminal 100, the operation information is not transmitted to MFP 110, and the display is changed on the mobile terminal itself, so that the processing intended by the user can be performed. Even when the screen is enlarged on mobile terminal 100, if the positions corresponding to previous button 1035 or next button 1036 on mobile terminal 100 are pressed, the page can be changed.

[0050] Fig. 6 is a flowchart showing the processing when mobile terminal 100 connects to MFP 110 via VNC and the user remotely controls MFP 110. Each step shown in the flowchart in Fig. 6 is processed by CPU 201 loading a control program stored in a memory such as ROM 202 into RAM 203 and executing it.

[0051] In step S601, when it is detected that an icon 911 has been tapped on the home screen 910, the MFP application 300 is launched. In step S602, the screen control unit 301 detects that a device search button 923 has been tapped on the application home screen 920 (device not selected) and executes an MFP search. In step S603, it is determined whether an MFP has been selected on the device search screen 930, and if so, the screen transitions to the application home screen (device selected) 940 and the process proceeds to step S604. If not, the process returns to S603.

[0052] In step S604, it is determined whether or not the remote operation button 922 has been selected on the application home screen (device selected) 940. If selected, the process proceeds to step S605; otherwise, the process returns to step S604.

[0053] In step S605, a connection request is made to the MFP selected in S604 using the RFB protocol. Then, in step S606, it is determined whether the connection request has been accepted, and if accepted, the process proceeds to step S607, and if not accepted, the process proceeds to step S620. The acceptance of the connection is determined based on the response data of the connection request. In step S620, the screen control unit displays a connection refusal error screen 980, and the process ends.

[0054] In step S607, an initialization process in the RFB protocol is performed. In the initialization process, the remote control unit 313 transmits VNC client information and receives VNC server information.

[0055] In step S608, the remote operation unit 313 receives the screen information transmitted from the MFP 110, that is, the VNC server. In S609, the screen control unit 301 draws a screen on the operation panel 206 based on the received screen information and the screen information generated by the screen control unit 301. For example, if the screen displayed on the DISPLAY 3090 is the keyboard login screen 1010, the keyboard login screen 960 is displayed on the operation panel 206. Also, each time screen information is transmitted from the MFP 110, a screen is displayed in S609.

[0056] In step S610, the remote control unit 313 determines whether or not an input operation has been accepted from the operation panel 206. If accepted, the process proceeds to step S611; if not, the process of S610 is repeated.

[0057] In step S611, the screen control unit 301 determines whether the operation received in step S610 is an operation to enlarge the screen. An operation to enlarge the screen is, for example, a pinch out or a double tap. If it is an enlargement operation, the operation information is not transmitted to the MFP 110, and the process proceeds to S609, where the screen control unit 301 displays the enlarged screen. If it is determined in step S611 that it is not an enlargement operation, the process proceeds to S612.

[0058] In S612, the screen control unit 301 determines whether the operation received in step S610 is a drag operation. If it is a drag operation, the process proceeds to S613, and if not, the process proceeds to S614. In S613, the process determines whether the screen displayed on the operation panel 206 is enlarged. If it is determined that it is enlarged, the process proceeds to S630, and if not, the process proceeds to S614. For example, if the menu display screen 1210 is displayed on the operation panel 206, it is determined that it is enlarged. That is, here, it is determined whether the screen received from the MFP 110 is enlarged from the state in which it is displayed as it is, that is, in the default state. In S630, the screen control unit 301 calculates the area to be displayed from the current enlargement ratio (enlargement ratio based on the default screen display state) and the movement amount of the received drag operation. Then, the process proceeds to S609, and the screen control unit 301 displays the screen in which the area to be enlarged is changed. At this time, operation information indicating dragging is not transmitted to the MFP 110.

[0059] In S614, the remote operation unit 313 transmits operation information corresponding to the operation accepted in step S610 to the MFP 110. If the operation accepted in S610 is to stop the remote operation, that is, to end the VNC connection, the remote operation unit 313 transmits a request to disconnect the VNC connection to the MFP 110.

[0060] According to the above flowchart, when the MFP is remotely operated from a mobile terminal, even if a drag operation is received on the mobile terminal while the screen is enlarged on the mobile terminal, operation information indicating a drag is not sent to the MFP. Therefore, it is possible to prevent page transitions that are not intended by the user.

[0061] When a drag operation is accepted, if the screen is enlarged ("Yes" in S613), the display is changed only on the mobile terminal, and if the screen is not enlarged ("No" in S613), operation information indicating the drag is sent to the MFP. In other words, whether or not to send operation information to the MFP is switched depending on whether the screen is enlarged. However, this control is not limited to whether or not the screen is enlarged. For example, whether or not to send operation information to the MFP may be switched depending on whether the mobile terminal is reducing and displaying a screen that is displayed based on screen information sent from the MFP. In other words, whether or not to send operation information to the MFP is switched depending on the state of the screen displayed on the mobile terminal.

[0062] Here, an example has been shown in which when an enlargement operation is accepted in S611, the mobile terminal 100 does not transmit operation information to the MFP 110, but operation information may be sent instead. In this case, since the MFP does not have a process corresponding to the received operation information, it does not process the operation information and invalidates it. Also, an example has been shown in which when an enlargement operation is accepted on the mobile terminal in S611, the enlarged screen is displayed only on the mobile terminal without transmitting operation information to the MFP. However, not limited to the enlargement operation, when an operation such as a pinch-in operation to reduce the screen is accepted, the reduced screen may be displayed only on the mobile terminal without transmitting operation information to the MFP.

[0063] Furthermore, although a drag operation has been given as an example of a page switching operation, other operations involving movement of a pointer, such as a flick or swipe operation, may also be used.

[0064] Next, the processing of MFP 110 when screen linking with mobile terminal 100 by VNC will be described with reference to Fig. 7. Each step shown in the flowchart of Fig. 7 is processed by CPU 3020 expanding a control program stored in a memory such as ROM 3030 into RAM 3040 and executing it.

[0065] In step S701, the connection determination unit 523 determines whether there is a VNC connection request from the mobile terminal 100, which is a VNC client, and proceeds to step S702 if there is a connection request. If there is no connection request, the process returns to step S701. In step S702, the connection determination unit 523 performs negotiation processing to connect to the VNC client. In step S703, the connection determination unit 523 performs initialization processing to receive client information and transmit server information.

[0066] In step S704, the screen transmission unit 521 transmits the screen displayed on the DISPLAY 3090 to the mobile terminal 100 which is a VNC client.

[0067] In step S705, the operation information receiving unit 522 determines whether or not an operation has been received from the user. If an operation has been received, the process proceeds to step S706, and if not, the process returns to step S705. For example, if operation information is received from the mobile terminal 100, or if a user operation is detected on the panel 3070 or hard key 3120 of the MFP, it is determined that an operation has been received from the user.

[0068] In S706, it is determined whether the operation accepted in S705 is an operation to disconnect the VNC connection. If it is a disconnection operation, the process proceeds to S707, where the VNC connection is disconnected, and if not, the process proceeds to S708.

[0069] In S708, processing is performed according to the operation accepted in S705. For example, this includes processing for transitioning the menu screen displayed on DISPLAY 3090 to the next page, processing for logging the user into the MFP 110, and processing for logging the user out. Next, in S709, the operation information receiving unit 522 determines whether the screen displayed on DISPLAY 3090 has been changed according to the processing performed in S708. If it has been changed, the process proceeds to S704, and if not, the process proceeds to S705.

[0070] Through the above processing, the screen displayed on the MFP can also be displayed on the mobile terminal. The screens displayed on the MFP and the mobile terminal can be changed based on an operation received on the MFP or the mobile terminal. In the above flow chart, the screen displayed on the mobile terminal is changed based on an operation received on the MFP or the mobile terminal, but even if the screen displayed on the MFP is changed without receiving an operation on the MFP or the mobile terminal, the MFP similarly transmits the changed screen information to the mobile terminal, and the mobile terminal displays the changed screen.

[0071] According to the above embodiment, when a screen is displayed on a mobile terminal based on screen information transmitted from the MFP in order to remotely operate the MFP on the mobile terminal, if the screen received from the MFP is not displayed on the mobile terminal as is (for example, enlarged), even if the mobile terminal accepts a predetermined operation for page transition such as a drag operation, the operation information can be prevented from being transmitted to the MFP. If the accepted operation information is transmitted to the MFP in a state in which the screen received from the MFP is not displayed on the mobile terminal as is, there is a possibility that processing unintended by the user will be performed. For example, if a drag operation is accepted when the screen received from the MFP is enlarged and information indicating the drag operation is transmitted to the MFP, a page transition will occur. Here, the user intends to move the display area by the drag operation, so if the operation information is transmitted to the MFP, processing different from the user's intention will be performed. The processing of this embodiment can prevent processing different from the user's intention from being performed.

[0072] In the above embodiments, the mobile terminal judges whether to transmit operation information to the MFP depending on the screen state of the mobile terminal itself. However, the present invention is not limited to this, and the MFP may grasp the display state of the mobile terminal, and upon receiving operation information from the mobile terminal, the MFP may take the display state of the mobile terminal into consideration and judge whether to perform processing, and switch.

[0073] Example 2 In the first embodiment, even if an operation of page transition such as dragging is accepted when the screen is enlarged on the mobile terminal, the mobile terminal does not transmit operation information indicating the operation to the MFP. However, even if the screen is enlarged on the mobile terminal, operation information indicating an operation such as tapping or pressing may be transmitted to the MFP. Therefore, in the present embodiment, an example is shown in which an operation such as tapping or pressing is accepted and operation information is transmitted to the MFP even if the screen is enlarged on the mobile terminal. The basic configuration of this embodiment is the same as that of the first embodiment, so only the differences are shown.

[0074] Fig. 8 is a flowchart showing the process of the mobile terminal 100 in this embodiment. Since the basic configuration is the same as that of the flowchart in Fig. 6 in the first embodiment, only the differences are shown.

[0075] In S801, the screen control unit 301 determines whether the operation received in S610 is a tap operation, and if it is a tap operation, the process proceeds to S802, where the screen control unit 301 determines whether the screen is enlarged, and if it is not a tap operation, the process proceeds to S614.

[0076] If it is determined in S802 that the screen is enlarged, the process proceeds to S803, where the screen control unit 301 calculates the tap position based on the current screen magnification ratio, the enlarged portion, and the tapped position information. Then, in S614, operation information indicating the calculated position coordinates is transmitted to the MFP 110. For example, based on the current screen magnification ratio, the enlarged portion, and the tapped position information, the screen control unit 301 derives the position of the accepted tap position on the screen before it was enlarged, and transmits the derivation result to the MFP 110.

[0077] According to the above embodiment, even when the screen is enlarged, the mobile terminal can accept operations such as tapping, and can correctly notify the operation information of the operation to the MFP 110. That is, when the user presses the position of the copy button 1031 on the menu display screen 1210 in FIG. 12, which shows the enlarged screen, the mobile terminal can transmit to the MFP that the copy button has been pressed, and transition the screen to the copy display screen 1320.

[0078] Example 3 In the first embodiment, a method was described in which the mobile terminal switches whether to transmit operation information indicating a drag operation to the MFP depending on the magnification state of the screen. However, it is also possible that the mobile terminal switches whether to transmit operation information to the MFP depending on the display state of the screen of the mobile terminal. For example, in a state in which the mobile terminal displays a soft keyboard on the screen received from the MFP, such as the screen of the menu display screen (hard key display) 975, even if the user presses the copy button 1031, operation information indicating the press is not transmitted to the MFP. That is, an example is shown in which whether to transmit operation information indicating a user operation on the screen received from the MFP to the MFP is controlled depending on whether the mobile terminal displays the hard key UI 977. For example, a description is given of an example in which, when the mobile terminal displays the hard key UI 977, the hard key UI 977 is hidden by tapping an area other than the hard key UI 977. If an area other than the hard key UI 977 is tapped to hide the hard key UI 977, and information indicating the tap is transmitted to the MFP, a process different from the user's intention is performed. The basic configuration of this embodiment is the same as that of the first embodiment, so only the difference is shown.

[0079] 13 shows screens displayed on mobile terminal 100 and MFP 110 when a tap operation is received while the mobile terminal is displaying menu display screen (hard keys not displayed) 970. When a tap operation 1311 is performed on copy button 1031 on menu display screen (before tap) 1310 not displaying hard key UI 977, copy display screen (after tap) 1320 is displayed. At this time, mobile terminal 100 transmits operation information indicating tap operation 1311 to MF 110P, and MFP 110 transitions the screen displayed on DISPLAY 3090 to copy screen 1330 (after tap) based on the received operation information and transmits the screen information to mobile terminal 100, and copy display screen (after tap) 1320 is displayed on mobile terminal 100.

[0080] 14 shows screens displayed on mobile terminal 100 and MFP 110 when a tap operation 1411 is received in a state in which the mobile terminal is displaying menu display screen (hard key display) 975. When a tap operation 1411 is performed on copy button 1031 on menu display screen (soft keyboard display, before tap) 1410 displaying hard key UI 977, mobile terminal 100 closes hard key UI 977 and displays menu display screen (after tap) 1420. At this time, mobile terminal 100 does not transmit operation information indicating tap operation 1411 to MFP 110. In other words, even if mobile terminal 100 receives a tap operation, MFP 110 continues to display menu display screen (after tap) 1430, which is similar to menu screen 1030.

[0081] The flowchart showing the processing of the mobile terminal 100 in this embodiment has the same basic configuration as that of FIG. 6 in the first embodiment, so only the differences are shown. In S611, it is determined whether the operation accepted in S610 is a tap operation on an area other than the hard key UI 977 area, and if it is a tap operation, the process proceeds to S613. In S613, it is determined whether the mobile terminal 100 displays the hard key UI 977, and if it is displayed, a process of hiding the hard key UI 977 is performed in S630. If the hard key UI 977 is not displayed, operation information is transmitted to the MFP 110 in S614. Also, if the operation accepted in S610 is a tap operation on the hard key UI 977, the operation information is transmitted from the mobile terminal 100 to the MFP 110.

[0082] According to the above embodiment, in the portable terminal that remotely operates the MFP, it is possible to switch whether or not to transmit accepted operation information to the MFP depending on whether or not a soft keyboard is displayed only on the portable terminal.

[0083] (Other Examples) In the above embodiment, an example has been shown in which the same screen as that displayed by the MFP via a VNC connection is displayed on the mobile terminal, but this is not limiting. For example, the above embodiment can also be applied to a case in which, instead of displaying the same screen as the MFP, a screen that is an extension of the MFP screen is displayed on the mobile terminal.

[0084] Furthermore, the present invention can also be realized by executing the following process. That is, software (program) that realizes the functions of the above-mentioned embodiments is supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program. In this case, the computer program and the storage medium storing the computer program constitute the present invention. [Explanation of symbols]

[0085] 100 Mobile Devices 110 MFP 201 CPU 202 ROM 203 RAM 204 HDD 205 RTC 206 Operation Panel 211 Radio Communication Department 300 MFP Applications

Claims

1. A program for causing a computer of an information processing device that communicates with an image processing device to execute a control method, The control method includes: a display step of displaying, on the display unit of the information processing device, a screen corresponding to a screen displayed on a display unit of the image processing device and based on display control of the image processing device; having when a predetermined user operation on the screen displayed in the display step is received, a screen based on information corresponding to the predetermined user operation is displayed on the image processing device; When the predetermined user operation is received in a state in which the display form of the screen displayed in the display step has been changed, a screen based on information corresponding to the predetermined user operation is not displayed on the image processing device, The state in which the display form of the screen displayed in the display step is changed is a state in which the magnification ratio of the screen displayed in the display step is changed. A program characterized by:

2. The predetermined user operation is an operation involving the movement of a pointer.

2. The program according to claim 1 .

3. The predetermined user operation is a drag operation, a flick operation, or a swipe operation.

3. The program according to claim 1 or 2.

4. The information corresponding to the predetermined user operation includes information regarding a position where the predetermined user operation is detected.

4. The program according to claim 1, wherein the program is a program for executing a program.

5. The program is a program for issuing a print instruction or a scan instruction to the image processing device.

5. The program according to claim 1, wherein the program is a program for executing a program.

6. The display step displays the same screen as that displayed on the display unit of the image processing device on the display unit of the information processing device.

6. The program according to claim 1, wherein the program is a program for executing a program.

7. The information processing device and the image processing device are connected via a VNC (Virtual Network Computing) connection, and the display control is performed using the VNC.

7. The program according to claim 1, wherein the program is a program for executing a program.

8. 8. The program according to claim 7, wherein the VNC connection communication is communication using an RFB (Remote Frame Buffer) protocol.

9. A program as described in any one of claims 1 to 8, characterized in that when the display form of the screen displayed in the display process is changed and the specified user operation is accepted on the screen displayed in the display process, the information processing device does not transmit operation information corresponding to the specified user operation to the image processing device, thereby preventing a screen based on information corresponding to the specified user operation from being displayed on the image processing device.

10. A program as described in any one of claims 1 to 8, characterized in that when the display form of the screen displayed in the display process is changed and the specified user operation is accepted on the screen displayed in the display process, display control based on information corresponding to the specified user operation is not executed by the image processing device, and therefore a screen based on operation information corresponding to the specified user operation is not displayed on the image processing device.

11. A program as described in any one of claims 1 to 10, characterized in that when the display form of the screen displayed in the display process is not changed and the specified user operation is accepted on the screen displayed in the display process, a screen based on information corresponding to the specified user operation is displayed on the image processing device.

12. The program described in claim 11, characterized in that when the display form of the screen displayed in the display process is not changed and the specified user operation is accepted on the screen displayed in the display process, the information processing device sends information corresponding to the specified user operation to the image processing device, thereby displaying a screen based on the information corresponding to the specified user operation on the image processing device.

13. An information processing device that communicates with an image processing device, a display unit for displaying, on the display unit of the information processing device, a screen corresponding to a screen displayed on the display unit of the image processing device, the screen being based on display control of the image processing device; having when a predetermined user operation on the screen displayed by the display means is received, a screen based on information corresponding to the predetermined user operation is displayed on the image processing device; When the predetermined user operation is received in a state in which the display form of the screen displayed by the display means has been changed, a screen based on information corresponding to the predetermined user operation is not displayed on the image processing device, The state in which the display format of the screen displayed on the display means is changed means that the magnification ratio of the screen displayed on the display means is changed.

23. An information processing apparatus comprising:

14. A control method executed by an information processing device that communicates with an image processing device, comprising: a display step of displaying, on the display unit of the information processing device, a screen corresponding to a screen displayed on a display unit of the image processing device and based on display control of the image processing device; having when a predetermined user operation on the screen displayed in the display step is received, a screen based on information corresponding to the predetermined user operation is displayed on the image processing device; when the predetermined user operation is received in a state in which the display form of the screen displayed in the display step has been changed, a screen based on information corresponding to the predetermined user operation is not displayed on the image processing device, The state in which the display form of the screen displayed in the display step is changed is a state in which the magnification ratio of the screen displayed in the display step is changed. A control method comprising:

Images