Information processing apparatus, method for controlling information processing, and storage medium

Abstract

An information processing apparatus, a method for controlling information processing, and a storage medium are disclosed. In a case where an image processing apparatus and a terminal display a screen in an associated manner, if the terminal receives a user operation in a state where only an enlarged screen is displayed on the terminal, and the terminal transmits information about the operation as it is to the image processing apparatus, a user-unintended processing can be performed. Therefore, when the terminal receives a user operation in a state where only an enlarged screen is displayed on the terminal, the terminal does not transmit operation information as it is to the image processing apparatus.

Description

Technical Field

[0001] This disclosure relates to remote control methods. Background Technology

[0002] A method for remotely operating the operation screen of an image processing device from a personal computer (PC) or portable terminal using Virtual Network Computing (VNC) compliant with the Remote Frame Buffer (RFB) protocol has been discussed. More specifically, the image processing device with VNC server functionality sends screen information of the operation screen to a terminal with VNC client functionality.

[0003] The terminal displays the same screen as the operation screen displayed on the operation panel of the image processing device based on the received screen information. When the user operates the screen displayed on the terminal, information about the operation is sent from the terminal to the image processing device. The image processing device performs screen transition processing based on the sent information and sends information about the changed screen back to the terminal. By performing operations on the screen displayed on the terminal in this way, the user can perform login operations and instruct the image processing device to perform processing via the operation panel provided by the image processing device, without having to operate the operation panel of the image processing device. Japanese Patent Application Publication No. 2014-6668 discusses a technique used when the terminal and the image processing device display screens in a correlated manner. When the terminal receives an operation that can be received by the terminal but not by the image processing device, the terminal invalidates the operation and does not send information about the operation to the image processing device. When the terminal receives an operation that can be received by both the terminal and the image processing device, the terminal sends information about the operation to the image processing device to make the operation effective. However, even when the terminal receives an operation that can be received by both the terminal and the image processing device, and the terminal sends information about the operation to the image processing device as is, problems may occur depending on the display format of the screen on the terminal. Consider an example scenario where both the terminal and the image processing device can receive drag operations and are controlled to perform a screen transition upon receiving a drag operation. Furthermore, assume that the terminal and the image processing device are associated with a single screen, and currently only the terminal displays a magnified image. In this state, receiving a drag operation from the user on the terminal indicates that the user wants to move the magnified display area because the image is magnified on the terminal. However, if the terminal, upon receiving the drag operation, sends the information about the drag operation as is to the image processing device, a screen transition will be performed, resulting in processing different from the user's intention. Summary of the Invention

[0004] Embodiments of this disclosure aim to provide a method for performing user-desired processing based on operations received from a user, taking into account the display format of the screen on the terminal and the screen association between the terminal and the image processing device.

[0005] According to embodiments of this disclosure, a method of an information processing apparatus for communicating with an image processing apparatus includes: displaying a screen corresponding to a screen displayed on a display unit of the image processing apparatus and generated based on display control of the image processing apparatus; and when a predetermined user operation on the displayed screen is received, sending operation information corresponding to the predetermined user operation to the image processing apparatus, wherein, if the predetermined user operation on the screen is received while the display mode of the displayed screen is changed, the operation information corresponding to the predetermined user operation is not sent to the image processing apparatus.

[0006] Other features of this disclosure will become clear from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0007] Figure 1 This is a diagram illustrating a system configuration according to a first exemplary embodiment.

[0008] Figure 2 This is a block diagram illustrating the hardware configuration of a portable terminal according to a first exemplary embodiment.

[0009] Figure 3 This is a block diagram illustrating the software configuration of a portable terminal according to a first exemplary embodiment.

[0010] Figure 4 This is a block diagram illustrating the hardware configuration of a multifunctional peripheral device (MFP) according to a first exemplary embodiment.

[0011] Figure 5 This is a block diagram illustrating the software configuration of an MFP according to a first exemplary embodiment.

[0012] Figure 6 This is a flowchart illustrating the processing of a portable terminal according to a first exemplary embodiment.

[0013] Figure 7 This is a flowchart illustrating the processing of an MFP according to a first exemplary embodiment.

[0014] Figure 8 This is a flowchart illustrating another process of a portable terminal according to a second exemplary embodiment.

[0015] Figure 9A and Figure 9BThis is a diagram illustrating the user interface (UI) displayed on the operation panel of a portable terminal according to a first exemplary embodiment.

[0016] Figure 10 The figure illustrates the UI displayed on the operation panel of a portable terminal and an MFP according to a first exemplary embodiment.

[0017] Figure 11 The figure illustrates the UI displayed on the operation panel of a portable terminal and an MFP according to a second exemplary embodiment.

[0018] Figure 12 This is a diagram illustrating the UI displayed on the operation panel of a portable terminal and an MFP according to a first exemplary embodiment.

[0019] Figure 13 This is a diagram illustrating the UI displayed on the operation panel of a portable terminal and an MFP according to a third exemplary embodiment.

[0020] Figure 14 This is a diagram illustrating the UI displayed on the operation panel of a portable terminal and an MFP according to a third exemplary embodiment. Detailed Implementation

[0021] Exemplary embodiments of the present disclosure will now be described with reference to the accompanying drawings. These exemplary embodiments are not intended to limit the scope of the disclosure. Not all combinations of features described in the exemplary embodiments are essential to the solutions provided in this disclosure.

[0022] <System configuration according to this exemplary embodiment>

[0023] First, the system configuration according to the first exemplary embodiment will be described. While the portable terminal 100 and the multifunction peripheral device (MFP) 110 will be described as examples of information processing apparatuses, the apparatuses included in the information processing system are not limited thereto. The portable terminal 100 and the MFP 110 are connected to a local area network (LAN) and are able to exchange information with each other. The portable terminal 100 is connected to the LAN via access point 120.

[0024] Portable terminal 100 has Virtual Network Computing (VNC) client functionality for remotely controlling MFP 110. More specifically, portable terminal 100 has VNC client software compliant with the Remote Frame Buffer (RFB) protocol. Portable terminal 100 establishes a VNC link to MFP 110 and displays the screen shown on the operation panel of MFP 110 on portable terminal 100. When MFP 110 has VNC server functionality and establishes a VNC link from portable terminal 100, MFP 110 sends screen information to portable terminal 100, which acts as a VNC client. Portable terminal 100 draws the screen based on the sent screen information. Screen association is performed to change the screen displayed on the display units of portable terminal 100 and MFP 110 based on operations received on the drawn screen and operations received on MFP 110.

[0025] Next, we will refer to Figure 2 The hardware configuration of the portable terminal 100 is described. It is assumed that the portable terminal 100 according to this exemplary embodiment is a smartphone or a tablet personal computer (PC), and may also be other information processing devices capable of communicating with the MFP 110.

[0026] The Central Processing Unit (CPU) 201 reads the control program stored in the Read-Only Memory (ROM) 202 and performs various processes to control the operation of the portable terminal 100. The ROM 202 stores the control program. The Random Access Memory (RAM) 203 serves as the main memory of the CPU 201 and a temporary storage area such as a working area. The storage device 204 is, for example, an embedded multimedia card (eMMC), a solid-state drive (SSD), or a hard disk drive (HDD) that stores various data such as image data and electronic documents. The operating system (OS) 350 (described below) and the MFP application 300 are also stored in the storage device 204. A Real-Time Clock (RTC) 205 measures time.

[0027] Although in the portable terminal 100, one CPU 201 executes each process shown in the flowchart (described below), other modes are also applicable. For example, multiple CPUs can cooperate to execute each process shown in the flowchart (described below).

[0028] The operation panel 206 has a touch panel function that detects user touch operations and displays various screens provided by the OS 350 and MFP application 300. Users can input desired operation commands to the portable terminal 100 by performing touch operations on the operation panel 206.

[0029] Speaker 207 and microphone 208 are used, for example, to make telephone calls to other portable terminals and landlines. Camera 209 captures images in response to imaging commands from the user. Images captured by camera 209 are stored in a predetermined area of ​​storage device 204. Wireless communication unit 211 performs wireless communication such as wireless local area network (LAN).

[0030] Next, we will refer to Figure 3 Describe the software configuration of the portable terminal 100. Figure 3 This is a block diagram illustrating the software functions implemented by the CPU 201 reading the control program stored in the ROM 202 and the storage device 204 and executing the control program.

[0031] OS 350 is software that controls the overall operation of portable terminal 100. Various applications, including MFP application 300 (described below), can be installed on portable terminal 100. OS 350 exchanges information with these applications and changes the screen displayed on operation panel 206 in response to instructions received from the applications. OS 350 has a set of device drivers for controlling various hardware components and provides application programming interfaces (APIs) for applications operating on the OS to use these hardware components. According to this exemplary embodiment, for example, the set of device drivers corresponds to wireless LAN control unit 351. Wireless LAN control unit 351 is a device driver for controlling wireless communication unit 211.

[0032] MFP Application 300 is an application installed on Portable Terminal 100. MFP Application 300 can issue print and scan commands to MFP 110. As described above, Portable Terminal 100 operates as a VNC client and has the function of displaying the operation screen of MFP 110 and remotely controlling MFP 110. Various other applications besides MFP Application 300 can be installed on Portable Terminal 100. Descriptions of these applications will be omitted.

[0033] The configuration of the MFP application 300 will be described in more detail below. The screen control unit 301 controls the screen displayed on the operation panel 206 via the OS 350. The screen control unit 301 displays the MFP application screen on the operation panel 206. The screen control unit 301 receives operation commands input by the user via the operation panel 206. The communication unit 302 controls the wireless communication unit 211 via the OS 350 to send / receive information to / from the MFP 110 and other external devices.

[0034] The device search control unit 311 generates search data for searching external devices connected to the portable terminal 100 and transmits the data to devices on the LAN 10 via the wireless communication unit 211. The device search control unit 311 then receives responses from the external devices responding to the search. The search data can be broadcast to all external devices on the LAN 10 or sent to an external device with a specified address.

[0035] The print job generation unit 305 generates a print job. The print job generated by the print job generation unit 305 is sent to the MFP 110 via the wireless communication unit 211. The MFP 110 executes printing in response to the received print job.

[0036] The scan job control unit 307 sends a scan command to the MFP 110 via the wireless communication unit 211. The MFP 110 responds to the command by performing a scan and sending the data generated by the scan (scan data) to the portable terminal 100. The portable terminal 100 displays the received scan data. When the scan data is stored, it is stored in the application storage unit 306. The stored scan data is managed by the document management unit 308.

[0037] The remote operation unit 313 displays a screen based on screen information obtained from the MFP 110, which is linked to the VNC, and receives operations performed on the displayed screen. In other words, the remote operation unit 313 operates as a VNC client. More specifically, the remote operation unit 313 displays a screen based on screen information received from the VNC server and sends operation information, including the location and type of the operations received on the displayed screen, to the MFP 110. The MFP 110 changes the screen displayed on the MFP 110 and the screen currently displayed on the portable terminal 100 based on the received operation information. The application storage unit 306 temporarily stores various information generated by the MFP application 300. The timer unit 304 measures time.

[0038] Figure 4 This is a block diagram illustrating an example hardware configuration of the MFP 110. The MFP 110 includes a CPU 3020 that executes programs stored in ROM 3030 or a large storage device 3110 such as a hard disk. The CPU 3020 is generally controlled by devices connected to the system bus 3010.

[0039] RAM 3040 serves as the main memory and working area for CPU 3020. PANELC 3060 is an external input controller that controls instructions input from the hard keyboard 3120, or panel 3070 is a touch panel provided on a multi-function peripheral device. DISPC 3080 is a display controller that controls the display of monitor 3090, which is, for example, a display module including an LCD. DKC 3100 is a disk controller for controlling a large storage device (HD) 3110.

[0040] Reference Figure 10 The screen 1000 describes the operating unit of the MFP 110. A screen is displayed on a monitor 3090, and a physical keyboard 3120 is arranged next to the monitor 3090. Both the physical keyboard 3120 and the panel (touch panel) 3070 on the monitor 3090 can receive user input. When a user presses the logout key 1001 on the physical keyboard 3120 while logged into the MFP 110, the key instructs the MFP 110 to perform a logout process.

[0041] Will describe again Figure 4 The hardware configuration of the MFP 110 is as follows: A network interface card (NIC) 3050 performs bidirectional data exchange with an external device (e.g., portable terminal 100) or file server via network 10. A WLAN 3400 is a wireless communication module used to operate the MFP 110 as an access point, enabling the establishment of a direct wireless communication link with the portable terminal 100 by connecting to or operating in access point mode. A printer 3200 is a printing unit that prints on paper using an electrophotographic method. The printing method does not necessarily have to be an electrophotographic method. A scanner 3300 is an image reading unit that reads images printed on paper. In many cases, an automatic document feeder (ADF) (not shown) is attached to the scanner 3300 as an option to automatically read multiple documents. In some cases, an HD 3110 can also be used for temporary image storage.

[0042] Figure 5 This is a block diagram illustrating the software configuration of the MFP 110. For example, the software is stored in ROM 3030 and executed by CPU 3020.

[0043] The user interface (UI) processing unit 510 analyzes input from the external input controller (PANELC) 3060 or operation information received from the operation information receiving unit 522 (described below). When a screen change is required, the UI processing unit 510 generates screen data and displays the data on the display 3090.

[0044] Monitor 3090 displays as follows Figure 10The various screens shown are illustrated. Examples of screens displayed on monitor 3090 include ID card login screen 1020, keyboard login screen 1010, menu display screen 1030 displayed during login operation, error screen (not shown) displayed when an error occurs, and counter screen (not shown) displaying counter information. Monitor 3090 is not displayed when MFP110 is in sleep mode.

[0045] ID card login screen 1020 is used to log in to MFP 110 using an ID card associated with user information. When the user holds the ID card on the ID card reader of MFP 110 while this screen is displayed, user information is read and login processing is performed. When the user presses the keyboard login screen change (to keyboard login) button 1021, keyboard login screen 1010 appears.

[0046] The keyboard login screen 1010 is used to log in to the MFP 110 by entering user information using the keyboard 3120. When the user fills in the username input field 1011 and password input field 1012 using the hard keyboard 3120 or the soft keyboard (not shown) and then presses the login button 1013, the MFP 110 performs the login process. When the user presses the ID card login change (to ID card login) button 1014, the ID card login screen 1020 appears.

[0047] After a user logs into the MFP 110 via the keyboard login screen 1010 or the ID card login screen 1020, the menu display screen 1030 appears. The user can use various functions of the MFP 110 by pressing the corresponding copy button 1031, scan button 1032, and print button 1033. The menu screen includes multiple pages. When the user presses the logout button 1034, the MFP 110 performs a logout process. The next button 1036 changes the menu screen to the next page. The previous button 1035 changes the menu screen to the previous page. Page transitions can also be performed by dragging or flicking.

[0048] The remote operation processing unit 520 includes a connection processing unit 523, an operation information receiving unit 522, and a screen sending unit 521. The remote operation processing unit 520 is a VNC server module that performs communication conforming to the RFB protocol; that is, a software module that communicates with the portable terminal 100, which operates as a VNC client, via a network. The connection processing unit 523 performs connection processing when it receives a connection request from the MFP 110 and the portable terminal 100, which acts as a VNC client, and performs disconnection processing when it receives a disconnection request from them.

[0049] The operation information receiving unit 522 receives screen operation information from the portable terminal 100 and MFP 110, and notifies the UI processing unit 510 of the information. The screen sending unit 521 is a display control unit that reads the screen data generated by the UI processing unit 510 from the RAM 3040 and sends the screen data to the portable terminal 100, which acts as a VNC client, to perform screen display control.

[0050] The login processing unit 530 is a module that performs login processing for the MFP 110. More specifically, the login processing unit 530 performs user authentication based on user information received from the UI processing unit 510, and performs a login operation after successful user authentication. Upon receiving a logout command from the UI processing unit 510, the login processing unit 530 performs logout processing. The network control unit 500 is a module that controls the NIC 3050. Other modules (not shown) include a printer module that controls the printer 3200 and a scanner module that controls the scanner 3300.

[0051] Figure 9A and Figure 9B The diagram illustrates an example of a screen displayed on the operation panel 206 of the portable terminal 100. When the portable terminal 100 is activated, the main screen 910 appears. The MFP application icon 911 is used to activate the MFP application 300. When the user interacts with the MFP application icon 911, the corresponding MFP application 300 is activated.

[0052] When MFP application 300 is activated, the application main screen 920 (without a device selected) appears. The selected device icon 921 displays information about the MFP that MFP application 300 is currently communicating with, as well as the target MFP for job delivery (hereinafter referred to as the target MFP). When no target MFP is set, "No printer" is displayed. The device search button 923 is used to search for the target MFP. The remote operation button 922 is used to establish a VNC link to remotely control the target MFP. The application main screen 920 can display a print button 924 for sending print jobs to the target MFP and a button for issuing scan commands.

[0053] When the user presses the device search button 923, the system searches for MFPs present in the network, and the device search screen 930 displays the search results. The device search screen 930 shows a list of MFPs that have responded to the search. For example, when the user selects search result 931 as one of the search results, the target MFP is selected.

[0054] When a target MFP is selected, the application's main screen 940 (Device Selected) appears. The selected device icon 921 displays information about the MFP selected in the device search screen 930. When the user taps the remote operation button 922 while the target MFP is currently selected, the remote control function of the target MFP is activated. The following description assumes that the target MFP is MFP110.

[0055] When the remote control function is activated, the device connection screen 950 appears. When the user presses the hard keyboard display button 951, the UI for operating the MFP 110 hard keyboard 3120 appears. Details are described below.

[0056] When a VNC link is established and remote control is enabled, the screen displayed on monitor 3090 of the MFP 110 appears on the operation panel 206. In this case, the same keyboard login screen 960 as keyboard login screen 1010 appears. Except for the hard keyboard display button 951 appearing on the operation panel 206, keyboard login screen 960 displays the same content as keyboard login screen 1010 of the MFP 110, and its description will be omitted. When the MFP 110 is currently displaying ID card login screen 1020, ID card login screen 965 appears.

[0057] When a login command is issued via the keyboard login screen 960, a menu display screen 970 (without a physical keyboard) identical to the menu display screen 1030 appears. The menu display screen 1030 is also displayed on the monitor 3090.

[0058] When the user clicks the physical keyboard display button 951, a menu display screen 975 (the physical keyboard is displayed) appears. More specifically, a software keyboard appears, which assigns a processing line to each physical keyboard 3120 of the MFP 110. The physical keyboard UI 977 is the UI used to operate the physical keyboards 3120 of the MFP 110. When the user clicks a button displayed on the physical keyboard UI 977, the key information of the pressed key is sent to the MFP 110. The MFP 110 performs processing based on the pressed key. When the user presses the logout key 979 corresponding to the logout key 1001 of the MFP 110, the user logs out of the MFP 110. When the user clicks the end button 978 at the top of the physical keyboard UI 977, the remote control function ends, i.e., the VNC connection ends. When the VNC connection ends, the application main screen 940 (the device is selected) appears on the operation panel 206. When the VNC connection ends, the screen before the VNC connection ended is continuously displayed on the display 3090.

[0059] Figure 11The illustration shows an example screen of portable terminal 100 and MFP 110 when a user performs a drag operation while menu display screen 970 is displayed on portable terminal 100. Based on screen information sent from MFP 110, menu display screen 1110 (before dragging) is displayed on portable terminal 100. Menu display screen 1110 (before dragging) is a default, unmagnified screen, allowing the full screen to be displayed based on screen information received from MFP 110. In this state (default screen state), the screen sent from MFP 110 is displayed as is, so that the screen size fits the horizontal and vertical dimensions of the operation panel 206 of portable terminal 100.

[0060] When a user performs a drag operation 1111 on menu display screen 1110 (before dragging), portable terminal 100 sends operation information indicating the drag operation to MFP 110 and displays menu display screen 1120 (after dragging) as the next menu page. At this time, MFP 110 also displays menu screen 1130 (after dragging) as the next page based on the operation information received from portable terminal 100. Menu display screen 1120 (after dragging) displays fax button 1121, mailbox (BOX) button 1122, and scan and send button 1123. When the user presses each button, the corresponding processing screen appears. Similarly, when a drag operation is received on MFP 110 while menu display screen 1030 is displayed on MFP 110, portable terminal 100 displays menu display screen 1120 (after dragging), and MFP 110 displays menu screen 1130 (after dragging). In this scenario, when a drag operation from the user is received while the MFP 110 is displaying the menu screen 1030, the drag operation is assigned to a process for page transition. Therefore, similarly, when the MFP 110 is remotely controlled from the portable terminal 100, a page transition can be performed upon receiving a drag operation on the portable terminal 100.

[0061] Figure 12 The figure shows the screens of portable terminal 100 and MFP 110 when a drag operation is performed while zooming in on the menu display screen 970 on portable terminal 100.

[0062] When a user's zoom-in operation, such as pinch-out or double-click, is received on the portable terminal 100 displaying menu screen 1110 (before dragging), menu screen 1210 (enlarged display, before dragging) appears on the portable terminal 100. The zoom-in operation information is not sent from the portable terminal 100 to the MFP 110. This is because, on the MFP 110, while menu screen 1030 is displayed, dragging is not assigned to the processing for zooming in. Therefore, after receiving a zoom-in operation from the user on the portable terminal 100 displaying menu screen 1110 (before dragging), the portable terminal 100 displays the enlarged menu screen 1210, and the MFP 110 displays the unenlarged menu screen 1030.

[0063] After receiving a drag operation 1211 from the user on the menu display screen 1210 (enlarged display, before dragging), the portable terminal 100 displays a menu display screen 1220 (enlarged display, after dragging) in the enlarged display area after the movement operation. At this time, the portable terminal 100 does not send operation information about the drag operation 1211 to the MFP 110. Therefore, the MFP 110 maintains the display of the menu screen 1230 (after dragging), similar to the menu display screen 1030. If the user performs a drag operation while enlarging the menu screen on the portable terminal 100, the user wants to move the enlarged display area in the drag direction. Therefore, the portable terminal 100 does not send operation information to the MFP 110, but instead performs processing to change the display area on the portable terminal 100 itself. When the portable terminal 100 receives a drag operation from the user while the menu screen is zoomed in, and then sends operation information about the drag operation to the MFP 110, the MFP 110 performs a page transition on the menu screen because the drag operation is assigned to processing for page transitions. Therefore, the MFP 110 performs processing different from the user's intention. To address this, when the portable terminal 100 receives a page transition operation such as a drag operation while the screen is zoomed in, the portable terminal 100 itself changes the screen display without sending the operation information to the MFP 110, thus enabling processing according to the user's intention. Even when the portable terminal 100 is displaying a zoomed-in screen, pressing the positions on the portable terminal 100 corresponding to the previous button 1035 and the next button 1036 will also enable a page transition.

[0064] Figure 6This diagram illustrates the process when a portable terminal 100 establishes a VNC link with an MFP 110 and the user performs remote control. The process is implemented when the CPU 201 loads the control program stored in a memory such as ROM 202 into RAM 203 and then executes that control program. Figure 6 Each step is shown in the flowchart.

[0065] When CPU 201 detects that the MFP application icon 911 has been clicked on the main screen 910, then in step S601, CPU 201 activates the corresponding MFP application 300. In step S602, when screen control unit 301 detects that the device search button 923 has been clicked on the application main screen 920 (no device selected), CPU 201 performs an MFP search. In step S603, CPU 201 determines whether an MFP has been selected on the device search screen 930. When an MFP is selected (yes in step S603), CPU 201 switches the screen to the application main screen (device selected) 940, and the process proceeds to step S604. On the other hand, when an MFP is not selected (no in step S603), the process in step S603 is repeated.

[0066] In step S604, CPU 201 determines whether the remote operation button 922 has been selected in the application main screen 940 (device selected). When the remote operation button 922 is selected (yes in step S604), the process proceeds to step S605. On the other hand, when the remote operation button 922 is not selected (no in step S604), the process in step S604 is repeated.

[0067] In step S605, CPU 201 sends a connection request conforming to the RFB protocol to the MFP selected in step S604. In step S606, CPU 201 determines whether a connection request has been received. If a connection request is received (yes in step S606), the process proceeds to step S607. On the other hand, if no connection request is received (no in step S606), the process proceeds to step S620. CPU 201 determines whether a connection request has been received based on the response data of the connection request. In step S620, screen control unit 301 displays a connection rejection error screen 980, and CPU 201 terminates the process.

[0068] In step S607, CPU 201 performs initialization processing conforming to the RFB protocol. During the initialization process, remote operation unit 313 sends VNC client information and receives VNC server information.

[0069] In step S608, the remote operation unit 313 receives screen information sent from MFP 110—the VNC server. In step S609, the screen control unit 301 draws a screen on the operation panel 206 based on the received screen information and the screen information generated by the screen control unit 301. For example, when the keyboard login screen 1010 is displayed on the monitor 3090, the keyboard login screen 960 is displayed on the operation panel 206. Each time screen information is sent from MFP 110, the CPU 201 continuously displays the screen in step S609.

[0070] In step S610, the remote operation unit 313 determines whether an input operation has been received from the operation panel 206. If an input operation is received (yes in step S610), the process proceeds to step S611. On the other hand, if no input operation is received (no in step S610), the process in step S610 is repeated.

[0071] In step S611, the screen control unit 301 determines whether the operation received in step S610 is a screen zoom-in operation. Examples of screen zoom-in operations include zooming in and double-clicking. When it is determined that the received operation is a screen zoom-in operation ("Yes" in step S611), the CPU 201 does not send the operation information to the MFP 110 and the process returns to step S609. In step S609, the screen control unit 301 displays the zoomed-in screen. On the other hand, when it is determined that the received operation is not a screen zoom-in operation ("No" in step S611), the process proceeds to step S612.

[0072] In step S612, the screen control unit 301 determines whether the operation received in step S610 is a drag operation. If it is determined that the received operation is a drag operation (yes in step S612), the process proceeds to step S613. On the other hand, if it is determined that the received operation is not a drag operation (no in step S612), the process proceeds to step S614. In step S613, the CPU 201 determines whether the screen displayed on the operation panel 206 is in magnified mode. If it is determined that the screen is in magnified mode (yes in step S613), the process proceeds to step S630. On the other hand, if it is determined that the screen is not in magnified mode (no in step S613), the process proceeds to step S614. For example, when the menu display screen 1210 is displayed on the operation panel 206, it is determined that the screen is magnified. More specifically, in this case, the CPU 201 determines whether the screen received from the MFP 110 is magnified from the default state where the screen is displayed as is. In step S630, the screen control unit 301 calculates the display area based on the current magnification (the magnification corresponding to the default screen display mode) and the amount of movement of the received drag operation. In step S609, the screen control unit 301 displays the screen with the magnified changed area. At this time, the operation information indicating the drag operation is not sent to the MFP 110.

[0073] In step S614, the remote operation unit 313 sends the operation information corresponding to the operation received in step S610 to the MFP 110. When the operation received in step S610 is a remote control termination operation, i.e., a VNC link termination operation, the remote operation unit 313 sends a VNC link disconnection request to the MFP 110.

[0074] According to the flowchart above, even when the MFP 110 is remotely controlled from the portable terminal 100, and a drag operation is received on the portable terminal 100 while the screen is being zoomed in, the portable terminal 100 does not send operation information about the drag operation to the MFP 110. This prevents unexpected page transitions from being performed by the user.

[0075] When a drag operation is received and it is determined that the screen is in a zoomed-in state ("Yes" in step S613), the process proceeds to step S630. In step S630, the screen display is changed only on the portable terminal 100. On the other hand, when it is determined that the screen is not in a zoomed-in state ("No" in step S613), the process proceeds to step S614. In step S614, the portable terminal 100 sends operation information indicating a drag operation to the MFP 110. In other words, the portable terminal 100 selects whether to send operation information to the MFP 110 based on whether the screen is zoomed in. However, this control is not limited to determining whether the screen is zoomed in. For example, the portable terminal 100 can select whether to send operation information to the MFP 110 based on whether the screen currently displayed is reduced in size based on the screen information sent from the MFP 110. In other words, the portable terminal 100 selects whether to send operation information to the MFP 110 based on the state of the screen displayed on the portable terminal 100.

[0076] In the above example, although the portable terminal 100 does not send operation information to the MFP 110 when it receives a screen zoom-in operation, the portable terminal 100 can still send operation information. In this case, since the MFP 110 does not process the received operation information in a manner corresponding to the received operation information, the MFP 110 invalidates the operation information and does not process it. In the above example, after receiving a screen zoom-in operation on the portable terminal 100 in step S611, the portable terminal 100 only displays the zoomed-in screen without sending operation information to the MFP 110. However, the processing is not limited to screen zoom-in operations. Similarly, when receiving a screen zoom-out operation such as pinch-in, the portable terminal 100 can display the zoomed-out screen without sending operation information to the MFP 110.

[0077] Although the drag operation has been described above as an example of a page transition operation, examples of page transition operations can include flicking, swiping, and other operations used to move the indicated object.

[0078] Figure 7 The diagram illustrates the processing of the MFP 110 when it performs VNC-based screen association with the portable terminal 100. The process is achieved when the CPU 3020 loads the control program stored in a memory such as ROM 3030 into RAM 3040 and then executes that control program. Figure 7 Each step is shown in the flowchart.

[0079] In step S701, the connection processing unit 523 determines whether a VNC link request has been received from the portable terminal 100, which is acting as a VNC client. If a VNC link request is received (yes in step S701), the process proceeds to step S702. Conversely, if no VNC link request is received (no in step S701), the process in step S701 is repeated. In step S702, the connection processing unit 523 performs negotiation processing for connecting with the VNC client. In step S703, the connection processing unit 523 performs initialization processing, receives client information, and sends server information.

[0080] In step S704, the screen sending unit 521 sends the screen currently displayed on the display 3090 to the portable terminal 100, which is a VNC client.

[0081] In step S705, the operation information receiving unit 522 determines whether an operation has been received from the user. If an operation is received ("Yes" in step S705), the process proceeds to step S706. On the other hand, if no operation is received ("No" in step S705), the process repeats the steps in S705. For example, when operation information is received from the portable terminal 100, or when a user operation is detected on the hard keyboard 3120 of the panel 3070 or MFP 110, the CPU 3020 determines that an operation has been received from the user.

[0082] In step S706, CPU 3020 determines whether the operation received in step S705 is a VNC link disconnection operation. If it is determined that the received operation is a VNC link disconnection operation ("Yes" in step S706), the process proceeds to step S707. In step S707, CPU 3020 disconnects the VNC link. On the other hand, if it is determined that the received operation is not a VNC link disconnection operation ("No" in step S706), the process proceeds to step S708.

[0083] In step S708, CPU 3020 executes processing corresponding to the operation received in step S705. Examples of corresponding processing include processing for changing the menu screen displayed on monitor 3090 to the next page, processing for logging into MFP 110, and processing for logging out of MFP 110. In step S709, operation information receiving unit 522 determines whether the screen displayed on monitor 3090 has changed based on the processing executed in step S708. If the screen has changed (yes in step S709), the processing returns to step S704. On the other hand, if the screen remains unchanged (no in step S709), the processing returns to step S705.

[0084] Through the above processing, the screen displayed on the MFP 110 can also be displayed on the portable terminal 100. The CPU 3020 can change the screen displayed on the MFP 110 and portable terminal 100 in response to an operation received on the MFP 110 or portable terminal 100. In the above flowchart, the screen displayed on the portable terminal 100 is changed in response to an operation received by the MFP 110 or portable terminal 100. If neither the MFP 110 nor the portable terminal 100 receives an operation, and the screen displayed on the MFP 110 changes, then the MFP 110 similarly sends the changed screen information to the portable terminal 100, and the portable terminal 100 displays the changed screen.

[0085] Suppose that portable terminal 100 displays a screen based on screen information sent from MFP 110 to remotely control MFP 110 from portable terminal 100. In this case, according to the exemplary embodiment described above, when portable terminal 100 receives a predetermined page transition operation such as a drag operation while portable terminal 100 is not displaying the screen received from MFP 110 as is (e.g., while the screen is being zoomed in), it can prevent portable terminal 100 from sending operation information to MFP 110. If portable terminal 100 sends received operation information to MFP 110 while portable terminal 100 is not displaying the screen received from MFP 110 as is, then MFP 110 may perform processing that is not intended by the user. For example, if portable terminal 100 receives a drag operation while portable terminal 100 is zooming in on the screen received from MFP 110, and then sends information about the drag operation to MFP 110, then a page transition occurs. In this case, the user intends to move the display area by dragging. Therefore, if the portable terminal 100 sends operation information to the MFP 110, the MFP 110 will perform processing that differs from the user's intent. This exemplary embodiment enables the prevention of performing processing that differs from the user's intent.

[0086] According to the exemplary embodiment described above, the portable terminal 100 determines whether to send operation information to the MFP 110 based on its own screen state. This disclosure is not limited thereto. The MFP 110 can understand the display state of the portable terminal 100. In this case, when receiving operation information from the portable terminal 100, the MFP 110 can consider the display state of the portable terminal 100 to determine whether to perform processing and select processing accordingly.

[0087] The first exemplary embodiment has been described above with reference to an example in which, even when the screen on the portable terminal 100 is zoomed in and a page transition operation such as a drag operation is received, the portable terminal 100 does not transmit operation information about the operation to the MFP 110. However, even when the screen on the portable terminal 100 is zoomed in, the portable terminal 100 may send operation information about operations such as clicks and presses to the MFP 110. The second exemplary embodiment will be described with reference to an example in which, even when the screen on the portable terminal 100 is zoomed in, the portable terminal 100 receives operations such as clicks and presses and sends operation information to the MFP 110. The basic configuration of the second exemplary embodiment is similar to that of the first exemplary embodiment, and only the differences will be described.

[0088] Figure 8 This is a flowchart illustrating the processing of a portable terminal 100 according to this exemplary embodiment. Figure 8 The basic configuration of the flowchart in the example is the same as that according to the first exemplary embodiment. Figure 6 The basic configuration of the flowcharts in the two languages ​​is similar, and only the differences are described below.

[0089] In step S801, the screen control unit 301 determines whether the operation received in step S610 is a click operation. If it is determined that the received operation is a click operation ("Yes" in step S801), the process proceeds to step S802. In step S802, the screen control unit 301 determines whether the screen is currently zoomed in. On the other hand, if it is determined that the received operation is not a click operation ("No" in step S801), the process proceeds to step S614.

[0090] When it is determined that the image is currently magnified ("Yes" in step S802), the process proceeds to step S803. In step S803, the image control unit 301 calculates the click position based on the current magnification, magnification position, and click position information. In step S614, the image control unit 301 sends operation information about the calculated position coordinates to the MFP 110. The image control unit 301 exports the position on the unmagnified image corresponding to the received click position based on the current magnification, magnification position, and click position information, and sends the exported result to the MFP 110.

[0091] The exemplary embodiment described above enables the portable terminal 100 to receive operations such as clicking even when the screen is magnified, and to correctly notify the MFP 110 of the operation information. In other words, when the user is viewing the magnified display screen... Figure 12When the copy button 1031 is pressed on the menu display screen 1210, the portable terminal 100 notifies the MFP 110 that the copy button 1031 has been pressed and switches the screen to the copy display screen (after clicking) 1320.

[0092] The first exemplary embodiment has been described above regarding a method performed by the portable terminal 100 to select whether to send operation information indicating a drag operation to the MFP 110 based on the zoom state of the screen. However, the portable terminal 100 may also select whether to send operation information to the MFP 110 based on the display mode of the portable terminal 100's screen. For example, as in menu display screen 975 (hard keyboard is displayed), when the user presses the copy button 1031 on the screen received from the MFP 110 while the soft keyboard is displayed on the portable terminal 100, the portable terminal 100 is prevented from sending operation information about the press to the MFP 110. More specifically, in the following example, the portable terminal 100 controls whether to send operation information about the user operation performed on the screen received from the MFP 110 to the MFP 110 based on whether the portable terminal 100 is currently displaying the hard keyboard UI 977. The description will focus on an example configuration where the hard keyboard UI 977 is hidden when the user clicks on an area other than the hard keyboard UI 977 while the portable terminal 100 displays it. If the user clicks on an area other than the hard keyboard UI 977 to hide the hard keyboard UI 977, and information about the click is sent to the MFP 110, then processing different from the user's intention will be performed. The basic configuration of the third exemplary embodiment is similar to that of the first exemplary embodiment, and only the differences will be described below.

[0093] Figure 13 The diagram illustrates the screens displayed on both the portable terminal 100 and the MFP 110 when a click operation is received while the menu display screen 970 is displayed on the portable terminal 100 (without a physical keyboard). When the user clicks the copy button 1031 on the menu display screen 1310 (before the click), which is currently not displaying the physical keyboard UI 977, the portable terminal 100 displays the copy display screen 1320 (after the click). At this time, the portable terminal 100 sends operation information about the click operation 1311 to the MFP 110. Based on the received operation information, the MFP 110 switches the screen of the display 3090 to the copy display screen 1330 (after the click) and sends the screen information to the portable terminal 100. Then, the portable terminal 100 displays the copy display screen 1320 (after the click).

[0094] Figure 14The diagram illustrates the screens displayed on both the portable terminal 100 and the MFP 110 when the portable terminal 100 displays menu display screen 975 (with the hard keyboard displayed) and simultaneously receives a click operation 1411. When the user clicks the copy button 1031 on menu display screen 1410 (before the soft keyboard is displayed), which currently displays the hard keyboard UI 977, the portable terminal 100 closes the hard keyboard UI 977 and displays menu display screen 1420 (after the click). At this time, the portable terminal 100 does not send operation information regarding the click operation 1411 to the MFP 110. In other words, even if the portable terminal 100 receives a click operation, the MFP 110 continues to display menu display screen 1430 (after the click), as shown in menu display screen 1030.

[0095] The basic configuration of the flowchart indicating the processing of the portable terminal 100 according to this exemplary embodiment is different from that according to the first exemplary embodiment. Figure 6 The basic configuration of the flowcharts is similar, and only the differences will be described below. In step S611, the screen control unit 301 determines whether the operation received in step S610 is a click operation on an area other than the area of ​​the hard keyboard UI 977. When it is determined that the received operation is a click operation ("Yes" in step S611), the process proceeds to step S613. In step S613, the CPU 201 determines whether the portable terminal 100 is currently displaying the hard keyboard UI 977. When it is determined that the hard keyboard UI 977 is currently displayed ("Yes" in step S613), the process proceeds to step S630. In step S630, the CPU 201 performs the process of hiding the hard keyboard UI 977. On the other hand, when it is determined that the hard keyboard UI 977 is not currently displayed ("No" in step S613), the process proceeds to step S614. In step S614, the CPU 201 sends the operation information to the MFP 110. When it is determined that the operation received in step S610 is a click operation on the hard keyboard UI 977, the operation information is sent from the portable terminal 100 to the MFP 110.

[0096] The exemplary embodiments described above enable the selection of whether to send received operation information to the MFP 110, depending on whether the soft keyboard is currently displayed only on the portable terminal 100 that remotely controls the MFP 110.

[0097] Although in the exemplary embodiments described above, the portable terminal 100 displays the same image as that displayed on the MFP 110 via a VNC link, this disclosure is not limited thereto. The exemplary embodiments described above are applicable to situations where, for example, the portable terminal 100 displays a magnified version of the image on the MFP 110 instead of displaying the same image as the image on the MFP 110.

[0098] Other embodiments

[0099] The embodiments of this disclosure can also be implemented by a computer that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transient computer-readable storage medium") to perform the functions of one or more embodiments described above and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more embodiments described above, and by a method executed by a computer of a system or device, for example, by reading and executing computer-executable instructions from a storage medium to perform the functions of one or more embodiments described above and / or controlling one or more circuits to perform the functions of one or more embodiments described above. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include a network of individual computers or individual processors to read and execute computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), a storage device for a distributed computing system, an optical disc (such as a CD, DVD, or Blu-ray disc), or a digital versatile disc (BD). TM ), flash memory devices, memory cards, etc.

[0100] Other embodiments

[0101] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0102] While this disclosure includes exemplary embodiments, it should be understood that this disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. A method for an information processing apparatus to communicate with an image processing apparatus, characterized by, The method includes: displaying a screen corresponding to a screen displayed on a display unit of an image processing apparatus and generated based on display control of the image processing apparatus; and when a predetermined user operation on the displayed screen is received, transmitting operation information corresponding to the predetermined user operation to the image processing apparatus, wherein in a case where the predetermined user operation on the screen is received in a state where a display form of the displayed screen is changed, the operation information corresponding to the predetermined user operation is not transmitted to the image processing apparatus, and wherein the state where the display form of the displayed screen is changed is a state where a magnification of the displayed screen is changed.

2. The method of claim 1, wherein, The predetermined user operation is an operation accompanied by movement of an indication object.

3. The method of claim 1, wherein, The predetermined user operation is a drag operation, a flick operation, or a swipe operation.

4. The method of claim 1, wherein, The operation information corresponding to the predetermined user operation includes information on a position at which the predetermined user operation is detected.

5. The method of claim 1, wherein, The information processing apparatus issues a print instruction or a scan instruction to the image processing apparatus.

6. The method of claim 1, wherein, In the display control, a screen identical to a screen displayed on a display unit of the image processing apparatus is displayed on a display unit of the information processing apparatus.

7. The method according to claim 1, wherein The information processing apparatus and the image processing apparatus establish a virtual network computing (VNC) link, and wherein the display control is control using VNC.

8. The method of claim 1, wherein, In a case where a drag operation on a screen is received by the information processing apparatus in a state where the screen is enlarged on the information processing apparatus and not enlarged on the image processing apparatus, operation information corresponding to the drag operation is not transmitted to the image processing apparatus, and an enlarged display region of the screen displayed on the information processing apparatus is changed.

9. The method according to claim 1, wherein In a case where the predetermined user operation on the screen is received in a state where a display form of the displayed screen remains unchanged, the operation information corresponding to the predetermined user operation is transmitted to the image processing apparatus.

10. The method of claim 1, wherein, In a case where an operation other than the predetermined user operation on the screen is received in a state where the display form of the displayed screen is changed, operation information corresponding to the operation other than the predetermined user operation is transmitted to the image processing apparatus.

11. The method of claim 10, wherein, In a case where an operation other than the predetermined user operation on the screen is received in a state where the display form of the displayed screen is changed, operation information corresponding to the operation other than the predetermined user operation is calculated based on at least a magnification of the screen and a position at which the operation other than the predetermined user operation is received, and the calculated operation information is transmitted to the image processing apparatus.

12. An information processing apparatus configured to communicate with an image processing apparatus, the information processing apparatus comprising: The information processing apparatus includes: a display unit configured to display a screen corresponding to a screen displayed on a display unit of an image processing apparatus and generated based on display control by the image processing apparatus; and a transmission unit configured to, in a case where a predetermined user operation on the screen displayed by the display unit is received, transmit operation information corresponding to the predetermined user operation to the image processing apparatus, wherein, in a case where the predetermined user operation on the displayed screen is received in a state where the display form of the displayed screen is changed, the transmission unit does not transmit operation information corresponding to the predetermined user operation to the image processing apparatus, and wherein the state where the display form of the displayed screen is changed is a state where the magnification of the displayed screen is changed.

13. A non-transitory computer-readable storage medium, comprising: stores one or more programs including instructions that, when executed by one or more processors of the information processing apparatus, cause the information processing apparatus, which is configured to communicate with an image processing apparatus, display a screen corresponding to a screen displayed on a display unit of the image processing apparatus and generated based on display control of the image processing apparatus; and transmit operation information corresponding to a predetermined user operation on the displayed screen to the image processing apparatus when the predetermined user operation is received, wherein, in a case where the predetermined user operation on the displayed screen is received in a state where the display form of the displayed screen is changed, the operation information corresponding to the predetermined user operation is not transmitted to the image processing apparatus, and wherein the state where the display form of the displayed screen is changed is a state where the magnification of the displayed screen is changed.

Images