Information processing apparatus, system, method for information processing apparatus, and storage medium

Abstract

Information processing apparatus, system, method for information processing apparatus, and storage medium are disclosed. A method for an information processing apparatus configured to communicate with an image processing apparatus is provided. The method includes displaying a screen of an application, wherein the screen corresponds to a screen displayed on a display unit of the image processing apparatus and is subject to display control by the image processing apparatus, and if a predetermined condition is satisfied in a state where the application is no longer operated as an operation target of a user on the information processing apparatus, sending an instruction to stop the display control to the image processing apparatus.

Description

Technical Field

[0001] This disclosure relates to remote operation of the control panel. Background Technology

[0002] As a method for remotely operating the operation screen of an image processing device, a method using Virtual Network Computing (VNC) utilizing the Remote Frame Buffer (RFB) protocol has been discussed. This method enables remote operation of the operation screen of the image processing device from a personal computer (PC) or mobile terminal. More specifically, the image processing device with VNC server functionality sends screen information about the operation screen to a terminal with VNC client functionality. The terminal displays the same screen as the operation screen displayed on the operation panel of the image processing device based on the received screen information. By performing operations on the screen displayed on the terminal, the user can log in to the image processing device on the operation screen provided by the image processing device and instruct the image processing device to perform processing without operating the operation panel of the image processing device.

[0003] Japanese Patent Application Publication No. 2007-159057 discusses a method for remotely operating an image processing device using VNC, wherein if the main operating unit of the image processing device has not received an operation for a certain period of time, the image processing device enters a power-saving mode.

[0004] Devices providing remote operation capabilities typically limit the number of devices that can remotely operate the device to one, thus limiting the number of operators to one. When using VNC to remotely operate an image processing device from a terminal, if a user has already established a VNC connection with the image processing device, other users cannot establish VNC connections with it. For example, suppose a user establishes a VNC connection with an information processing device on a terminal and operates the terminal to log in to the image processing device and instruct it to perform printing. In this case, if the user forgets to disconnect the VNC connection and switches the application that was operating as a VNC client on the terminal to a background state after processing is complete, other users will not be able to establish a VNC connection with the image processing device because the VNC connection is still established. Summary of the Invention

[0005] Embodiments of this disclosure are intended to provide a method in which another user can remotely operate the device even if the user has completed using the remote operation function of the device but has not stopped the remote operation function.

[0006] According to embodiments of this disclosure, a method for an information processing apparatus configured to communicate with an image processing device includes displaying a screen of an application, wherein the screen corresponds to a screen displayed on a display unit of the image processing device and is subject to display control by the image processing device, and sending an instruction to stop the display control to the image processing device if predetermined conditions are met when the application is no longer being operated as an operation target of a user on the information processing device.

[0007] Further features of this disclosure will become clear from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0008] Figure 1 This is a diagram illustrating a system configuration according to an exemplary embodiment.

[0009] Figure 2 This is a block diagram illustrating the hardware configuration of a mobile terminal according to an exemplary embodiment.

[0010] Figure 3 This is a block diagram illustrating the software configuration of a mobile terminal according to an exemplary embodiment.

[0011] Figure 4 This is a block diagram illustrating the hardware configuration of a multifunctional peripheral device (MFP) according to an exemplary embodiment.

[0012] Figure 5 This is a block diagram illustrating the software configuration of an MFP according to an exemplary embodiment.

[0013] Figure 6A This is a flowchart illustrating the processing performed by an application operating on a mobile terminal according to an exemplary embodiment. Figure 6B This is a flowchart illustrating a remote operation process performed by a mobile terminal according to an exemplary embodiment.

[0014] Figure 7 This is a flowchart illustrating remote operation processing performed by an MFP according to an exemplary embodiment.

[0015] Figure 8A and 8B This is a diagram illustrating the user interface (UI) displayed on the operation panel of a mobile terminal according to an exemplary embodiment.

[0016] Figure 9 This is a diagram illustrating the UI displayed on the operation panel of an MFP according to an exemplary embodiment. Detailed Implementation

[0017] Exemplary embodiments of the present disclosure will now be described with reference to the accompanying drawings. The exemplary embodiments described below are not intended to limit the present disclosure, and not all combinations of features described in the exemplary embodiments are necessarily essential to the solutions provided by the present disclosure.

[0018] <System Configuration>

[0019] First, refer to Figure 1 This section describes a system configuration according to exemplary embodiments of the present disclosure. In this exemplary embodiment, a mobile terminal 100 and a multifunction peripheral device (MFP) 110 will be described as examples of information processing apparatuses. However, the apparatus included in the information processing system is not limited thereto. The mobile terminal 100 and the MFP 110 are connected to a local area network (LAN) 10 and can exchange information with each other. The mobile terminal 100 is connected to the LAN 10 via an access point 120.

[0020] Mobile terminal 100 has Virtual Network Computing (VNC) client functionality for remotely operating MFP 110. More specifically, mobile terminal 100 includes VNC client software supporting the Remote Frame Buffer (RFB) protocol. Mobile terminal 100 can establish a VNC connection with MFP 110 and operate on the control panel 206 (see...). Figure 2 The display is shown on the MFP 110's control panel (DISPLAY) 3090 (see [link]). Figure 4 The screen displayed on the mobile terminal 100 is the same as the screen displayed on the MFP 110. The MFP 110 has VNC server functionality. When the mobile terminal 100 has established a VNC connection with the MFP 110, the MFP 110 sends screen information to the mobile terminal 100, which acts as a VNC client. The mobile terminal 100 draws a screen based on the screen information sent from the MFP 110. The screens displayed on the operation panels (display units) 206 and 3090 of the mobile terminal 100 and the MFP 110 are linked to each other, so that the screen changes based on the operation received on the drawn screen or the operation received on the MFP 110.

[0021] Next, we will refer to Figure 2 The hardware configuration of the mobile terminal 100 is described. It is assumed that the mobile terminal 100 according to this exemplary embodiment is a device such as a smartphone or a tablet personal computer (PC). Alternatively, the mobile terminal 100 may be another device, as long as it is an information processing device capable of communicating with the MFP 110.

[0022] The Central Processing Unit (CPU) 201 reads the control program stored in the Read-Only Memory (ROM) 202 and executes various types of processing for controlling the operation of the mobile terminal 100. The ROM 202 stores the control program. The Random Access Memory (RAM) 203 serves as a temporary storage area, such as the main memory or working area of ​​the CPU 201. The storage device 204 is, for example, an embedded multimedia card (eMMC), a solid-state drive (SSD), or a hard disk drive (HDD). The storage device 204 stores various types of data, such as pictures and electronic documents. The Operating System (OS) 350, which will be described below (see [link]...) Figure 3 ) and MFP Application 300 (see Figure 3 The data is also stored in storage device 204. Real-time clock (RTC) 205 keeps track of time.

[0023] Although the mobile terminal 100 is configured such that a single CPU 201 executes each of the processes shown in the flowchart described below, any other configuration may be used. For example, multiple CPUs 201 may execute each of the processes shown in the flowchart in a cooperative manner.

[0024] The operation panel 206 includes a touch panel function capable of detecting user touch operations and displays various screens provided by the OS 350 and MFP application 300. Users can input desired operation commands to the mobile terminal 100 by inputting touch operations onto the operation panel 206.

[0025] Speaker 207 and microphone 208 are used, for example, when a user calls another mobile terminal or subscriber phone. Camera 209 captures images based on the user's image capture command. The images captured by camera 209 are stored in a predetermined area of ​​storage device 204. Wireless communication unit 211 performs wireless communication such as wireless LAN communication.

[0026] Next, we will refer to Figure 3 Describe the software configuration of mobile terminal 100. Figure 3 It is a functional block diagram of the software implemented by the CPU 201 reading the control program stored in the ROM 202 or storage device 204.

[0027] OS 350 is software used to control the overall operation of mobile terminal 100. Various applications, including MFP application 300 (described below), can be installed on mobile terminal 100. OS 350 exchanges information with the applications and changes the screen displayed on operation panel 206 based on instructions received from the applications. OS 350 also includes a set of device drivers for controlling various hardware blocks and provides application programming interfaces (APIs) for using the hardware blocks to applications operating on OS 350. In this exemplary embodiment, for example, wireless LAN control unit 351 corresponds to this set of device drivers. Wireless LAN control unit 351 is a device driver for controlling wireless communication unit 211.

[0028] MFP Application 300 is an application installed on mobile terminal 100. MFP Application 300 can issue print and scan commands to MFP 110. As mentioned above, MFP Application 300 also functions as a VNC client to display the operation screen of MFP 110 and remotely operate MFP 110. Although various applications other than MFP Application 300 can be installed on mobile terminal 100, their descriptions will be omitted here.

[0029] The configuration of the MFP application 300 will be described in more detail. The screen control unit 301 controls the screen displayed on the operation panel 206 via the OS 350. The screen of the MFP application 300 is displayed on the operation panel 206 by the screen control unit 301. The screen control unit 301 also receives operation commands input by the user via the operation panel 206. The communication unit 302 controls the wireless communication of the wireless communication unit 211 via the OS 350, and sends and receives information from external devices such as the MFP 110.

[0030] The device search control unit 311 generates search data for searching for external devices to connect to the mobile terminal 100, and transmits the search data to the external devices on the LAN 10 via the wireless communication unit 211. The device search control unit 311 receives responses from the external devices that respond to the search. The search data can be broadcast to all external devices on the LAN 10, or it can be sent to an external device by specifying an address.

[0031] The print job generation unit 305 generates a print job. The print job generated by the print job generation unit 305 is sent to the MFP 110 via the wireless communication unit 211. The MFP 110 performs printing based on the received print job.

[0032] The scan job control unit 307 sends a scan command to the MFP 110 via the wireless communication unit 211. The MFP 110 performs a scan based on the scan command and sends the data generated by the scan (scan data) to the mobile terminal 100. The mobile terminal 100 displays the received scan data. When the scan data needs to be saved, it is stored in the application storage unit 306. The stored scan data is managed by the document management unit 308.

[0033] The remote operation unit 313 displays a screen based on screen information obtained from the MFP 110, which is connected to the mobile terminal 100 via VNC, and receives operations on the displayed screen. In other words, the remote operation unit 313 operates as a VNC client. More specifically, the remote operation unit 313 displays a screen based on screen information received from the VNC server and sends operation information, including the location and type of the operations received on the displayed screen, to the MFP 110. The MFP 110 changes the screen displayed on the MFP 110 and the screen displayed on the mobile terminal 100 based on the received operation information. The application storage unit 306 temporarily stores various types of information generated by the MFP application 300. The timing unit 304 keeps track of time.

[0034] Figure 4 An example of the hardware configuration of the MFP 110 is illustrated. The MFP 110 includes a CPU 3020 that executes programs stored in ROM 3030 or a mass storage device (HD) 3110 such as a hard disk. The CPU 3020 centrally controls devices connected to the system bus 3010.

[0035] RAM 3040 serves as the main memory or working area for CPU 3020. External Input Controller (PANELC) 3060 controls commands input via various hardware keys 3120 or touch panel (PANEL) 3070 provided on MFP 110. Display Controller (DISPC) 3080 controls the display on a display module (DISPLAY) 3090, such as an LCD screen. Disk Controller (DKC) 3100 controls mass storage device (HD) 3110.

[0036] Now refer to Figure 9 The operating unit 900 of the MFP 110 is described. A screen is displayed on the display 3090. Hardware keys 3120 are located next to the display 3090. Both the touch panel 3070 and the hardware keys 3120 on the display 3090 can receive user input. The logout key 901 (one of the hardware keys 3120) is pressed by the user logged into the MFP 110 to instruct the MFP 110 to perform a logout process.

[0037] Return to Figure 4 The Network Interface Card (NIC) 3050 exchanges data bidirectionally with external devices (such as mobile terminal 100) or a file server via LAN 10. The Wireless Communication Module (WLAN) 3400 connects to or operates in access point mode with access point 120, enabling the MFP 110 to operate as an access point and establish a direct wireless communication connection with mobile terminal 100. The printer 3200 is a paper printing unit using an electrophotographic method. The printing method is not limited to electrophotographic methods. The scanner 3300 is an image reading unit for reading images printed on paper. The scanner 3300 is typically equipped with an Automatic Document Feeder (ADS) (not shown) as an option and can automatically read multiple document pages. In some cases, a mass storage device (HD) 3110 is used as a temporary storage location for images.

[0038] Figure 5 This is a block diagram illustrating the software configuration of the MFP 110. The software is stored, for example, in ROM 3030 and executed by CPU 3020.

[0039] The user interface (UI) processing unit 510 analyzes input from the external input controller 3060 or operation information received from the operation information receiving unit 522 (described below). When the screen needs to be changed, the UI processing unit 510 generates image data and displays the image data on the display 3090.

[0040] Display 3090 shows as follows Figure 9 The various screens shown are illustrated. Examples of screens include ID card login screen 920, keyboard login screen 910, menu screen 930 to be displayed during login, error screen (not shown) to be displayed when an error occurs, and counter screen (not shown) for displaying counter information. The Display 3090 is hidden when the MFP 110 is in sleep mode.

[0041] ID card login screen 920 is used to log in to the MFP 110 using an ID card associated with user information. When the ID card is placed on the ID card reader of the MFP 110 while ID card login screen 920 is displayed, user information is read and login processing is performed. When the keypad login button 921 is pressed, ID card login screen 920 switches to keypad login screen 910.

[0042] Keyboard login screen 910 is used to log in to the MFP 110 by entering user information using the keyboard. The login process is performed by entering user information into the username input field 911 and password input field 912 using the hardware key 3120 or the software keyboard (not shown), and then pressing the login button 913. When the ID card login button 914 is pressed, keyboard login screen 910 switches to ID card login screen 920.

[0043] After the user logs into the MFP110 via the keyboard login screen 910 or the ID card login screen 920, the menu screen 930 is displayed. The user can use the various functions of the MFP110 by pressing the copy button 931, the scan button 932, or the print button 933, according to the intended use. When the user presses the exit button 934, the exit process is performed.

[0044] Return to Figure 5 The remote operation processing unit 520 includes modules such as a connection processing unit 523, an operation information receiving unit 522, and a screen sending unit 521. The remote operation processing unit 520 is a VNC server module that performs communication using the RFB protocol; that is, a software module that communicates with the mobile terminal 100, which operates as a VNC client, via LAN 10. Upon receiving a connection request from the MFP 110 or the mobile terminal 100 acting as a VNC client, the connection processing unit 523 performs connection processing. Upon receiving a connection disconnection request from the MFP 110 or the mobile terminal 100, the connection processing unit 523 performs disconnection processing.

[0045] The operation information receiving unit 522 receives screen operation information from the mobile terminal 100 or MFP 110 and notifies the UI processing unit 510 of the screen operation information. The screen sending unit 521 is a display control unit that performs screen display control by reading image data generated by the UI processing unit 510 from RAM 3040 and sending the image data to the mobile terminal 100, which is used as a VNC client.

[0046] The login processing unit 530 is a module for performing login processing on the MFP 110. The login processing unit 530 performs user authentication based on user information received from the UI processing unit 510, and performs login if authentication is successful. If the login processing unit 530 receives an exit command from the UI processing unit 510, it performs exit processing. The network control unit 500 is a module for controlling the NIC 3050. Although in Figure 5 Not shown, but the MFP 110 also includes a printer module for controlling printer 3200 and a scanner module for controlling scanner 3300.

[0047] Figure 8A and 8B The illustration shows an example of a screen displayed on the operation panel 206 of the mobile terminal 100. When the mobile terminal 100 is activated, the main screen 810 appears. The MFP application icon 811 is used to activate the MFP application 300. When the MFP application icon 811 is operated, the MFP application 300 is activated.

[0048] When MFP application 300 is activated, the application's main screen (with no device selected) 820 appears. The selected device icon 821 displays information about the target MFP with which MFP application 300 is currently communicating and to which the print job will be sent. If no target MFP is selected, the selected device icon 821 displays "No printer". The device search button 823 is used to search for a target MFP. The remote operation button 822 is used to establish a VNC connection with the target MFP and to remotely operate the target MFP. The application's main screen (with no device selected) 820 may also display other buttons, such as a print button 824 for sending print jobs to the target MFP and a button for issuing scan commands.

[0049] When the device search button 823 is pressed, a search is performed on the MFPs on LAN 10, and the device search screen 830 appears to display the search results. More specifically, the device search screen 830 displays a list of MFPs that have responded to the search. For example, selecting a search result 831 from the displayed list allows the selection of a target MFP.

[0050] When a target MFP is selected, the application's main screen (where the device is selected) 840 appears. The selected device icon 821 displays information about the MFP selected on the device search screen 830. If the user clicks the remote operation button 822 while the target MFP is selected, the remote operation function of the target MFP is activated. The following description assumes that the target MFP is MFP 110.

[0051] When the remote operation function is activated, the device connection screen 850 appears. When the hardware key display button 851 is pressed, the UI for operating the MFP 110 via hardware key 3120 appears. Details are described below.

[0052] When the VNC connection is established and the remote operation function is set up, the screen displayed on the DISPLAY 3090 of the MFP 110 is displayed on the operation panel 206. Here, a keyboard login screen 860, similar to the keyboard login screen 910, is displayed on the operation panel 206. The screen content is the same as that of the keyboard login screen 910 on the MFP 110, except that the hardware key display button 851 is displayed on the operation panel 206. Therefore, its description will be omitted. If the ID card login screen 920 is displayed on the MFP 110, then the ID card login screen 865 is displayed on the operation panel 206.

[0053] When a login command is given on the keyboard login screen 860, a menu screen similar to the menu screen 930 (with hardware keys hidden) 870 is displayed on the operation panel 206. At this time, the menu screen 930 is also displayed on the display 3090.

[0054] If the hardware key display button 851 is clicked, a menu screen (where the hardware keys are displayed) 875 is displayed. In other words, a software keyboard assigning functions to the various hardware keys 3120 of the MFP 110 is displayed. The hardware key UI 877 is the UI used to operate the hardware keys 3120 of the MFP 110. If a key displayed on the hardware key UI 877 is clicked, information about the clicked key is sent to the MFP 110, and the MFP 110 performs processing based on the clicked key. If the exit (ID) key 879 corresponding to the exit key 901 of the MFP 110 is clicked, the user exits the MFP 110. If the end button 878 on the hardware key UI 877 is clicked, the remote operation function ends, i.e., the VNC connection ends. After the VNC connection ends, the application main screen (where the device is selected) 840 is displayed on the operation panel 206. After the VNC connection ends, the Display 3090 continues to display the screen that was displayed before the VNC connection ended.

[0055] If a user who has finished using the remote operation function keeps the MFP application 300, which is being operated as a VNC client, in a non-foreground state (e.g., in the background) without performing the operation to terminate the VNC connection, the following problem arises. Devices with login configurations, such as the MFP 110, typically limit the number of external devices that can establish a VNC connection with that device to one, thus limiting the number of operators to one. If the user forgets to perform the operation to disconnect the VNC connection after the remote operation function ends, other users will not be able to establish a VNC connection with the MFP 110. For example, suppose a user initiates the remote operation function, instructs the MFP 110 to perform a print or scan operation from the mobile terminal 100, then forgets to press the end button 878 and uses another application without pressing the end button 878 (which switches the MFP application 300 to the background). In this case, because the VNC connection between the mobile terminal 100 and the MFP 110 continues, other users will not be able to establish a VNC connection with the MFP 110. To address this issue, mobile terminal 100 performs control such that if the MFP application 300, acting as a VNC client, is no longer in the foreground on mobile terminal 100, another device can establish a VNC connection with MFP 110. For example, if a predetermined time has elapsed since the MFP application 300 transitioned from the foreground state on mobile terminal 100, the VNC connection is either disconnected or the "priority state" of the VNC connection (described in detail below) is canceled. In this exemplary embodiment, the foreground state of MFP application 300 refers to the state in which MFP application 300 is active, such as when MFP application 300 is the target of a user's operation on the information processing device (mobile terminal 100). In other words, MFP application 300 is displayed on the information processing device and performs processing in response to a user's operation, such as a click. When MFP application 300 is in the foreground state, the OS 350 of the information processing device prioritizes the processing of MFP application 300. The background state of MFP application 300 refers to the state in which MFP application 300 is running but inactive and is not the target of a user's operation.

[0056] Figure 6A This is a flowchart illustrating the process of establishing a VNC connection between mobile terminal 100 and MFP 110, performing remote operations, and then disconnecting the VNC connection. Figure 6A The steps shown in the flowchart are processed by CPU 201 loading the control program stored in a memory such as ROM 202 into RAM 203 and executing the control program.

[0057] In step S601, when the screen control unit 301 detects a click on the MFP application icon 811 on the main screen 810, the MFP application 300 is activated. In step S602, the screen control unit 301 detects a click on the device search button 823 on the application main screen (where no device is selected) 820 and performs a search for MFPs. In step S603, the screen control unit 301 determines whether an MFP has been selected on the device search screen 830. If the screen control unit 301 determines that an MFP has been selected on the device search screen 830 ("Yes" in step S603), then the application main screen (where no device is selected) 820 is switched to the application main screen (where a device is selected) 840 and the process proceeds to step S604. If not ("No" in step S603), then the process returns to step S603.

[0058] In step S604, the screen control unit 301 determines whether the remote operation button 822 has been selected on the application main screen (where the device is selected) 840. If the screen control unit 301 determines that the remote operation button 822 has been selected on the application main screen (where the device is selected) 840 (yes in step S604), then the process proceeds to step S605. If not (no in step S604), then the process returns to step S604.

[0059] In step S605, the remote operation unit 313 sends a connection request using the RFB protocol to the MFP selected in step S603. In step S606, the remote operation unit 313 performs remote operation processing.

[0060] Reference Figure 6B The remote operation process in step S606 is described in detail. First, in step S607, the remote operation unit 313 determines whether the connection request using the RFB protocol is accepted. If the connection request is accepted ("Yes" in step S607), the process proceeds to step S609. If the connection request is not accepted ("No" in step S607), the process proceeds to step S608. Whether the connection request is accepted is determined based on the response data regarding the connection request. In step S608, the screen control unit 301 displays a connection rejection error screen 880 (see...). Figure 8B ), and the processing is complete.

[0061] In step S609, the remote operation unit 313 performs initialization processing based on the RFB protocol. During initialization, the remote operation unit 313 sends VNC client information to the MFP 110 and receives VNC server information from the MFP 110.

[0062] In step S610, the remote operation unit 313 displays a screen on the operation panel 206 based on the screen information received from the MFP 110 (i.e., the VNC server). For example, if the screen displayed on the display 3090 is the keyboard login screen 910, then the keyboard login screen 860 is displayed on the operation panel 206. The remote operation unit 313 displays a screen in step S610 whenever screen information is sent from the MFP 110.

[0063] In step S611, the remote operation unit 313 determines whether the mobile terminal 100 is in a priority state. If the mobile terminal 100 is in a priority state ("Yes" in step S611), the process proceeds to step S612. If not ("No" in step S611), the process proceeds to step S617. The term "priority state" refers to a state where, when the mobile terminal 100 establishes a VNC connection with the MFP 110, other devices are rejected from establishing a VNC connection with the MFP 110. For example, the VNC connection has two modes: first priority mode and last priority mode. In first priority mode, VNC connections with previously connected devices are given priority, and any other devices that subsequently request a VNC connection are prevented from connecting. In last priority mode, VNC connections with the last device that requested a VNC connection are given priority. In other words, if another device subsequently requests a VNC connection, the VNC connection with the previously connected device is disconnected, and a VNC connection is established with the other device that subsequently requested a VNC connection. In the first-priority mode, since subsequent connection requests can be rejected after a VNC connection is established, the mobile terminal 100 enters a "priority state" once a VNC connection has been successfully established. In contrast, in the last-priority mode, even if a VNC connection has been successfully established, the VNC connection will be disconnected if a subsequent VNC connection request is received from another device. Therefore, in the last-priority mode, for example, the following control is implemented: The mobile terminal 100 enters a "priority state" after a predetermined time has elapsed since the VNC connection was established. While the mobile terminal 100 is in the "priority state," if the VNC connection is disconnected due to a subsequent VNC connection request from another device, the mobile terminal 100 automatically re-establishes the VNC connection to restore it. This control obviously prevents devices that subsequently attempt to establish a VNC connection from doing so. In the last-priority mode, this state will be referred to as the "priority state." Although an example of the mobile terminal 100 entering a "priority state" after a predetermined time has elapsed since the VNC connection was established has been described above, this is not limiting. If mobile terminal 100 establishes a VNC connection with MFP 110 and the user logs into MFP 110, then mobile terminal 100 can enter "priority state".

[0064] In step S612, the remote operation unit 313 determines whether the MFP application 300 has transitioned to a background state on the mobile terminal 100. If the MFP application 300 is in the background state ("Yes" in step S612), the process proceeds to step S613. If not, i.e., the MFP application 300 is in the foreground state ("No" in step S612), the process proceeds to step S617. If the MFP application 300 has transitioned to a background state, the OS 350 notifies the MFP application 300 of the transition to the background state, and the MFP application 300 detects the transition.

[0065] In step S613, the remote operation unit 313 determines whether the VNC connection is in the first-priority mode or the last-priority mode. The remote operation unit 313 can query the MFP 110 to determine which mode the VNC connection is in. If the VNC connection is in the first-priority mode ("Yes" in step S613), the process proceeds to step S614. If the VNC connection is in the last-priority mode ("No" in step S613), the process proceeds to step S615.

[0066] In step S614, the remote operation unit 313 cancels the priority state. In step S620, the remote operation unit 313 performs the process for disconnecting the VNC connection. In the first-priority mode, once a VNC connection has been successfully established, the mobile terminal 100 enters the "priority state." Therefore, canceling the "priority state" is synonymous with disconnecting the VNC connection. In this example, if the MFP application 300 has transitioned to the background state, the process proceeds immediately to steps S614 and S620. Alternatively, the process can proceed to steps S614 and S620 after a predetermined time has elapsed. In other words, the VNC connection can be disconnected immediately after the MFP application 300 transitions to the background state, or it can be disconnected after a predetermined time has elapsed since transitioning to the background state.

[0067] In step S615, the remote operation unit 313 cancels the "priority state." Canceling the "priority state" means entering a state where, even if another device later establishes a VNC connection with MFP 110, mobile terminal 100 does not perform the reconnection process to restore the VNC connection. In other words, mobile terminal 100 transitions from a "priority state" where, if another device establishes a VNC connection with MFP 110, mobile terminal 100 reconnects to restore the VNC connection, to a state where, even if another device establishes a VNC connection with MFP 110, mobile terminal 100 does not perform the reconnection process to restore the VNC connection. This allows another device to establish a VNC connection with MFP 110. The process then proceeds to step S616.

[0068] In step S616, the remote operation unit 313 determines whether the MFP application 300 has returned to the foreground state within a certain time period. If the MFP application 300 has returned to the foreground state within the certain time period ("Yes" in step S616), the process proceeds to step S617. If not ("No" in step S616), the process proceeds to step S620. In step S620, the remote operation unit 313 performs a process for disconnecting the VNC connection. The disconnection process refers to, for example, sending the operation information corresponding to the end button 878 to the MFP 110 and requesting the MFP 110 to disconnect the VNC connection.

[0069] In step S617, the remote operation unit 313 determines whether an input operation has been performed on the mobile terminal 100. If an input operation has been performed on the mobile terminal 100 (yes in step S617), the process proceeds to step S618. If not (no in step S617), the process proceeds to step S611. In step S618, the remote operation unit 313 sends operation information regarding the operation received on the mobile terminal 100 to the MFP 110.

[0070] In step S619, the remote operation unit 313 determines whether a command to disconnect the VNC connection has been received. If a command to disconnect the VNC connection is received (yes in step S619), the process proceeds to step S620. If not (no in step S619), the process proceeds to step S611. The command to disconnect the VNC connection is given, for example, by pressing the end button 878.

[0071] According to the above processing, when a VNC connection is established between the mobile terminal 100 and the MFP 110, even if the MFP application 300, operating as a VNC client, transitions to a background state on the mobile terminal 100 in response to the user ending processing, and the user forgets to give a command to disconnect the VNC connection, another device can still establish a VNC connection with the MFP 110. Furthermore, since the VNC connection between the mobile terminal 100 and the MFP 110 can be disconnected after a predetermined condition is met (i.e., a predetermined time has elapsed since the MFP application 300 transitioned to a background state), if another application (such as a telephone application) is used during the VNC connection, the mobile terminal 100 can maintain the VNC connection and continue operating the MFP 110 as long as the predetermined time has not elapsed.

[0072] As an example, the above processing has been described using the scenario where the VNC connection is disconnected if the MFP application 300 switches to a background state. However, this is not limiting, and the VNC connection can also be disconnected if the MFP application 300 is closed by the user. In this case, when the MFP application 300 is closed, the mobile terminal 100 sends a VNC disconnect request to the MFP 110 to immediately (rather than after a predetermined time) disconnect the VNC connection.

[0073] Next, we will refer to Figure 7 This describes the processing performed by the MFP 110 for screen linking with the mobile terminal 100 using VNC. Figure 7 The steps shown in the flowchart involve the CPU 3020 loading a control program stored in a memory such as ROM 3030 into RAM 3040 and executing the control program for processing.

[0074] In step S701, the connection processing unit 523 determines whether a VNC connection request has been received from the mobile terminal 100, which is being used as a VNC client. If a VNC connection request is received from the mobile terminal 100 (yes in step S701), the process proceeds to step S702. If no VNC connection request is received from the mobile terminal 100 (no in step S701), the process returns to step S701. In step S702, the connection processing unit 523 performs a negotiation process and connects to the VNC client. In step S703, the connection processing unit 523 performs an initialization process to receive client information from the mobile terminal 100 and send server information to the mobile terminal 100.

[0075] In step S704, the screen sending unit 521 sends screen information about the screen displayed on the DISPLAY 3090 to the mobile terminal 100, which is used as a VNC client.

[0076] In step S705, the operation information receiving unit 522 determines whether an operation has been received from the user. If an operation has been received from the user ("Yes" in step S705), the process proceeds to step S706. If not ("No" in step S705), the process returns to step S705. For example, if operation information is received from the mobile terminal 100 or if a user operation is detected via the hardware key 3120 of the PANEL 3070 or MFP 110, then the operation information receiving unit 522 determines that an operation has been received from the user.

[0077] In step S706, the operation information receiving unit 522 determines whether the operation received in step S705 is a VNC connection disconnection operation. If the received operation is a VNC connection disconnection operation ("Yes" in step S706), the process proceeds to step S707. In step S707, the connection processing unit 523 disconnects the VNC connection. If not ("No" in step S706), the process proceeds to step S708.

[0078] In step S708, processing corresponding to the operation received in step S705 is performed. Examples of processing include processing to change the menu screen displayed on DISPLAY 3090 to the next page, processing to enable the user to log in to MFP 110, and processing to enable the user to log out of MFP 110. In step S709, the operation information receiving unit 522 determines whether the screen displayed on DISPLAY 3090 has changed due to the processing performed in step S708. If the screen has changed (yes in step S709), the processing proceeds to step S704. If not (no in step S709), the processing proceeds to step S705.

[0079] By using the above processing, the screen displayed on the MFP 110 can also be displayed on the mobile terminal 100. The screens displayed on the MFP 110 and the mobile terminal 100 can be changed based on an operation received by the MFP 110 or the mobile terminal 100. The above processing has been described using the case where the screen displayed on the mobile terminal 100 changes based on an operation received by the MFP 110 or the mobile terminal 100. Moreover, if the screen displayed on the MFP 110 changes without an operation received by the MFP 110 or the mobile terminal 100, the MFP 110 similarly sends the changed screen information to the mobile terminal 100, and the mobile terminal 100 displays the changed screen. The processing in the above flowchart enables the VNC connection to be disconnected in response to a VNC disconnect request from the mobile terminal 100. If the user is still logged into the MFP 110 when the VNC connection is disconnected, an automatic logout process can be performed. Furthermore, the MFP 110 can automatically perform an automatic cleanup process when the VNC connection is disconnected. Automatic clearing process refers to the process used to restore the displayed screen to the initial screen that has been set.

[0080] According to the exemplary embodiments described above, even if a user switches the MFP application 300, which operates as a VNC client, to a background state without performing the processing for disconnecting the VNC connection, another user can still establish a VNC connection. In the exemplary embodiments described above, the mobile terminal 100 considers the "priority state" of the mobile terminal 100 to determine whether to disconnect the VNC connection and sends a request to disconnect the VNC connection. However, this is not limiting. The MFP 110 may consider the "priority state" of the mobile terminal 100 to determine whether to disconnect the VNC connection and disconnect the VNC connection. In this case, the MFP 110 obtains information from the mobile terminal 100 instructing the MFP application 300 to transition to a background state and determines whether to disconnect the VNC connection.

[0081] In the exemplary embodiments described above, an example has been described of displaying the same screen on the mobile terminal 100 as the screen displayed on the MFP 110 using a VNC connection. However, this is not limiting. For example, the exemplary embodiments described above can also be applied to situations where the mobile terminal 100 displays a screen obtained by extending the screen on the MFP 110, instead of displaying the same screen as the MFP 110.

[0082] Other embodiments

[0083] Embodiments of this disclosure can also be implemented by a computer of a system or apparatus that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a 'non-transitory computer-readable storage medium') to perform one or more functions of the above embodiments and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions of the above embodiments, and by a method performed by a computer of the system or apparatus by, for example, reading and executing computer-executable instructions from the storage medium to perform one or more functions of the above embodiments and / or controlling one or more circuits to perform one or more functions of the above embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include separate computers or networks of separate processors to read and execute computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), storage devices for distributed computing systems, optical discs (such as CDs, DVDs, or Blu-ray discs). TM One or more of the following: flash memory devices, memory cards, etc.

[0084] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0085] While this disclosure includes exemplary embodiments, it is to be understood that this disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be given the broadest interpretation in order to cover all such modifications and equivalent structures and functions.

Claims

1. A method for an information processing apparatus configured to communicate with an image processing apparatus, the method comprising: displaying a screen of an application, wherein the screen corresponds to a screen displayed on a display unit of the image processing apparatus and is subject to display control by the image processing apparatus; and if a predetermined condition is satisfied in a state where the application is no longer operated as an operation target of a user on the information processing apparatus, sending an instruction to stop the display control to the image processing apparatus, wherein in a case where a predetermined connection setting between the image processing apparatus and the information processing apparatus is a first setting, the predetermined condition is that a predetermined time elapses in the state where the application is no longer operated as an operation target of a user on the information processing apparatus, and in a case where the predetermined connection setting is a second setting, the predetermined condition is that the application is not operated as an operation target of a user for the predetermined time in the state where the application is no longer operated as an operation target of a user on the information processing apparatus.

2. The method of claim 1, wherein, the state where the application is no longer operated as an operation target of a user on the information processing apparatus is a state where the application is no longer operated in a foreground state on the information processing apparatus.

3. The method of claim 1, wherein, the state where the application is no longer operated as an operation target of a user on the information processing apparatus is a state where the application is being operated in a background state on the information processing apparatus.

4. The method of claim 1, wherein, the state where the application is no longer operated as an operation target of a user on the information processing apparatus is a state where operation of the application is stopped.

5. The method of claim 1, wherein, the instruction to stop the display control is an instruction to disconnect communication for performing the display control.

6. The method of claim 1, wherein, the application is an application configured to issue a print instruction or a scan instruction to the image processing apparatus.

7. The method of claim 1, wherein, the display control is control for displaying the same screen on a display unit of the information processing apparatus as a screen displayed on a display unit of the image processing apparatus.

8. The method of claim 1, wherein, the information processing apparatus and the image processing apparatus establish a virtual network computing (VNC) connection with each other, and the display control is control using VNC.

9. The method according to claim 1, wherein the instruction to stop the display control is not sent to the image processing apparatus in a case where the application starts to be operated as an operation target of a user before the predetermined time elapses in the state where the application is no longer operated as an operation target of a user on the information processing apparatus.

10. The method according to claim 1, wherein the first setting is a state where control is performed so that a connection between the image processing apparatus and another apparatus cannot be established when a predetermined connection is active, and the second setting is a state where control is performed so that a connection between the image processing apparatus and another apparatus can be established when the predetermined connection is active.

11. The method according to claim 1, wherein In a case where the predetermined connection setting is the second setting, in a case where the predetermined connection is valid at the time of establishing a connection between the image processing apparatus and another apparatus, the information processing apparatus operates in a priority state in which the predetermined connection is automatically executed again.

12. The method according to claim 1, wherein, In a case where the predetermined connection setting is the second setting, in a case where the application no longer operates on the information processing apparatus as an operation target of a user, a priority state is cleared.

13. A system comprising: an information processing apparatus; and an image processing apparatus, wherein the system includes: a display control unit configured to perform display control so that a screen corresponding to a screen displayed on a display unit of the image processing apparatus is displayed on a display unit of the information processing apparatus, and a stop unit configured to stop display control of the display control unit if a predetermined condition is satisfied in a state where an application operating on the information processing apparatus and configured to display a screen subjected to the display control of the display control unit no longer operates on the information processing apparatus as an operation target of a user, wherein in a case where a predetermined connection setting between the image processing apparatus and the information processing apparatus is a first setting, the predetermined condition is that a predetermined time elapses in the state where the application no longer operates on the information processing apparatus as an operation target of a user, and in a case where the predetermined connection setting is a second setting, the predetermined condition is that the predetermined time elapses without the application operating on the information processing apparatus as an operation target of a user in the state where the application no longer operates on the information processing apparatus as an operation target of a user.

14. The system of claim 13, wherein, an instruction to stop display control of the display control unit is transmitted from the information processing apparatus to the image processing apparatus if the predetermined condition is satisfied in the state where the application no longer operates on the information processing apparatus as an operation target of a user.

15. The system of claim 13, wherein, information indicating the state is transmitted from the information processing apparatus to the image processing apparatus if the predetermined condition is satisfied in the state where the application no longer operates on the information processing apparatus as an operation target of a user, and the stop unit stops display control of the display control unit based on the transmitted information.

16. An information processing apparatus configured to communicate with an image processing apparatus, the information processing apparatus comprising: a display unit configured to display a screen corresponding to a screen displayed on a display unit of the image processing apparatus and subjected to display control of the image processing apparatus; and a transmission unit configured to transmit an instruction to stop the display control to the image processing apparatus if a predetermined condition is satisfied in a state where an application configured to display a screen subjected to the display control no longer operates on the information processing apparatus as an operation target of a user, wherein In a case where the predetermined connection setting between the image processing apparatus and the information processing apparatus is a first setting, the predetermined condition is that a predetermined time elapses in a state where the application is no longer operated as an operation target of the user on the information processing apparatus, and In a case where the predetermined connection setting is a second setting, the predetermined condition is that the predetermined time elapses in a state where the application is not operated as an operation target of the user after the application is no longer operated as an operation target of the user on the information processing apparatus.

17. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an information processing apparatus configured to communicate with an image processing apparatus, cause the information processing apparatus to perform a method comprising: displaying a screen of an application, wherein the screen corresponds to a screen displayed on a display unit of the image processing apparatus and is subjected to display control of the image processing apparatus; and if a predetermined condition is satisfied in a state where the application is no longer operated as an operation target of the user on the information processing apparatus, then an instruction to stop the display control is transmitted to the image processing apparatus, wherein, in a case where the predetermined connection setting between the image processing apparatus and the information processing apparatus is a first setting, the predetermined condition is that a predetermined time elapses in a state where the application is no longer operated as an operation target of the user on the information processing apparatus, and in a case where the predetermined connection setting is a second setting, the predetermined condition is that the predetermined time elapses in a state where the application is not operated as an operation target of the user after the application is no longer operated as an operation target of the user on the information processing apparatus.

Images