Information processing device, information processing system, information processing method, and program

Abstract

To improve the operability of remote operation of image forming apparatus. [Solution] The information processing device of the present disclosure is an information processing device capable of remote operation of an image forming apparatus, and includes a notification means for notifying notification information regarding the progress of the remote connection, wherein the notification information is determined based on first information indicating the status of the remote operation between the information processing device and the image forming apparatus, and second information indicating the usage status of the image forming apparatus.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing system, an information processing method, and a program.

Background Art

[0002] There is a function called remote operation that allows an operation panel of an image forming apparatus such as a printer to be operated from an information processing apparatus located remotely. This is realized, for example, by using software called VNC (Virtual Network Computing). VNC includes server software and client software. The VNC server software is started on the server side, and the client side starts the VNC client software to connect to the server and perform remote operations on the server. The RFB (Remote Frame Buffer) protocol (RFC6143) is used for communication between the VNC server and the client. If the VNC server software is started on the image forming apparatus, an information processing apparatus such as a PC or a mobile terminal can remotely operate the image forming apparatus using the VNC client software.

[0003] It is also possible to remotely operate the image forming apparatus by connecting to the image forming apparatus using a web browser of the information processing apparatus and connecting to the VNC server of the image forming apparatus via the web browser.

[0004] Regarding remote operation of an image forming apparatus (remote operation), Patent Document 1 is disclosed. Patent Document 1 describes a technique in which, when a remote user who uses an image forming apparatus by remote operation receives a disconnection request for the remote connection while logged in to the image forming apparatus, the image forming apparatus automatically executes logout processing for the remote user.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-073282 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, there is a need to further improve the operability of remote operation of image forming apparatus.

[0007] The purpose of this disclosure is to improve the operability of remote operation of an image forming apparatus. [Means for solving the problem]

[0008] This disclosure relates to an information processing device capable of remote operation of an image forming apparatus, comprising notification means for notifying notification information regarding the progress of a remote connection, wherein the notification information is determined based on first information indicating the status of the remote operation between the information processing device and the image forming apparatus, and second information indicating the usage status of the image forming apparatus. [Effects of the Invention]

[0009] The information processing device disclosed herein makes it possible to improve the operability of remote operation of an image forming apparatus. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the system configuration of the information processing system disclosed herein. [Figure 2] This is a hardware configuration diagram of an image forming apparatus. [Figure 3] This is a software configuration diagram for an image forming apparatus. [Figure 4] This is a hardware configuration diagram for a PC. [Figure 5] This is a software configuration diagram for a PC. [Figure 6] This diagram shows the functional configuration of an information processing system. [Figure 7] It is a diagram showing an example of notification information (message). [Figure 8] It is a diagram showing an example of setting values for remote operation settings. [Figure 9] It is a diagram showing an example of a remote operation setting screen. [Figure 10] It is a flowchart explaining the display process of a remote UI screen. [Figure 11] It is a diagram showing an example of the top page of a remote UI screen. [Figure 12] It is a flowchart explaining the display process of a remote operation screen. [Figure 13] It is a diagram showing an example of a remote operation screen before the start of remote operation. [Figure 14] It is a diagram showing another example of a remote operation screen before the start of remote operation. [Figure 15] It is a flowchart explaining the start process of remote operation. [Figure 16] It is a diagram showing an example of a remote operation screen during preparation for remote operation execution. [Figure 17] It is a diagram showing another example of a remote operation screen during preparation for remote operation execution. [Figure 18] It is a diagram showing an example of a confirmation screen for permission / denial of remote operation. [Figure 19] It is a diagram showing an example of a remote operation screen during remote operation execution. [Figure 20] It is a diagram showing another example of a remote operation screen during remote operation execution. [Figure 21] It is a diagram showing an example of a remote operation screen for notifying a connection rejection error. [Figure 22] It is a flowchart explaining the login process to an image forming apparatus. [Figure 23]It is a diagram showing an example of a login screen. [Figure 24] It is an example of a home screen displayed on an operation unit of an image forming apparatus. [Figure 25] It is a flowchart for explaining a logout process of an image forming apparatus. [Figure 26] It is a flowchart for explaining an end process of remote operation. [Figure 27] It is a diagram showing an example of a top page of a remote UI screen in a modified example. [Figure 28] It is a diagram showing an example of a remote operation screen during remote operation in a modified example. [Figure 29] It is a diagram showing a functional configuration of an information processing system in a second embodiment.

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Conventionally, a user (remote user) of an information processing apparatus who uses an image forming apparatus by remote operation may wait in a state where they cannot recognize how much time is required until a remote connection between the image forming apparatus and the information processing apparatus is established. Therefore, in the present disclosure, the usage status of the image forming apparatus by a user other than the remote user is displayed on the top screen of the remote UI. Thereby, the operability of the remote operation by the remote user is improved. In the following embodiments, as an example, an information processing system including an image forming apparatus having a web server function and a VNC server function, and a personal computer (PC) which is an information processing apparatus having a web browser function will be described. In the information processing system, the PC accesses the image forming apparatus using a web browser and connects to the VNC server of the image forming apparatus via the web browser, thereby enabling remote operation of the image forming apparatus.

[0012] <First Embodiment> Figure 1 shows an example of the system configuration of the information processing system 100 according to the first embodiment. In the information processing system 100, the image forming apparatus 102 and the PC 101 are communicated to each other via a network 103. The image forming apparatus 102 has a web server function and a VNC server function and transmits display screen data to the PC 101. The PC 101 has a web browser function and receives display screen data from the image forming apparatus 102 and displays it on the display unit of the PC 101. The network 103 includes LAN (Local Area Network), WAN (Wide Area Network), the Internet, etc., and mediates communication between the image forming apparatus 102 and the PC 101. If the image forming apparatus 102 and the PC 101 have wireless communication functions, the network 103 may be wireless.

[0013] In the following description, the image forming apparatus 102 will be referred to as the server, and information processing devices such as the PC 101 or mobile terminals connected to the server will be referred to as clients. In this embodiment, an example using the PC 101 as the information processing device will be described, but the information processing device is not limited to the PC 101 and only needs to have a web browser function. The information processing device may be a mobile terminal such as a smartphone or tablet, or it may be other image forming apparatus other than the image forming apparatus 102 that functions as a server. The image forming apparatus 102 may be a printer with printing capabilities, or an MFP (Multi-function Peripheral) with printing capabilities as well as copying, transmission, faxing, and storage capabilities.

[0014] First, an overview of remote connection in the information processing system 100 will be explained. Initially, the image forming apparatus 102 starts the web server 330 and the VNC server software, which will be described later. It then waits for connections from clients. Next, the PC 101 displays the remote UI screen of the image forming apparatus 102 using the web browser 502, which will be described later. The remote UI is software that allows users to check the status of the image forming apparatus 102, operate jobs, and perform various settings. When the web browser 502 connects to the image forming apparatus 102 by specifying a predetermined URL, the image forming apparatus 102 processes the data based on the specified URL and sends web page data written in HTML or a scripting language to the client. URL is an abbreviation for Uniform Resource Locator, and HTML is an abbreviation for Hyper Text Markup Language. The client's web browser 502 constructs and displays the screen based on the received web page data. In the following explanation, web page data will also be referred to as screen data. Communication between the image forming apparatus 102 and the client's web browser 502 is performed according to the HTTP (Hyper Text Transfer Protocol) protocol.

[0015] The remote UI screen displays link information to multiple functions of the image forming apparatus. When a user on PC101 specifies link information, the web browser 502 receives the corresponding screen data from the image forming apparatus 102 based on the specified link information and displays it on the display device of PC101. For example, if link information to remote operation is specified from multiple menus, the web browser 502 receives and displays the screen for the remote operation function from the image forming apparatus 102. Hereinafter, the screen for the remote operation function will be referred to as the remote operation screen. When the user operates the start button displayed on the remote operation screen, the remote operation start process is initiated between the client web browser 502 and the image forming apparatus 102. In the start process, the client web browser 502 communicates with the VNC server 320 via the image forming apparatus 102's web server 330. Communication with the VNC server 320 begins with a handshake process. In the handshake process, the server and client exchange information and establish a VNC connection. Once a VNC connection is established, the server and client can exchange information such as the displayed screen and screen operations, allowing the client to remotely control the server.

[0016] The image forming apparatus 102 performs user authentication (login authentication) to limit the users who can access the image forming apparatus 102. By performing user authentication, the functions that can be used can be changed for each user. For example, functions that can be used can be assigned based on the user's permission information, such as prohibiting copying or allowing full functionality. In addition, personalization functions become available by performing user authentication. Personalization functions include functions such as displaying the personal address book of the logged-in user.

[0017] Figure 2 is a hardware configuration diagram of the image forming apparatus 102. As shown in Figure 2, the image forming apparatus 102 includes a controller unit 201, an operation unit 207, a scanner 210, and a printer 211. The controller unit 201 includes a CPU 202, RAM 203, ROM 204, HDD (Hard Disk Drive) 205, an operation unit I / F 206, a network I / F 208, and a device I / F 209. The operation unit 207 is connected to the controller unit 201 via the operation unit I / F 206. The scanner 210 and printer 211 are connected to the controller unit 201 via the device I / F 209. Note that the configuration of the image forming apparatus 102 shown in Figure 2 is an example, and it may include other components not listed above.

[0018] In the controller unit 201 within the image forming apparatus 102, the CPU 202 is a computing unit that controls the entire system. The CPU 202 comprehensively controls each device connected to the system bus 212. Although Figure 2 shows only one CPU, there may be multiple CPUs. Furthermore, a processor such as a GPU (Graphics Processing Unit) may also be included.

[0019] ROM204 is a non-volatile memory that stores image data, other data, and various programs necessary for the CPU202 to operate.

[0020] RAM203 is volatile memory and is used as the main memory and temporary storage area for the CPU202, such as the work area. RAM203 also stores programs and data, including the operating system, system software, and application software.

[0021] The CPU 202 controls the various parts of the image forming apparatus 102 by loading programs stored in ROM 204 or the like into RAM 203 and executing them. Note that the programs required for the CPU 202 to operate are not limited to those stored in ROM 204; programs pre-stored in HDD 205 or the like may also be used.

[0022] The HDD205 stores the operating system, system software, application software, image data, configuration data, etc. The HDD205 only needs to be able to store data; other storage devices such as SSDs (Solid State Drives), memory cards, or eMMCs (embedded Multi Media Cards) may be used instead.

[0023] The control unit I / F 206 is the interface unit with the control unit 207 and outputs display information output from the CPU 202 to the control unit 207. It also receives information input by the user from the control unit 207 and inputs it to the CPU 201. The control unit 207 is equipped with a touch panel and hard keys for receiving user input. The touch panel is a flat panel and notifies the control unit I / F 206 of the coordinate information of the position where the user touches. The hard keys are physically pressable keys such as numeric keys, start keys, and stop keys, and notify the control unit I / F 206 when a key is pressed. The control unit 207 is also equipped with a display that displays characters and images. The display receives the GUI (Graphical User Interface) screen generated by the CPU 202 from the control unit I / F 206 and displays it. By installing the touch panel integrated on top of the display, it becomes possible for the user to control the screen displayed on the display as if they were directly operating it.

[0024] Network I / F 208 connects to network 103 and performs information input and output with PC 101, which is connected on the same network.

[0025] Device I / F 209 connects the image input / output devices, scanner 210 and printer 211, to the controller unit 201, and performs image data input and output.

[0026] Figure 3 is a software configuration diagram of the image forming apparatus 102. The system software 301 is stored in one of the memory units, RAM 203, ROM 204, or HDD 205, and executed by the CPU 202.

[0027] The job control processing unit 302 controls each software module, including software modules not shown, and controls jobs performed by the image forming apparatus 102, such as copying, printing, and scanning.

[0028] The scan processing unit 303 controls the scanner 210 via the device I / F 209 based on instructions from the job control processing unit 302, and performs the process of reading the document placed on the document tray of the scanner 210.

[0029] The print processing unit 304 controls the printer 211 via the device interface 209 based on instructions from the job control processing unit 302 to perform the printing process for the specified image.

[0030] The operation input analysis unit 305 analyzes the information supplied from the operation unit 207 via the operation unit I / F 206 and notifies the event processing unit 307 of events corresponding to user operations. The information analyzed by the operation input analysis unit 305 includes instances where the user touches the touch panel or presses a hard key.

[0031] The event analysis unit 306 receives and analyzes various events occurring in the image forming apparatus 102 from the job control processing unit 302 and other sources, in addition to user input, and notifies the event processing unit 307 of the events corresponding to those events. The various events analyzed by the event analysis unit 306 are events other than those caused by user operations, such as the completion of a job or running out of printing paper.

[0032] The event processing unit 307 receives events from the operation input analysis unit 305, the event analysis unit 306, the operation information receiving unit 322, or the remote UI control unit 331, etc. Depending on the event, the event processing unit 307 notifies the job control processing unit 302, the screen generation unit 308, or the soft keyboard control unit 310 of the event and controls the respective parts of the image forming apparatus 102. The event processing unit 307 also notifies the authentication unit 311 of the authentication information received from the operation input analysis unit 305 and receives the result of the user authentication process from the authentication unit 311. In addition, the event processing unit 307 notifies the connection processing unit 321 of the disconnection of the VNC connection.

[0033] The screen generation unit 308 performs the process of generating screen data to be displayed on the display on the RAM 203.

[0034] The screen display unit 309 reads the screen data generated by the screen generation unit 308 from the RAM 203 and displays it on the display of the operation unit 207 via the operation unit I / F 206.

[0035] The soft keyboard control unit 310 receives notifications from the event processing unit 307 and controls the display of the soft keyboard screen. The soft keyboard control unit 310 also holds information indicating whether or not the soft keyboard is displayed on the display.

[0036] The authentication unit 311 receives authentication information from the event processing unit 307, performs user authentication processing including login or logout processing, and returns the result.

[0037] The VNC server 320 is a software module that provides functions for remotely controlling the operation unit 207 from the PC 101, and includes a connection processing unit 321, an operation information receiving unit 322, and a screen transmission unit 323. The VNC server 320 transmits and receives information related to remote operation with the data conversion unit 340.

[0038] The connection processing unit 321 responds to the remote operation start instruction from the data conversion unit 340 by performing a start process to initiate the VNC connection and establishing the VNC connection. The connection processing unit 321 also receives notification of VNC disconnection from the event processing unit 307 and performs termination processing, including disconnecting the VNC connection.

[0039] The operation information receiving unit 322 receives VNC operation information from the data conversion unit 340 and notifies the event processing unit 307 of events corresponding to user operations. The events notified by the operation information receiving unit 322 are equivalent to the events notified by the operation input analysis unit 305 and are events for the user to operate the image forming apparatus 102.

[0040] The screen transmission unit 323 reads the screen data generated by the screen generation unit 308 from the RAM 203 and transmits it to the data conversion unit 340.

[0041] Furthermore, if a VNC client (not shown) is installed on PC101, the various components of the VNC server 320 can also communicate with the VNC client via LAN103.

[0042] The web server 330 receives HTTP requests sent by the web browser 502 via the network 103. It then analyzes the received data and requests processing from the remote UI control unit 331 or the data conversion unit 340 depending on the analysis results. Specifically, if the data analysis results in the received data relating to VNC, it requests processing from the data conversion unit 340; otherwise, it requests processing from the remote UI control unit 331. Alternatively, the web server 330 receives a response from the remote UI control unit 331 or the data conversion unit 340. The web server 330 then sends the received response to the web browser 502 via the network 103.

[0043] The remote UI control unit 331 receives a processing request from the web server 330. It then notifies the event processing unit 307 according to the received request. Furthermore, upon receiving notification from the event processing unit 307, it responds to the web server 330.

[0044] The data conversion unit 340 sends and receives VNC-related information to and from the web server 330 and the VNC server 320, respectively.

[0045] Figure 4 is a hardware configuration diagram of PC101. As shown in Figure 4, PC101 has a CPU 401, ROM 402, RAM 403, HDD 404, input device 405, display device 406, and network I / F 407. Note that the configuration of PC101 shown in Figure 4 is just one example, and it may include other components not listed above.

[0046] Within PC101, CPU401 is the arithmetic unit that controls the entire system. CPU401 comprehensively controls each device connected to the system bus 408. Although Figure 4 shows only one CPU, it may have multiple CPUs. It may also include a processor such as a GPU (Graphics Processing Unit).

[0047] ROM402 is a non-volatile memory that stores image data, other data, and various programs necessary for the CPU401 to operate.

[0048] RAM403 is a volatile memory used as the main memory and temporary storage area for the CPU401, such as the work area. RAM403 also stores programs and data, including the operating system, system software, and application software.

[0049] The CPU 401 controls various parts of the PC 101 by loading programs stored in ROM 402, etc., into RAM 403 and executing them. Note that the programs required for the CPU 401 to operate are not limited to those stored in ROM 402; programs pre-stored in HDD 404, etc., may also be used.

[0050] The HDD404 stores the operating system, system software, application software, photos, electronic documents, and configuration data. However, any storage device that can store data can be used, such as an SSD, SD memory card, or eMMC.

[0051] The input device 405 is a keyboard or pointing device that receives instructions from the user to the PC 101 and inputs the received information to the CPU 201.

[0052] The display device 406 includes a display and a display control circuit, and displays display information output from the CPU 401. Remote UI screens and remote operation screens obtained from the web server 330 via the web browser 502 are displayed on the display device 406. These screens display the operating status of the image forming apparatus 102 and information output by each program running on the PC 101.

[0053] The network interface 407 connects to network 103 and performs information input and output with other PCs and image forming apparatus 102 connected on the same network.

[0054] The above elements are arranged on the system bus 408. The CPU 401 executes processing based on programs corresponding to the operating system and applications stored in the HDD 404, thereby realizing the functions (or processing) of the operating system and applications.

[0055] Figure 5 is a software configuration diagram of PC101. The system software 501 is stored in one of the memory units, RAM403, ROM402, or HDD404, and executed by CPU401.

[0056] As shown in Figure 5, the system software 501 includes a web browser 502. The web browser 502 requests the web server 330 to retrieve web content. The web browser 502 also provides functions such as parsing the HTML of the web content retrieved from the web server 330 and displaying it on the display device 406, and parsing and executing JAVASCRIPT®.

[0057] The system software 501 may also include a VNC client (not shown). The VNC client communicates with various parts of the VNC server 320 via the network 103.

[0058] (Functional configuration of information processing systems) Next, with reference to Figure 6, the functional configuration of the information processing system 100 according to the embodiment of this disclosure will be described. As shown in Figure 6, the PC 101 has an acquisition unit 601, a decision unit 602, a notification unit 603, and a remote operation request transmission unit 604. In addition, the storage unit 404 of the PC 101 stores message data 610, which includes multiple message candidates.

[0059] The acquisition unit 601 acquires first information indicating the status of remote operation between the PC 101 and the image forming apparatus 102, and second information indicating the usage status of the image forming apparatus 102. The first information is determined based on whether or not a remote operation start instruction has been sent from the PC 101 to the image forming apparatus 102, or the response from the image forming apparatus 102 to the remote operation start instruction (connection permitted / connection denied). In this embodiment, the remote operation start instruction and its response are transmitted and received between the remote operation request transmission unit 604 and the VNC server 320 via the web browser 502 of the PC 101.

[0060] The status of the remote operation indicated by the first information is specifically one of the following: "Before Start," "Preparing for Execution," or "In Execution." "Before Start" is the state before the remote operation start command is issued on the PC 101, or after a response indicating that remote operation is not permitted has been received from the image forming apparatus 102. "Preparing for Execution" is the state after the user has issued the remote operation start command on the PC 101, the remote operation request transmission unit 604 has transmitted the remote operation start command to the image forming apparatus 102, and before a response indicating whether remote operation is permitted or not has been received from the image forming apparatus 102. "In Execution" is the state in which remote operation has become possible to execute as a result of the remote operation start command and response.

[0061] The second piece of information indicates the usage status of the image forming apparatus 102, specifically whether or not there are users logged into the image forming apparatus 102 other than the PC 101 user. If there are users logged into the image forming apparatus 102, i.e., if there are users using the image forming apparatus 102, the second piece of information is "Logged-in users present". If there are no users logged into the image forming apparatus 102, i.e., if there are no users using the image forming apparatus 102, the second piece of information is "No logged-in users". The second piece of information can be obtained from the authentication unit 311 of the image forming apparatus 102 via the web server 330 and the web browser 502.

[0062] The decision unit 602 determines notification information to be notified to the user of PC 101 based on the first and second information acquired by the acquisition unit 601. The decision unit 602 determines a message (notification information) corresponding to the first and second information acquired by the acquisition unit 601 from a plurality of message data 610 pre-stored in the HDD 404. Figure 7 shows an example of a candidate message data 610. As shown in Figure 7, the message data 610 stores, in association with the first information, the remote operation status 701, the second information, the device login status 702, and a message 703 representing the progress of the remote connection. The device in this case is the image forming apparatus 102 connected to PC 101. The message data 610 will be described later.

[0063] The notification unit 603 notifies the user of PC 101 of the notification information (message) determined by the decision unit 602. The notification is made by the web browser 502 displaying the notification information (message) on the display device 406 of PC 101 so that it can be viewed. The decision unit 602 and the notification unit 603 first determine and display the message to be displayed before the operation to start remote operation is performed. For example, when the "Remote Operation" menu is selected on the remote UI screen, the decision unit 602 determines the message to be displayed. The notification unit 603 displays the determined message on the remote UI screen. At this stage, since the operation to start remote operation has not been performed, the status of the remote operation (first information) is "before start". If the device login status at this stage is "no logged-in user", the message "You can start remote operation." is displayed. If the device login status is "logged-in user", the message "You can start remote operation. Another user is currently using the device. When you start remote operation, we will wait for permission from the user using the device." is displayed.

[0064] Furthermore, the notification unit 603 updates the notification information when at least one of the first information and the second information is changed. Specifically, the notification unit 603 updates the notification information after sending a remote operation start instruction to the image forming apparatus 102. This is because the status of the remote operation changes from "before start" to "preparing for execution". The notification unit 603 also updates the notification information after the remote operation becomes ready to be executed. This is because the status of the remote operation changes from "preparing for execution" to "executing". The state in which the remote operation is ready to be executed is the state in which a VNC connection has been established. The notification unit 603 also updates the notification information after a remote operation end instruction has been sent to the image forming apparatus 102, or when the remote connection is disconnected. This is because the status of the remote operation changes from "executing" or "preparing for execution" to "before start".

[0065] If the status of the remote operation indicated by the first information is "before start" or "preparing for execution," and the usage status of the image forming apparatus 102 indicated by the second information is "logged-in user," the notification information includes information that the apparatus is waiting for permission to use from the user logged in to the image forming apparatus.

[0066] If the status of the remote operation indicated by the first piece of information is "running" and the usage status of the image forming apparatus 102 indicated by the second piece of information is "logged-in user," the notification information will include information indicating that the operation screen will be shared with the user logged into the image forming apparatus 102.

[0067] The remote operation request transmission unit 604 transmits a remote operation start command to the image forming apparatus 102 when the user of PC 101 performs the remote operation start command operation. When the image forming apparatus 102 receives the remote operation start command from PC 101, the connection processing unit 321 executes the remote operation connection process.

[0068] The connection processing unit 321 determines whether or not remote operation is possible during the remote operation connection process and returns a response to the PC 101 indicating whether or not remote operation is permitted. The response indicating whether or not remote operation is permitted is either a response granting permission to connect or a response denying the connection. When the connection processing unit 321 receives a remote operation start instruction, if there is a user using (logged into) the image forming apparatus 102, it asks the user of the image forming apparatus 102 whether or not to allow the remote user to use the image forming apparatus 102. Depending on the answer to this inquiry, the connection processing unit 321 determines whether to allow or deny remote operation by the remote user (the user of PC 101). If the answer to the inquiry indicates permission for remote operation or permission for remote operation while the user continues to operate, the connection processing unit 321 responds with "Connection permitted". If the answer to the inquiry indicates denial of remote operation, it responds with "Connection denied". The connection processing unit 321 (VNC server 320) sends the determined response to the PC 101.

[0069] The authentication unit 311 decides whether to allow the user using the image forming apparatus 102 to remain logged in or to log them out, based on the response to the inquiry regarding whether to permit the use of the image forming apparatus 102. If the response to the inquiry indicates permission for remote operation, the authentication unit 311 logs the user out. This means that the right to operate the image forming apparatus 102 is transferred to the remote user. If the response to the inquiry indicates denial of remote operation, the authentication unit 311 maintains the user's logged-in status. This means that the right to operate the image forming apparatus 102 is not transferred to the remote user, and operation continues. The user's logged-in status is also maintained if permission for remote operation by the remote user is indicated while the user continues operation. In this case, it means that the user logged into the image forming apparatus 102 and the remote user share the same operation screen. When the login status changes, the authentication unit 311 sends the login status to the PC 101 via the web server 330.

[0070] When the connection processing unit 321 receives a remote operation start instruction, if there are no users currently using (logged into) the image forming apparatus 102, it does not perform the above inquiry and sends a response to the PC 101 indicating that remote operation is permitted.

[0071] When the remote operation request transmission unit 604 transmits a remote operation start instruction, the acquisition unit 601 of PC101 changes the remote operation status from "starting" to "preparing for execution" and stores it in RAM 403. Furthermore, when the acquisition unit 601 receives a response from the image forming apparatus 102 to the remote operation start instruction, it determines the remote operation status (first information). If it receives a response granting permission for remote operation, the acquisition unit 601 changes the remote operation status from "preparing for execution" to "executing" and stores it in RAM 403. If it receives a response rejecting remote operation, the acquisition unit 601 returns the remote operation status from "preparing for execution" to "before start" and stores it in RAM 403. Before the remote operation request transmission unit 604 transmits a remote operation start instruction, the acquisition unit 601 stores the remote operation status as "before start" in RAM 403.

[0072] Furthermore, when the image forming apparatus 102 performs login or logout processing in the authentication unit 311 and a change occurs in the login status, it sends information indicating the login status to the PC 101. The acquisition unit 601 of the PC 101 acquires the login status transmitted from the image forming apparatus 102 and stores it in the RAM 403.

[0073] Figure 7 shows an example of a candidate message data 610. As shown in Figure 7, the message data 610 stores the remote operation status 701, the login status 702 of the device (image forming apparatus 102), and a message 703 representing the progress of the remote connection, in association with each other.

[0074] For example, as shown in line 704, when the remote operation status 701 is "Before Start" and the device login status 702 is "No Logged-in Users", the message 703 associated with this is "You can start remote operation." When the device login status 702 is "No Logged-in Users", it indicates that there are no users logged into the image forming apparatus 102. When the device login status 702 is "Logged-in Users", it indicates that there is one or more users logged into the image forming apparatus 102. In line 705, when the remote operation status 701 is "Before Start" and the device login status 702 is "Logged-in Users", the message 703 associated with this is "You can start remote operation. Another user is currently using the device. When you start remote operation, we will wait for permission from the user currently using the device."

[0075] Line 706 is associated with the message 703 "Preparing to execute remote operation." when the remote operation status 701 is "Preparing to execute" and the device login status 702 is "No logged-in user". Line 707 is associated with the message 703 "Preparing to execute remote operation. Waiting for permission from the user using the device." when the remote operation status 701 is "Preparing to execute" and the device login status 702 is "Logged-in user exists".

[0076] Line 708 is associated with the message 703 "Remote operation in progress." when the remote operation status 701 is "running" and the device login status 702 is "no logged-in user". Line 709 is associated with the message 703 "Remote operation in progress. Screen sharing is being performed with the user using the device." when the remote operation status 701 is "running" and the device login status 702 is "logged-in user".

[0077] Note that these messages are examples only, and the wording of the messages is not limited to the example in Figure 7. The remote operation status is shown in three types: "Before Start," "Preparing to Execute," and "Executing," but other states may also be included.

[0078] (Remote operation) Next, the process of remotely operating the control unit 207 of the image forming apparatus 102 using the web browser 502 of PC 101 will be explained in detail with reference to a flowchart and screen examples.

[0079] First, the remote operation settings for the image forming apparatus 102 will be explained. These settings are performed to enable remote operation of the image forming apparatus 102 from the PC 101. Typically, this setting operation is performed by a user with administrator privileges for the image forming apparatus 102. When the user powers on the image forming apparatus 102, the controller unit 201 of the image forming apparatus 102 starts the VNC server 320 and the web server 330. During the VNC server startup process, the connection processing unit 321 refers to the VNC-related setting information 800 stored on its HDD to determine whether the remote operation setting 801 is enabled (ON). The VNC-related setting information 800 may be set in advance by the administrator of the image forming apparatus 102, or it may be set at any time. During the setting of the VNC-related setting information 800, the CPU 202 of the image forming apparatus 102 displays the remote operation setting screen 900 on the operation unit 207 and performs the VNC-related setting process.

[0080] Figure 8 shows an example of VNC configuration information 800. VNC configuration information 800 stores the setting value of remote operation setting 801 and the value of password 802. The setting value of remote operation setting 801 is either enabled (ON) or disabled (OFF). The password is a password that enables configuration operations on the remote operation settings screen 900 and is set by the administrator. In the example in Figure 8, the setting value of remote operation setting 801 is "ON" and the value of password 802 is not set. If the setting value of remote operation setting 801 is enabled (ON), the connection processing unit 321 opens the port for VNC connection and waits for a connection from the client. If the setting value of remote operation setting 801 is disabled (OFF), the connection processing unit 321 may wait for remote operation setting 801 to be enabled (ON) before opening the port for VNC connection and waiting for a connection from the client.

[0081] Figure 9 shows an example of the remote operation settings screen 900. The remote operation settings screen 900 is provided with an ON button 901 and an OFF button 902. The ON button 901 is selected when VNC is used in the image forming apparatus 102. When the ON button 901 is selected, the administrator may set a password. The OFF button 902 is selected when VNC is not used in the image forming apparatus 102. The password field 903 is an input field for the administrator password used in the VNC authentication process. The OK button 904 is used to confirm the settings on the remote operation settings screen 900. The Cancel button 905 is used to cancel the setting changes on the remote operation settings screen. When the user changes the settings of the ON button 901, OFF button 902, and password field 903, and the OK button 904 is operated, the CPU 202 of the PC 101 saves the changed settings to the HDD 205. When the user operates the Cancel button 905, the CPU 202 closes the remote operation settings screen 900 and returns to the original screen.

[0082] The settings configured on the remote operation settings screen 900 are managed as VNC-related settings information 800 and stored on the HDD 205. The VNC-related settings information 800 can be accessed at any time. Similarly, for the web server 330, it may be possible to configure whether or not to use the web server using a web server settings screen (not shown), and save the settings to the HDD 205. Then, during the startup process of the web server 330, the settings may be referenced, and if the settings are valid (ON), the web server port may be opened and it may wait for connections from clients.

[0083] (Remote UI screen display processing) Referring to Figure 10, the display process of the remote UI screen in the information processing system 100 will be described. In this embodiment, the remote UI screen is displayed using the web browser 502 of the PC 101. Figure 10(a) is a flowchart illustrating the process executed by the CPU 401 of the PC 101. Figure 10(b) is a flowchart illustrating the process executed by the CPU 202 of the image forming apparatus 102. These flowcharts start when the user enters the URL (e.g., IP address) of the image forming apparatus 102 into the web browser 502.

[0084] In S1001, the user operates the web browser 502 using the display device 406 and input device 405 of the PC 101. When the web browser 502 detects that the user has entered the URL of the image forming apparatus 102, it sends an HTTP request to the web server 330 using the entered URL of the image forming apparatus 102.

[0085] The web server 330 of the image forming apparatus 102 is waiting for access from the web browser 502. In S1002, the web server 330 determines whether or not there has been access from the web browser 502. Specifically, in S1002, if the web server 330 receives an HTTP request from the web browser 502, it determines that there has been access from the web browser 502. In S1002, if the web server 330 determines that there has been access from the web browser 502, it proceeds to S1003; if it determines that there has been no access from the web browser 502, it repeats the process in S1002.

[0086] In S1003, the web server 330 requests processing from the remote UI control unit 331. This request is for the remote UI screen. The remote UI control unit 331 sends the remote UI screen data to the web server 330. The web server 330 sends the remote UI screen data to the web browser 502, and the process in the flowchart of Figure 10(b) is completed.

[0087] In S1004, the web browser 502 receives remote UI screen data sent from the web server 330 as a response to the HTTP request sent in S1001.

[0088] In S1005, the web browser 502 analyzes the received screen data of the remote UI, displays the remote UI screen on the display device 406, and terminates the process in the flowchart of Figure 10(a).

[0089] Figure 11 shows an example of a remote UI screen displayed by the web browser 502 of PC 101. In the example in Figure 11, the remote UI top screen 1100 (top page) is shown. As described above, the remote UI top screen 1100 is a screen that the web browser 502 receives in S1004 from the screen data transmitted by the image forming apparatus 102 in S1003 and displays in S1005.

[0090] The remote UI top screen 1100 includes a device status display area 1101, a consumables information display area 1102, and a menu area 1103. The device status display area 1101 displays information indicating the status of the scanner 210 and printer 211 of the image forming apparatus 102. The consumables information display area 1102 displays information about consumables, such as the type and remaining amount of paper and ink. The menu area 1103 displays link buttons to pages for each process. The link buttons in the menu area 1103 include a link button 1104 to remote operation. The link button 1104 to remote operation is a button for calling the remote operation function. The link button 1104 to remote operation may be displayed only when the remote operation setting 801 is enabled (ON). Also, if the image forming apparatus 102 uses user authentication (login authentication), the CPU 202 of the image forming apparatus 102 may decide whether or not to display the link button 1104 to remote operation based on the user information of the logged-in user. For example, the link button 1104 to remote operation may be displayed only if the user who displays the remote UI top screen 1100 has administrator access privileges. Alternatively, user permissions for using the remote operation function may be set in advance, and the link button 1104 to remote operation may be displayed according to the permissions of the user who displays the remote UI top screen 1100. In this case, for example, the permissions of users who can perform remote operations can be set as administrators only, all users can, etc.

[0091] (Remote operation screen display process) Referring to Figure 12, the display process of the remote operation screen in the information processing system 100 will be described. Figure 12(a) is a flowchart illustrating the process executed by the CPU 401 of the PC 101. Figure 12(b) is a flowchart illustrating the process executed by the CPU 202 of the image forming apparatus 102. These flowcharts are initiated when a user gives an instruction to display the remote operation screen. In this embodiment, the instruction to display the remote operation screen is input by the user operating, for example, the remote operation link button 1104 displayed on the top screen 1100 of the remote UI.

[0092] In S1201, when the web browser 502 of PC 101 detects that the remote operation link button 1104 has been pressed, it sends an HTTP request to the web server 330 of the image forming apparatus 102 requesting the remote operation screen.

[0093] The web server 330 of the image forming apparatus 102 is waiting for a request for the remote operation screen from the web browser 502 of the PC 101. In S1202, the web server 330 determines whether a request for the remote operation screen has been received from the web browser 502. Specifically, in S1202, if the web server 330 receives an HTTP request for the remote operation screen from the web browser 502, it determines that a request for the remote operation screen has been received from the web browser 502. In S1202, if the web server 330 determines that a request for the remote operation screen has been received from the web browser 502, it proceeds to S1203. If it determines that there is no request for the remote operation screen from the web browser 502, it repeats the process in S1202.

[0094] In S1203, the web server 330 requests processing from the remote UI control unit 331. Since the request is for a remote operation screen, the remote UI control unit 331 sends screen data for the remote operation screen to the web server 330. The web server 330 sends the received remote operation screen data to the web browser 502.

[0095] Next, in S1204, the web server 330 receives information indicating the login status to the image forming apparatus 102 from the authentication unit 311 via the event processing unit 307 and sends it to the remote UI control unit 331. The information indicating the login status includes information such as whether or not there are logged-in users. However, the information indicating the login status is not limited to this and may also include, for example, the username, the date and time of login, and the duration of login. The web server 330 sends the acquired login status information to the web browser 502 and terminates the processing of the flowchart in Figure 12(b).

[0096] In S1205, the web browser 502 of PC101 receives screen data of the remote operation screen sent from the web server 330 in S1203 as a response to the HTTP request.

[0097] At this stage, since only screen data from the remote operation screen has been received, in S1206 the web browser 502 saves the remote operation status to RAM 203 as "before start".

[0098] In S1207, the web browser 502 receives the login status sent from the web server 330 in S1204.

[0099] In S1208, the web browser 502 determines a message indicating the progress of the remote connection based on the status of the remote operation and the login status of the image forming apparatus 102. In S1209, the web browser 502 displays the remote operation screen on the display device 406 of the PC 101 based on the screen data received in S1205, and then terminates the process shown in the flowchart of Figure 12(a).

[0100] Note that while the flowchart in Figure 12 illustrates an example where screen data for the remote operation screen and information indicating the login status are sent separately in S1203 and S1204, it is not limited to this example. The screen data for the remote operation screen and information indicating the login status may be sent together at once.

[0101] Figures 13 and 14 are examples of remote operation screens displayed in S1209. If the login status information received in S1207 is "No logged-in user", the first remote operation screen 1300 shown in Figure 13 is displayed on the display device 406 of PC101. If the login status information received in S1207 is "Logged-in user exists", the second remote operation screen 1400 shown in Figure 14 is displayed.

[0102] The first remote operation screen 1300 and the second remote operation screen 1400 are provided with a menu area 1103, an operation area 1301, and an operation screen display area 1302. The menu area 1103 is the same as the menu area of ​​the remote UI screen shown in Figure 11. The operation area 1301 displays a start button 1311, an end button 1312, and a message 1313. The start button 1311 is a button to start the remote operation. The end button 1312 is a button to end the remote operation. As described above, the status of the remote operation is "before start" and has ended, so the color of the end button 1312 is changed to indicate that the end button 1312 is selected. The message 1313 is a message indicating the progress of the remote connection. The message 1313 that is determined in S1208 is displayed. The operation screen display area 1302 displays the same screen as the screen displayed on the operation unit 207 of the image forming apparatus 102 when the status of the remote operation becomes "running". The first remote operation screen 1300 and the second remote operation screen 1400 are screens that are displayed when the remote operation status is "before start," so nothing is displayed in the operation screen display area 1302.

[0103] The message 1313 displayed on the first remote operation screen 1300 shown in Figure 13 shows the message from line 704 in Figure 7. This is because the remote operation status 701 is "Before Start" and the device login status 702 is "No logged-in users". Specifically, the message "You can start remote operation." is displayed. A remote user who sees this message can understand that they can start remote operation without any particular restrictions.

[0104] The message 1401 displayed on the second remote operation screen 1400 shown in Figure 14 is the message shown on line 705 in Figure 7. This is because the remote operation status 701 is "Before Start" and the device login status 702 is "Logged-in user exists". Specifically, the message displayed is: "You can start remote operation. Another user is currently using the device. When you start remote operation, we will wait for permission from the user currently using the device." A remote user who sees this message can recognize that another user is currently logged into and using the image forming apparatus 102. They can also recognize that if they start remote operation, they may not be able to start immediately because they will have to wait for permission from the other user.

[0105] (Remote operation initiation process) Next, the process for starting remote operations will be explained with reference to Figures 15 to 21.

[0106] Figure 15 is a flowchart illustrating the process of starting a remote operation performed in the information processing system 100. Figure 15(a) is a flowchart illustrating the process performed by the CPU 401 of the PC 101. Figure 15(b) is a flowchart illustrating the process performed by the CPU 202 of the image forming apparatus 102. These flowcharts are started when a user inputs an instruction to start a remote operation. In this embodiment, the instruction to start a remote operation is input by the user operating the start button 1311 displayed on the first remote operation screen 1300 or the second remote operation screen 1400.

[0107] In S1501, when the web browser 502 of PC101 detects that the start button 1311 has been pressed, it sends an HTTP request to the web server 330 to initiate remote operation. In S1502, the web browser 502 updates the remote operation status to "Preparing to execute" and saves it to RAM 203. In S1503, the web browser 502 determines a message indicating the progress of the remote connection based on the remote operation status 701 and the device login status 702, and updates the message to be displayed.

[0108] Figures 16 and 17 are examples of remote operation screens displayed in S1503. Compared to the remote operation screens shown in Figures 13 and 14, the messages have been updated. If the login status information received in S1207 during the remote operation screen display process is "No logged-in user", the third remote operation screen 1600 shown in Figure 16 is displayed on the display device 406 of the PC 101. If the login status information received in S1207 is "Logged-in user exists", the fourth remote operation screen 1700 shown in Figure 17 is displayed. The third remote operation screen 1600 and the fourth remote operation screen 1700 are provided with a menu area 1103, an operation area 1301, and an operation screen display area 1302. However, since the start process is being executed, the start button 1311 is displayed in a color that indicates the selected state, and the end button 1312 is displayed in a color that indicates the unselected state. Message 1601 is a message that indicates the progress of the remote connection. The message 1601 that is determined in S1503 is displayed. The operation screen display area 1302 displays the same screen as the one displayed on the operation unit 207. The third remote operation screen 1600 and the fourth remote operation screen 1700 are screens that are displayed when the remote operation status is "Preparing to execute," and the remote operation start process has not yet been completed. Therefore, nothing is displayed in the operation screen display area 1302.

[0109] As message 1601 displayed on the third remote operation screen 1600 shown in Figure 16, the message from line 706 of the message data 610 shown in Figure 7 is displayed. This is because the remote operation status 701 is "Preparing to execute" and the device login status 702 is "No logged-in user". Specifically, "Preparing to execute remote operation." is displayed. A remote user who sees this message can recognize that the remote operation status is "Preparing to execute" without any particular restrictions.

[0110] As message 1701 displayed on the fourth remote operation screen 1700 shown in Figure 17, the message from line 707 of the message data 610 shown in Figure 7 is displayed. This is because the remote operation status 701 is "Preparing to execute" and the device login status 702 is "Logged-in user exists". Specifically, the message "Preparing to execute remote operation. Waiting for permission from the user using the device." is displayed. A remote user who sees this message can recognize that another user is currently logged into and using the image forming apparatus 102. They can also recognize that the remote operation process they started may not be able to start immediately because it is waiting for permission from the other user. Specifically, in S1504, which will be described later, a remote connection confirmation screen 1800 is displayed on the operation unit 207 of the image forming apparatus 102, and it is waiting for a response from the user who is logged into and using the image forming apparatus 102. The remote connection confirmation screen 1800 will be described later.

[0111] Returning to the explanation of Figure 15, the web server 330 of the image forming apparatus 102 is waiting for a request to start remote operation from the web browser 502 of the PC 101. In S1504, the web server 330 determines whether it has received a remote operation start instruction from the web browser 502. If the web server 330 receives an HTTP request from the web browser 502 indicating a remote operation start instruction, the web server 330 determines that there has been a request to start remote operation from the web browser 502 (it has received a remote operation start instruction). When the web server 330 receives the HTTP request sent from the web browser 502 in S1501, the web server 330 requests processing from the remote UI control unit 331. The remote UI control unit 331 obtains information from the authentication unit 311 via the event processing unit 307 indicating the login status of the user logged into the image forming apparatus 102.

[0112] In S1505, the remote UI control unit 331 determines whether or not there is a user logged into the image forming apparatus 102 based on information indicating the login status. If the remote UI control unit 331 determines that there is no user logged in as a result of the determination in S1505, it proceeds to S1512; otherwise, it proceeds to S1506.

[0113] In S1506, the remote UI control unit 331 notifies the event processing unit 307 that there is a user logged in. The remote UI control unit 331 also displays a remote connection confirmation screen 1800 on the operation unit 207 and asks the logged-in user whether they want to connect remotely. The display of the remote connection confirmation screen 1800 is executed by the CPU 202 of the image forming apparatus 102.

[0114] Figure 18 shows the remote connection confirmation screen 1800 displayed on the operation unit 207 of the image forming apparatus 102. Checkbox 1801 is selected to allow the logged-in user to continue operations and to reject remote connections by remote users. A remote user is the user who sent the remote operation start command from PC 101. Checkbox 1802 is selected to interrupt the logged-in user's operations and log them out. Checkbox 1803 is selected to allow remote connections while allowing the logged-in user to continue operations. Checkbox 1803 is selected, for example, when the logged-in user has trouble operating the screen and wants to share the screen with a remote user from a remote location to receive support. Only one of checkboxes 1801 to 1803 can be selected. The OK button 1804 is operated to confirm the settings of checkboxes 1801 to 1803. When the logged-in user selects one of the checkboxes 1801 to 1803 and the OK button 1804 is selected, the remote UI control unit 331 closes the confirmation screen 1800 and executes the process selected by the logged-in user.

[0115] In S1507, the remote UI control unit 331 receives information (answers) of the checkboxes selected by the logged-in user.

[0116] In S1508, the remote UI control unit 331 determines whether the checkbox selected by the logged-in user is checkbox 1801 (continue user operation and reject remote connection). If the remote UI control unit 331 determines that checkbox 1801 is selected as a result of the determination in S1508, it proceeds to S1509. If it determines that the checkbox selected by the logged-in user is not checkbox 1801, it proceeds to S1510.

[0117] In S1509, the remote UI control unit 331 sends a connection rejection message to the web browser 502 via the web server 330, and terminates the processing of the flowchart in Figure 15(b).

[0118] If, as a result of the judgment in S1508, checkbox 1801 is not selected, the remote UI control unit 331 proceeds to S1510.

[0119] In S1510, the remote UI control unit 331 determines whether the checkbox selected by the logged-in user is checkbox 1802 (interrupt the user's operation and log out). If the determination in S1510 is that checkbox 1802 is selected, the process proceeds to S1511. If the determination is that the checkbox selected by the logged-in user is not checkbox 1802, the process proceeds to S1512.

[0120] In S1511, the remote UI control unit 331 logs out the user currently logged into the image forming apparatus 102. This process is performed when the remote UI control unit 331 notifies the authentication unit 311, via the event processing unit 307, to execute the logout process. After the logout process is completed, the process proceeds to S1512.

[0121] If, as a result of the judgment in S1510, checkbox 1802 is not selected, in this embodiment, it indicates that checkbox 1803 (Allow remote connection while the user continues operations) is selected. In that case, proceed to S1512.

[0122] In S1512, the remote UI control unit 331 sends information indicating "connection permitted" to the web browser 502 via the web server 330.

[0123] Next, in S1513, the connection processing unit 321 performs the VNC connection process.

[0124] In S1514, the web browser 502 receives information indicating whether a connection is possible, which is sent from the web server 330 in S1509 or S1512, as a response to the HTTP request.

[0125] In S1515, the web browser 502 determines whether the information received in S1514 indicates that the connection is permitted or denied. If the web browser 502 determines that it has received information indicating that the connection is permitted, it proceeds to S1516. If it determines that the information received in S1514 indicates that the connection is denied, it proceeds to S1522. In S1516, the web browser 502 performs the VNC connection process.

[0126] During the VNC connection process, the PC 101 and the image forming apparatus 102 exchange information regarding the connection. Specifically, in S1513 and S1516, the connection processing unit 321 of the VNC server 320 performs the VNC connection process via the browser 502 of the PC 101, the web server 330 of the image forming apparatus 102, and the data conversion unit 340. Since the VNC connection process is a well-known process based on the RFB (Remote Frame Buffer) protocol (RFC6143), a detailed explanation is omitted.

[0127] Once the VNC connection process is complete and the VNC connection is established, the image forming apparatus 102 transmits screen data of the screen displayed on the operation unit 207 to the PC 101. This process can be initiated by the PC 101 requesting screen data after the VNC connection is established, or the image forming apparatus 102 can proactively transmit screen data to the PC 101. In this embodiment, an example in which the image forming apparatus 102 proactively transmits screen data to the PC 101 will be described.

[0128] In S1517, the screen transmission unit 323 reads the screen data generated by the screen generation unit 308 from the RAM 203 and sends it to the web browser 502 via the data conversion unit 340 and the web server 330. After that, the process shown in the flowchart in Figure 15(b) is terminated.

[0129] Meanwhile, on PC101, after the VNC connection is established, the web browser 502 in S1518 updates the remote operation status to "running" and saves it to RAM203.

[0130] In S1519, the web browser 502 determines a message indicating the progress of the remote connection based on the remote operation status 701 and the device login status 702, and updates the message to be displayed on the remote operation screen.

[0131] In S1520, the web browser 502 receives screen data from the operation unit 207 transmitted from the image forming apparatus 102 in S1517.

[0132] In S1521, the web browser 502 displays the same screen as the one shown on the operation unit 207 based on the received screen data, and terminates the process of the flowchart in Figure 15(a).

[0133] Figures 19 and 20 show examples of remote operation screens displayed in S1521. Compared to the remote operation screens shown in Figures 16 and 17, the message indicating the progress of the remote connection has been updated. If the login status information received in S1207 is "No logged-in user", the fifth remote operation screen 1900 shown in Figure 19 is displayed on the display device 406 of the PC 101. If the login status information received in S1207 is "Logged-in user exists", the sixth remote operation screen 2000 shown in Figure 20 is displayed. The fifth remote operation screen 1900 and the sixth remote operation screen 2000 are provided with a menu area 1103, an operation area 1301, and an operation screen display area 1302. Message 1901 is a message indicating the progress of the remote connection. Message 1901 displays the message indicating the progress of the remote connection determined in S1519. The operation screen display area 1302 displays the same screen as the screen displayed on the operation unit 207 of the image forming apparatus 102. The screen is displayed based on the screen data received by S1520. For example, if the screen displayed on the operation unit 207 of the image forming apparatus 102 is the home screen, the same home screen 1902 is also displayed in the operation screen display area 1302. The message 1901 displayed on the fifth remote operation screen 1900 is the message from line 708 of the message data 610 shown in Figure 7. This is because the remote operation status 701 is "running" and the device login status 702 is "no logged-in user". Specifically, "Remote operation in progress." is displayed. A remote user who sees this message can recognize that they can perform remote operation without any particular restrictions.

[0134] The message 2001 displayed on the sixth remote operation screen 2000 shown in Figure 20 is line 709 of the message data 610 shown in Figure 7. This is because the remote operation status 701 is "running" and the device login status 702 is "logged-in user". Specifically, the message "Remote operation is in progress. Screen sharing is being done with the user using the device." is displayed. The remote user who sees this message can recognize that the current remote operation status is "running". Also, because another user is logged into and using the image forming apparatus 102, the same screen 2002 that is being displayed is also displayed on the operation panel 207 of the image forming apparatus 102. The remote user can recognize that the same screen 2002 is visible to the other user who is logged into and using the image forming apparatus 102.

[0135] Returning to the explanation of Figure 15, if information indicating connection rejection is transmitted from the image forming apparatus 102 in S1515, proceed to S1522. In S1522, the web browser 502 displays error screen 2100.

[0136] Figure 21 shows an example of an error screen 2100 displayed by the web browser 502 in S1522. As shown in Figure 21, a pop-up screen 2101 containing an error message 2102 is displayed on the remote operation screen. When the OK button 2103 is pressed, the pop-up screen 2101 is closed. In the example in Figure 21, the error message 2102 displays something like, "The user using the device did not authorize the use of remote operation. Please wait a while before starting the operation." The screen data for this pop-up screen 2101 is stored in the HDD 404 beforehand.

[0137] Next, in S1523, the web browser 502 updates the remote operation status to "before start" and saves it to RAM 203. In S1524, the web browser 502 determines a message 703 representing the progress of the remote connection based on the remote operation status 701 and the device login status 702. The web browser 502 updates the message to be displayed on the remote operation screen to the determined message and terminates the processing of the flowchart in Figure 15(a). At this stage, the remote operation status 701 is "before start". If the device login status is "no logged-in user", the web browser 502 updates the displayed screen to the first remote operation screen 1300 shown in Figure 13. If the device login status 702 is "logged-in user", the web browser 502 updates the displayed screen to the second remote operation screen 1400 shown in Figure 14.

[0138] From this point onward, information such as display screens is transmitted and received between the PC 101 and the image forming apparatus 102 based on the RFB protocol. Specifically, the web browser 502 of the PC 101 transmits operation information from the remote operation screen displayed on the display device 406 by the user to the web server 330 of the image forming apparatus 102. Since this operation information is data related to VNC operation on the remote operation screen, the web server 330 requests processing from the operation information receiving unit 322 via the data conversion unit 340. The screen transmission unit 323 also transmits screen information to the web server 330 via the data conversion unit 340. The web server 330 transmits the received screen information to the web browser 502. The web browser 502 displays the received screen information on the display device 406. As a result, the image forming apparatus 102 can be remotely operated from the PC 101 in the same way as if the operation unit 207 of the image forming apparatus 102 were being operated.

[0139] In addition, in S1506, when displaying the remote connection confirmation screen 1800 (Figure 18) on the operation unit 207 of the image forming apparatus 102, the CPU 201 may have the timing unit measure the time. The timing unit shall be provided by the system software 301. If a predetermined time has elapsed since the start of displaying the confirmation screen 1800 and no response (checkbox selection information) has been received from the user, the connection processing unit 321 closes the confirmation screen 1800. Furthermore, the connection processing unit 321 performs the same processing as when any of the checkboxes 1801 to 1803 is selected. For example, similar to when checkbox 1802 is selected, the connection processing unit 321 logs out the user currently logged into the image forming apparatus 102 (equivalent to S1511) and then executes the processing from S1512 onwards. In this way, the waiting time for the user operating the PC 101 can be reduced. In other words, while the remote connection confirmation screen 1800 is displayed on the image forming apparatus 102, from the perspective of the remote user operating the PC 101, it is in a state of waiting for a response to the remote operation start command in S1503 ("preparing for execution"). Therefore, by implementing such a timeout process, it becomes possible to end the waiting time for the remote user and proceed with the remote operation start process.

[0140] (User authentication process) Next, referring to Figures 22 to 25, we will explain in detail the processes that the information processing system 100 executes when a user using the image forming apparatus 102 performs user authentication (login or logout) on the image forming apparatus 102.

[0141] (Login process) Figure 22 is a flowchart illustrating the login process performed in the information processing system 100. Figure 22(a) is a flowchart explaining the process performed by the CPU 401 of the PC 101. Figure 22(b) is a flowchart explaining the process performed by the CPU 202 of the image forming apparatus 102. This flowchart starts when a user operates the operation unit 207 of the image forming apparatus 102 to initiate the login process to log in to the image forming apparatus 102. It is assumed that the image forming apparatus 102 is configured to perform user authentication.

[0142] After the image forming apparatus 102 is started, in S2201, the authentication unit 311 sends an event to the event processing unit 307 to display the login screen, and the login screen is displayed on the operation unit 207.

[0143] Figure 23 shows an example of a login screen 2300. The login screen 2300 includes a message 2301 requesting user input such as "Please enter your username and password and press [Login]," as well as a username input field 2302 and a password input field 2303. A login button 2304 is also provided, which is operated when logging in. When a user of the image forming apparatus 102 enters their username and password into the username input field 2302 and password input field 2303, respectively, and presses the login button 2304, the authentication unit 311 starts the login process.

[0144] In S2202, the event processing unit 307 waits for a login request to be received. When the user presses the login button 2304 on the login screen 2300, the event processing unit 307 determines that a login request has been received. If the event processing unit 307 determines that a login request has been received, it proceeds to S2203; otherwise, it repeats the process in S2202.

[0145] In S2203, the event processing unit 307 receives a login event based on a login request from the operation input analysis unit 305. The event processing unit 307 notifies the authentication unit 311 of the username and password entered by the user. The authentication unit 311 performs user authentication. The user information list, which shows information of users who can use the image forming apparatus 102, is assumed to be registered in advance by the administrator of the image forming apparatus 102 on the HDD 205 of the image forming apparatus 102 or on an authentication server (not shown). The authentication unit 311 checks whether user information matching the received username and password is registered in the user information list. If it is determined that there is a match, user authentication is successful; if it is determined that there is no match, user authentication is failed. The authentication unit 311 returns the result of user authentication to the event processing unit 307.

[0146] In S2204, the event processing unit 307 determines whether user authentication was successful or not. Based on the result of the determination in S2204, if the event processing unit 307 determines that user authentication was successful, it proceeds to S2206; otherwise, it proceeds to S2205.

[0147] In S2205, the screen generation unit 308 generates an error screen (not shown), the screen display unit 309 displays the error screen on the operation unit 207, and the system returns to waiting for login request (S2202). If user authentication is successful as a result of the judgment in S2204, the system proceeds to S2206.

[0148] In S2206, the screen generation unit 308 generates an initial screen, and the screen display unit 309 displays the initial screen on the operation unit 207. If, for example, the home screen is set as the initial screen, the operation unit 207 will display, for example, the home screen 2400 shown in Figure 24.

[0149] In S2207, the event processing unit 307 transmits information indicating the login status of the image forming apparatus 102 to the web browser 502 via the remote UI control unit 331 and the web server 330, and terminates the processing of the flowchart in Figure 22(b). The information indicating the login status includes information on whether or not there are logged-in users. If there are logged-in users, "Logged-in users present" is sent to the PC 101 as information indicating the login status. However, the information indicating the login status is not limited to this, and may also include, for example, the username, the date and time the user logged in, and the duration of the login.

[0150] In S2208, the web browser 502 of PC101 waits for information indicating the login status sent from the web server 330 in S2207. If it determines that it has received the information indicating the login status, it proceeds to S2209. If the web browser 502 determines in S2208 that it has not received the information indicating the login status, it repeats the process in S2208.

[0151] In S2209, the web browser 502 determines a message 703 indicating the progress of the remote connection based on the remote operation status 701 and the device login status 702. The web browser 502 then updates the message to be displayed on the remote UI screen or the remote operation screen and terminates the process shown in the flowchart in Figure 22(a).

[0152] When the first remote operation screen 1300 is displayed, if information indicating a login status of "Logged-in user exists" is received in S2208, the web browser 502 updates the displayed screen to the second remote operation screen 1400. The state in which the first remote operation screen 1300 is displayed is that the remote operation status 701 is "Before Start" and the device login status 702 is "No logged-in user". Also, when the third remote operation screen 1600 is displayed, if information indicating a login status of "Logged-in user exists" is received in S2208, the web browser 502 updates the displayed screen to the fourth remote operation screen 1700. The state in which the third remote operation screen 1600 is displayed is that the remote operation status 701 is "Preparing for Execution" and the device login status 702 is "No logged-in user". Furthermore, if the fifth remote operation screen 1900 is displayed and S2208 receives information indicating the login status, such as "Logged-in user exists," the web browser 502 updates the displayed screen to the sixth remote operation screen 2000. The state in which the fifth remote operation screen 1900 is displayed is that the remote operation status 701 is "running" and the device login status 702 is "No logged-in user."

[0153] (Logout process) Figure 25 is a flowchart illustrating the logout process performed in the information processing system 100. Figure 25(a) is a flowchart explaining the process performed by the CPU 401 of the PC 101. Figure 25(b) is a flowchart explaining the process performed by the CPU 202 of the image forming apparatus 102. This flowchart is initiated when the user operates the control panel 207 of the image forming apparatus 102 to start the logout process to log out of the image forming apparatus 102.

[0154] In S2501, the event processing unit 307 waits for a logout request and determines whether there is a logout request. For example, if the user presses the logout button 2401 displayed on the home screen 2400, the event processing unit 307 determines that it has received a logout event based on a logout request from the operation input analysis unit 305. In S2501, if the event processing unit 307 determines that there is a logout request, it proceeds to S2502; if it determines that there is no logout request, it repeats the process in S2501.

[0155] In S2502, the event processing unit 307 notifies the authentication unit 311 of the logout event, and the authentication unit 311 performs the logout process.

[0156] In S2503, the screen generation unit 308 generates the login screen 2300, and the screen display unit 309 displays the login screen 2300 on the operation unit 207.

[0157] In S2504, the event processing unit 307 sends information indicating the login status to the web browser 502 via the remote UI control unit 331 and the web server 330, and terminates the processing of the flowchart in Figure 25(b). The information indicating the login status includes information on whether or not there are logged-in users. In Figure 24, since the user has logged out, the information "No logged-in users" is sent to the PC 101 as information indicating the login status. However, the authentication information is not limited to this, and may also include information such as the date and time the user logged out and the time they were logged in.

[0158] In S2505, the web browser 502 on PC101 waits to receive information indicating the login status sent from the web server 330 in S2504. If information indicating the login status is received, the process proceeds to S2506.

[0159] In S2506, the web browser 502 of PC101 determines a message 703 indicating the progress of the remote connection based on the remote operation status 701 and the device login status 702. The web browser 502 then updates the message to be displayed on the remote UI screen or the remote operation screen and terminates the process of the flowchart in Figure 25(a).

[0160] When the second remote operation screen 1400 is displayed, if information indicating the login status of "No logged-in user" is received in S2506, the web browser 502 updates the displayed screen to the first remote operation screen 1300. The state in which the second remote operation screen 1400 is displayed is that the remote operation status 701 is "Before Start" and the device login status 702 is "Logged-in user exists". Also, when the fourth remote operation screen 1700 is displayed, if information indicating the login status of "No logged-in user" is received in S2506, the web browser 502 updates the displayed screen to the third remote operation screen 1600. The state in which the fourth remote operation screen 1700 is displayed is that the remote operation status 701 is "Preparing for Execution" and the device login status 702 is "Logged-in user exists". Furthermore, if the sixth remote operation screen 2000 is displayed and S2506 receives information indicating the login status as "no logged-in user", the web browser 502 updates the displayed screen to the fifth remote operation screen 1900. The state in which the sixth remote operation screen 2000 is displayed is that the remote operation status 701 is "running" and the device login status 702 is "logged-in user exists".

[0161] In this way, when a user using the image forming apparatus 102 performs user authentication (login or logout) on the image forming apparatus 102 and the login status changes, the web browser 502 changes and displays a message indicating the progress of the remote connection. This allows the user who sees the message to check the progress of the remote connection, which changes depending on whether or not the user is logged in.

[0162] (Termination process for remote operations) Next, with reference to Figure 26, the termination process for remote operations will be explained.

[0163] Figure 26 is a flowchart illustrating the process for terminating a remote operation performed in the information processing system 100. Figure 26(a) is a flowchart illustrating the process performed by the CPU 401 of the PC 101. Figure 26(b) is a flowchart illustrating the process performed by the CPU 202 of the image forming apparatus 102. This flowchart is initiated when a user inputs an instruction to terminate the remote operation. In this embodiment, the instruction to terminate the remote operation is input by the user operating the terminate button 1312 displayed on the first to sixth remote operation screens.

[0164] In S2601, when the web browser 502 of PC101 detects that the exit button 1312 has been pressed, it sends an HTTP request to the web server 330 indicating an instruction to terminate the remote operation.

[0165] In S2602, the web browser 502 updates the remote operation status to "before start" and saves it to RAM 203.

[0166] In S2603, the web browser 502 determines a message indicating the progress of the remote connection based on the remote operation status 701 and the device login status 702, updates the message displayed on the screen, and terminates the process of the flowchart in Figure 26(a).

[0167] If the remote operation status 701 is "Before Start" and a command to end the remote operation is entered, and the device login status 702 is "No logged-in user", the first remote operation screen 1300 shown in Figure 13 will be displayed. If the remote operation status 701 is "Before Start" and a command to end the remote operation is entered, and the device login status 702 is "Logged-in user exists", the second remote operation screen 1400 shown in Figure 14 will be displayed. The content of the messages indicating the progress of the remote connection displayed on each screen is as described above.

[0168] If the remote operation status 701 is "Preparing to execute" and a command to terminate the remote operation is entered, and the device login status 702 is "No logged-in user", the third remote operation screen 1600 shown in Figure 16 will be displayed. If the remote operation status 701 is "Preparing to execute" and a command to terminate the remote operation is entered, and the device login status 702 is "Logged-in user exists", the fourth remote operation screen 1700 shown in Figure 17 will be displayed. The content of the messages indicating the progress of the remote connection displayed on each screen is as described above.

[0169] If the remote operation status 701 is "running" and a command to terminate the remote operation is entered, and the device login status 702 is "no logged-in user", the fifth remote operation screen 1900 shown in Figure 19 will be displayed. If the remote operation status 701 is "running" and a command to terminate the remote operation is entered, and the device login status 702 is "logged-in user", the sixth remote operation screen 2000 shown in Figure 20 will be displayed. The content of the messages indicating the progress of the remote connection displayed on each screen is as described above.

[0170] In S2604, the web server 330 waits for an HTTP request from the web browser 502 and determines whether it has received an instruction to terminate the remote operation. If the web server 330 determines in S2604 that it has received an instruction to terminate the remote operation, it proceeds to S2605; if it determines that it has not received an instruction to terminate the remote operation, it repeats the process in S2604. If the web server 330 receives an HTTP request from the web browser 502 indicating an instruction to terminate the remote operation, it requests the remote UI control unit 331 to disconnect the VNC connection.

[0171] In S2605, the remote UI control unit 331 notifies the connection processing unit 321 of the VNC connection disconnection via the event processing unit 307. The connection processing unit 321 disconnects the VNC connection and terminates the processing of the flowchart in Figure 26(b).

[0172] As described above, in the information processing system 100, the image forming apparatus 102 is configured to be remotely operated from the PC 101. The PC 101 displays messages on the display device 406 as notification information indicating the progress of the remote connection, corresponding to the status of the remote operation, such as "before starting," "preparing for execution," and "executing." Therefore, the remote user can check the progress of the remote connection on the screen of the PC 101. In this embodiment, the PC 101 also displays messages 703 on the display device 406 of the PC 101, corresponding to the status of the remote operation 701 and the usage status (login status 702) of the image forming apparatus 102. This allows the remote user to check not only the status of the remote operation but also whether other users are using the image forming apparatus 102 when checking the progress of the remote connection on the screen of the PC 101. Conventionally, remote users operated the image forming apparatus 102 without knowing the usage status of the image forming apparatus 102 by other users. In particular, it was not possible to predict how long it would take from sending a remote operation start command until the remote connection was established. In this embodiment, different messages are displayed on the PC 101 depending on the usage status, such as whether the image forming apparatus 102 is being used by another user, thus making the usage status of the image forming apparatus 102 located remotely visible. Therefore, remote users can grasp the progress of the remote connection in more detail. This improves the operability of remote operation.

[0173] In the above-described embodiment, all three states of remote operation—"before start," "preparing for execution," and "executing"—were notified. However, this disclosure is not limited to this example, and it is also possible to notify only one or two of these states.

[0174] <Example 1> In the embodiment described above, the web browser 502 displayed a message indicating the progress of the remote connection on a screen that instructed the user to start / end the remote operation. The screen that instructed the user to start / end the remote operation is, for example, the first remote operation screen 1300 or the second remote operation screen 1400. However, this disclosure is not limited to this procedure. A message indicating the progress of the remote connection may be displayed on the top screen of the remote UI.

[0175] Figure 27 shows an example of the second top screen 2700 of the remote UI displayed in Modification 1. The differences between the second top screen 2700 shown in Figure 27 and the top screen 1100 shown in Figure 11 will be explained.

[0176] The second top screen 2700 displays the information shown on the top screen 1100 in Figure 11, as well as a message 2701 indicating the progress of the remote connection. In the example in Figure 27, the content of message 2701 is "Remote operation can be started." This indicates that the remote operation status 701 is "Before start" and the device login status 702 is "No logged-in user." By displaying such a message indicating the progress of the remote connection on the remote UI screen, the display of the first remote operation screen 1300 or the second remote operation screen 1400 can be skipped.

[0177] Specifically, in the remote UI screen display process (Figure 10), before S1003, the web server 330 of the image forming apparatus 102 obtains the login status of the image forming apparatus 102 from the authentication unit 311. Then, in S1003, the web server 330 sends the obtained login status information (second information) to the web browser 502 along with the screen data of the remote UI top screen. In S1004, the web browser 502 receives the login status information of the image forming apparatus 102 and the screen data of the remote UI top screen. In S1005, the web server 330 determines the content of the message to be displayed on the top screen based on the received login status information and the remote operation status, configures the second top screen 2700 based on the screen data, and displays it on the display device 406.

[0178] Alternatively, before S1003, the web server 330 of the image forming apparatus 102 obtains the login status of the image forming apparatus 102 from the authentication unit 311 and determines a message corresponding to the login status and the status of remote operation. Then, in S1003, the web server 330 sends the screen data of the second top screen 2700, including the determined message, to the web browser 502. The content of the message can be determined from the message data 610 based on the status of remote operation and the login status of the device, as in the embodiment described above.

[0179] If the remote operation status 701 is "Before Start" and the device login status 702 is "Logged-in User," the second top screen 2700 displays the message shown in line 705 of the message data 610 in Figure 7: "You can start remote operation. Another user is currently using the device. When you start remote operation, we will wait for permission from the user using the device." When the remote user selects the remote operation link button 1104 on the second top screen 2700, the remote operation start process shown in Figure 15 is executed.

[0180] Figure 28 shows an example of the remote operation screen 2800 in Modification 1. In Modification 1, the remote operation screen 2800 shows the case where the remote operation status is "running" and the device login status is "no logged-in user". In Modification 1, the remote operation screen 2800 may be displayed by updating the display of the second top screen 2700, or it may be displayed in a different tab or a different window. In Modification 1, the operation screen display area 2802 of the remote operation screen 2800 displays the same operation screen 2803 as the screen displayed on the operation unit 207 of the image forming apparatus 102, and it can be operated from the PC 101. In addition, area 2804 displays hard key buttons 2805 that have the same functions as the hard keys provided on the operation unit 207 of the image forming apparatus 102, and they can be operated from the PC 101. The hard key buttons 2805 include, for example, a button to transition to the home screen, a button to transition to the counter / device information screen, a button to stop the operation running on the image forming apparatus 102, and a button to switch the image forming apparatus 102 to power saving mode / return to power saving mode.

[0181] The remote operation screen 2800 shown in Figure 28 is displayed when the remote operation status is "running". Therefore, the start button 1311 and end button 1312 that were displayed on each remote operation screen shown in the first embodiment are not displayed. To end the remote operation, simply use the "×" button 2806 to close the screen.

[0182] Message 2801 is a message indicating the progress of the remote connection. The web browser determines which message 703 to display from the message data 610 based on the remote operation status 701 and the device login status 702, and displays it on the remote operation screen 2800. If the remote operation status 701 or the device login status 702 changes, the web browser 502 updates the content of the displayed message according to the changed remote operation status 701 and device login status 702.

[0183] As explained above, according to Modification 1, the PC 101 displays a message indicating the progress of the remote connection on the top screen of the remote UI. Then, it skips the screen that instructs the start of remote operation (first remote operation screen 1300 or second remote operation screen 1400) and starts remote operation directly from the remote UI screen. This simplifies the operation. The remote user can see the usage status of the image forming apparatus 102 by other users on the top screen of the remote UI. Thus, the operability of remote operation is improved.

[0184] <Modification 2> In the first embodiment, a message indicating the progress of the remote connection, corresponding to the status of the remote operation and the usage status of the image forming apparatus 102, was displayed on the web browser 502 of the PC 101. However, the system is not limited to this example, and the message indicating the progress of the remote connection may only show the status of the remote operation, regardless of the usage status of the image forming apparatus 102. Specifically, if the status of the remote operation is "before start," the web browser 502 displays only the message "You can start the remote operation," shown in line 704 of the message data shown in Figure 7. Alternatively, the web browser 502 may only display a status indicating that the status of the remote operation is "before start." Similarly, if the status of the remote operation is "preparing to execute," the web browser 502 may display "Preparing to execute the remote operation," shown in line 706 of the message data shown in Figure 7, regardless of the usage status of the image forming apparatus 102. Alternatively, the web browser 502 may only display a status indicating "preparing to execute." Furthermore, if the remote operation status is "running", the web browser 502 may display "Remote operation in progress." as shown in line 708 of the message data in Figure 7, regardless of the usage status of the image forming apparatus 102. Alternatively, the web browser 502 may simply display a status indicating "running".

[0185] In the information processing system 100 shown in Modification 2, the PC 101 can access the VNC server 320 of the image forming apparatus 102 via the web browser 502 and perform remote operations. The web browser 502 then acquires the status of the remote operation between the PC 101 and the image forming apparatus 102. The status of the remote operation is determined, as described in the first embodiment, based on whether or not a remote operation start instruction has been sent in the remote operation request transmission unit 604, and information indicating whether or not a connection is possible, obtained from the connection processing unit 321 of the VNC server 320. Depending on the status of the remote operation, the web browser 502 determines the appropriate message from the message data 610 and displays it on the remote operation screen or remote UI screen as a message indicating the progress of the remote connection. Alternatively, it displays a status according to the status of the remote operation as the progress of the remote connection.

[0186] As mentioned above, the status of a remote operation is "before starting," "preparing to run," or "running." The web browser 502 also notifies the user of a message or displays the status before the remote user initiates the remote operation. Furthermore, if the status of the remote operation changes, the web browser 502 updates the content of the notification message or the status display according to the changed status of the remote operation.

[0187] This allows PC101 to perform remote operation of the image forming apparatus 102 via the web browser 502, and the remote user can at least check the status of the remote operation on the PC101 screen. Therefore, the progress of the remote connection via the web browser 502 can be displayed in a simplified manner.

[0188] <Variation 3> The message indicating the progress of the remote connection may be designed to handle both cases where the image forming apparatus 102 is logged in ("logged-in user") and cases where it is not logged in ("no logged-in user"), regardless of the actual usage status of the image forming apparatus 102. That is, when the remote operation status is "before start" or "preparing to run," the web browser 502 may display a general message, for example, "Depending on the device usage, the remote connection may take some time," regardless of the usage status of the image forming apparatus 102. This message may be displayed on the remote operation screen where the start of the remote operation can be instructed, as shown in the first embodiment, or on the top screen of the remote UI, as shown in Modification 1. The web browser 502 may also display a status indicating the remote operation status, such as "before start" or "preparing to run," along with the general message mentioned above. Furthermore, when the remote operation status is "running," the web browser 502 may display a general message such as "If there is a user using the device, the screen will be shared." The web browser 502 may also display the status "running" along with the general message.

[0189] In the case of Modification 3, regardless of the login status of the image forming apparatus 102, a display can be made on the PC 101 indicating that there is a user using the image forming apparatus 102. This allows the remote user to infer that another user is using the image forming apparatus 102 if there is a long waiting time for the connection after the remote user enters the command to start remote operation.

[0190] In the first embodiment and its modified form, the PC 101 used a web browser 502 to obtain information indicating the login status of the image forming apparatus 102, thereby determining the message to be displayed on the PC 101 side. The reason for using the web browser 502 to obtain information is that the RFB protocol used by the VNC server 320 does not support the transmission and reception of the login status of the image forming apparatus 102 or the transmission and reception of strings to be displayed on the PC 101 side. However, it is also possible to display and update messages indicating the progress of the remote connection as described above without using the web browser 502. In that case, the CPU 401 of the PC 101 executes processing according to an application program that performs various processing related to remote operation as described in the first embodiment. The application program shall have functions equivalent to the acquisition unit 601, determination unit 602, notification unit 603, and remote operation request transmission unit 604 (Figure 6) shown in Figure 6. In this case, the acquisition unit 601 uses an extended protocol, which is an extension of the RFB protocol, to obtain information indicating the login status of the image forming apparatus 102 and information indicating whether remote operation is possible. Alternatively, the acquisition unit 601 uses a network communication method other than the RFB protocol (for example, MIB; Management Information Base) to acquire information indicating the login status of the image forming apparatus 102 and information indicating whether remote operation is possible.

[0191] <Second Embodiment> In the first embodiment, the content of the message indicating the progress of the remote connection displayed on the PC 101 was determined on the PC 101 side, but it may also be determined on the image forming apparatus 102 side. In the second embodiment, a case in which the image forming apparatus 102A of the second embodiment determines the content of the message and displays it on the PC 101A of the second embodiment will be described. In the second embodiment, the information processing system, PC, and image forming apparatus will be denoted by the reference numerals 100A, 101A, and 102A, respectively. The system configuration of the information processing system 100A, and the hardware and software configurations of the PC 101A and image forming apparatus 102A are the same as in the first embodiment shown in Figures 1 to 5, so their description will be omitted. In the following description, parts that are the same as in the first embodiment will be denoted by the same reference numerals.

[0192] Figure 29 is a diagram showing the functional configuration of the information processing system 100A according to the second embodiment. As shown in Figure 29, in the information processing system 100A according to the second embodiment, the image forming apparatus 102A of the second embodiment has an acquisition unit 2901, a determination unit 2902, an authentication unit 311, and a VNC server 320, etc. In addition, multiple message data 610 are stored in the HDD 404 of the image forming apparatus 102A. The message data 610 includes candidate messages that represent the progress of the remote connection, similar to the first embodiment. The PC 101A of the second embodiment has a notification unit 603 and a remote operation request transmission unit 604, etc. In the second embodiment as in the first embodiment, the image forming apparatus 102A functions as a web server 330 and a VNC server 320. The image forming apparatus 102A also sends and receives screen data, the status of the image forming apparatus 102A, setting data, usage status (login status), and other data with the web browser 502 of the PC 101A. Furthermore, data related to remote operation is transmitted and received by connecting to the VNC server 320 of the image forming apparatus 102A via the web browser 502 of PC 101A and the web server 330 of the image forming apparatus 102A. The RFB protocol (RFC6143) is used for communication between the VNC server and client. The HTTP protocol is used for communication between the web browser 502 and the web server 330.

[0193] The acquisition unit 2901 acquires first information indicating the status of remote operation between PC 101A and image forming apparatus 102A, and second information indicating the usage status of image forming apparatus 102A. The first information is determined based on whether or not a remote operation start instruction has been sent from PC 101A to image forming apparatus 102A, or the response of image forming apparatus 102A to the remote operation start instruction (connection permitted / connection denied). In this embodiment, the remote operation start instruction and its response are transmitted and received between the remote operation request transmission unit 604 and the VNC server 320 via the web browser 502 of PC 101A.

[0194] The state of remote operation indicated by the first information is the same as in the first embodiment, and is one of the following: "Before Start," "Preparing for Execution," or "In Execution." "Before Start" is the state before the PC 101A is operated to give a remote operation start command, or the state after the image forming apparatus 102A has decided to respond by rejecting the remote connection. "Preparing for Execution" is the state after the PC 101A has been operated by the user to give a remote operation start command and has received the remote operation start command from the remote operation request transmission unit 604, but before the image forming apparatus 102A has decided to respond by indicating whether or not to allow the remote operation. "In Execution" is the state in which, as a result of the remote operation start command and response, a remote connection has been established and the remote operation can be executed.

[0195] The second piece of information indicates the usage status of the image forming apparatus 102A, which is the same as in the first embodiment. It indicates whether or not there is a user logged into the image forming apparatus 102A. If there is a user logged into the image forming apparatus 102A, that is, if there is a user using the image forming apparatus 102A, the second piece of information is "Logged-in user present". If there is no user logged into the image forming apparatus 102A, that is, if there is no user using the image forming apparatus 102A, the second piece of information is "No logged-in user". The second piece of information is transmitted from the authentication unit 311 of the image forming apparatus 102A to the acquisition unit 2901.

[0196] The decision unit 2902 determines notification information to be notified to the user of PC101A based on the first and second information acquired by the acquisition unit 2901. The decision unit 2902 determines a message (notification information) corresponding to the first and second information acquired by the acquisition unit 2901 from message data 610 pre-stored in the HDD 205. The message data 610 is the same as in the first embodiment, and as shown in Figure 7, the first information, the remote operation status 701, the second information, the device login status 702, and a message 703 representing the progress of the remote connection are stored in association. The decision unit 2902 sends the determined notification information to the web browser 502 of PC101A.

[0197] When the notification unit 603 receives notification information (message) determined by the decision unit 2902 of the image forming apparatus 102A, it notifies the user of PC 101A. The notification is made by the web browser 502 displaying the notification information (message) on the display device 406 of PC 101A so that it can be viewed. The timing of the determination of notification information by the decision unit 2902 and the notification by the notification unit 603 is the same as in the first embodiment. That is, the decision unit 2902 first determines and displays a message to be displayed according to the login status before receiving a remote operation start instruction. For example, when the "Remote Operation" menu is selected on the remote UI screen, the decision unit 2902 determines the message to be displayed and sends it to PC 101A along with the screen data of the remote operation screen. The notification unit 603 displays the remote operation screen based on the received screen data and also displays the message determined by the decision unit 2902. At this stage, since a remote operation start instruction has not been input, the remote operation status (first information) is "before start". If the device's login status at this stage is "No logged-in user," the message "You can start remote operations." will be displayed. If the device's login status is "Logged-in user," the message "You can start remote operations. Another user is currently using the device. When you start remote operations, we will wait for permission from the user currently using the device." will be displayed.

[0198] Furthermore, the decision unit 2902 updates the notification information and sends it to PC101A when at least one of the first information and the second information is changed. Specifically, when the VNC server 320 receives a remote operation start instruction from PC101A, the decision unit 2902 updates the notification information based on the information indicating whether remote operation is possible and the login status, and sends it to PC101A. The notification unit 603 receives the updated notification information and displays it on the remote operation screen. Also, when the remote operation becomes ready to be executed, the decision unit 2902 updates the notification information based on the login status and sends it to PC101A. The notification unit 603 receives the updated notification information and displays it on the remote operation screen.

[0199] The information indicating whether remote operation is permitted is determined by querying the user of the image forming apparatus 102A, as in the first embodiment. If the user's response to the query (confirmation screen 1800) indicates permission for remote connection or permission for remote connection while the user continues to operate, the connection processing unit 321 responds with "Connection permitted". If the response to the query indicates denial of remote connection, it responds with "Connection denied". The connection processing unit 321 (VNC server 320) transmits the determined response to the acquisition unit 2901.

[0200] Furthermore, depending on the response to the inquiry, the authentication unit 311 either allows the user to remain logged in or logs them out. Similar to the first embodiment, if the response to the inquiry indicates permission for remote connection, the authentication unit 311 logs the user out. If the response to the inquiry indicates denial of remote connection, or permission for remote connection by a remote user while the user continues to operate, the authentication unit 311 maintains the user's logged-in status. If the login status (logged-in user present / logged-in user absent) changes, the authentication unit 311 transmits the login status to the acquisition unit 2901.

[0201] If no user is currently using (logged in to) the image forming apparatus 102A when the remote operation start command is received, the connection processing unit 321 will not perform the above-mentioned inquiry. The connection processing unit 321 will send a response to the acquisition unit 2901 and PC 101A granting permission for remote operation.

[0202] When the acquisition unit 2901 receives a remote operation start instruction from the remote operation request transmission unit 604, it changes the remote operation status to "Preparing for execution" and stores it in the RAM 203. Furthermore, when the acquisition unit 2901 receives a response to the remote operation start instruction from the connection processing unit 321, it determines the remote operation status (first information). If it receives a response granting permission for the remote operation, the acquisition unit 2901 changes the remote operation status to "Executing" and stores it in the RAM 203. If it receives a response rejecting the remote operation, the acquisition unit 2901 returns the remote operation status to "Before Start" and stores it in the RAM 203. Before receiving a remote operation start instruction from the remote operation request transmission unit 604, the acquisition unit 2901 stores the remote operation status as "Before Start" in the RAM 203.

[0203] Furthermore, when the authentication unit 311 of the image forming apparatus 102A performs login or logout processing, or when there is a change in the login status, it sends information indicating the login status to the PC 101A.

[0204] In the second embodiment, the example messages and screens displayed on the PC101A are the same as in the first embodiment. That is, if the remote operation status is "before start" and the usage status of the image forming apparatus 102A (device login status) is "no logged-in user", the first remote operation screen 1300 shown in Figure 13 is displayed on the PC101A. If the remote operation status is "before start" and the usage status of the image forming apparatus 102A (device login status) is "logged-in user", the second remote operation screen 1400 shown in Figure 14 is displayed on the PC101A.

[0205] If the remote operation status is "Preparing for execution" and the usage status of the image forming apparatus 102A (device login status) is "No logged-in users", the third remote operation screen 1600 shown in Figure 16 is displayed on PC101A. If the remote operation status is "Preparing for execution" and the usage status of the image forming apparatus 102A (device login status) is "Logged-in users exist", the fourth remote operation screen 1700 shown in Figure 17 is displayed on PC101A.

[0206] If the remote operation status is "running" and the usage status of the image forming apparatus 102A (device login status) is "no logged-in user", the fifth remote operation screen 1900 shown in Figure 19 is displayed on PC101A. If the remote operation status is "running" and the usage status of the image forming apparatus 102A (device login status) is "logged-in user", the sixth remote operation screen 2000 shown in Figure 20 is displayed on PC101A.

[0207] If the user of the image forming apparatus 102A refuses remote operation, the error screen 2100 shown in Figure 21 will be displayed on the PC 101A.

[0208] As described above, in the information processing system 100A of the second embodiment, the content of a message indicating the progress of the remote connection can be determined on the image forming apparatus 102A side and displayed on the PC 101A. The content of the message is determined according to the remote operation status 701 and the usage status of the image forming apparatus 102A (device login status 702), similar to the first embodiment. As a result, when a remote user checks the progress of the remote connection on the screen of the PC 101A, they can check not only the status of the remote operation but also whether or not another user is using the image forming apparatus 102A. Conventionally, remote users operated the image forming apparatus 102A without knowing whether or not other users were using the image forming apparatus 102A. In this embodiment, different messages are displayed on the PC 101 side depending on the usage status, such as whether or not the image forming apparatus 102 is being used by another user, so the usage status of the image forming apparatus 102 in a remote location is made visible. Therefore, remote users can grasp the progress of the remote connection in more detail. This improves the operability of remote operation.

[0209] Preferred embodiments of the present disclosure have been described above with reference to the attached drawings, but the present disclosure is not limited to such examples. For example, the modifications described above may be applied to the configuration of the second embodiment. Furthermore, the processing procedures shown in each flowchart, the content displayed on each screen, and the wording of the messages indicating the progress of the remote connection are examples for illustrating the present disclosure. It will be obvious to those skilled in the art that various modifications or alterations can be conceived within the scope of the disclosed technical idea, and these will naturally also fall within the technical scope of the present disclosure.

[0210] <Other Embodiments> The embodiments described above can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various storage media, and having the computer (CPU, MPU, etc.) of that system or device read and execute the program. The program may be executed on a single computer or executed in conjunction with multiple computers. Furthermore, it is not necessary to realize all of the above processes in software; some or all of the processes may be realized in hardware such as ASICs. Also, the CPU is not limited to one CPU that performs all the processing; multiple CPUs may perform the processing in coordination as appropriate.

[0211] Furthermore, the functionality of the aforementioned embodiment is not only realized by the execution of program code read by the computer. It also includes cases where the operating system running on the computer performs some or all of the actual processing based on the instructions of that program code, and the functionality of the aforementioned embodiment is realized through that processing.

[0212] The above-described embodiments include the following configurations. (Composition 1) An information processing device that enables remote operation of an image forming apparatus, It includes a notification mechanism for notifying information regarding the progress of remote connections, The notification information is determined based on first information indicating the status of the remote operation between the information processing device and the image forming apparatus, and second information indicating the usage status of the image forming apparatus.

[0213] (Configuration 2) An acquisition means for acquiring the first information and the second information, A determination means that determines the notification information based on the first information and second information acquired by the acquisition means, The information processing device according to configuration 1, characterized by comprising:

[0214] (Composition 3) The system further comprises receiving means for receiving the notification information determined by the image forming apparatus, The information processing apparatus according to configuration 1, characterized in that the notification means notifies the notification information received by the receiving means.

[0215] (Composition 4) The information processing device according to any one of configurations 1 to 3, characterized in that the notification means notifies the notification information before the operation of the remote operation start instruction is performed.

[0216] (Composition 5) The information processing apparatus according to configuration 4, wherein the notification means further updates the notification information when at least one of the first information and the second information is changed.

[0217] (Composition 6) The information processing apparatus according to configuration 5, characterized in that the notification means updates the notification information when the instruction to start the remote operation is transmitted to the image forming apparatus.

[0218] (Composition 7) The information processing device according to configuration 5, characterized in that the notification means updates the notification information when the remote operation becomes possible to perform.

[0219] (Composition 8) The information processing device according to any one of configurations 1 to 7, characterized in that the first information is determined based on whether or not an instruction to start the remote operation is transmitted from the information processing device to the image forming apparatus, or the response from the image forming apparatus to the instruction to start.

[0220] (Composition 9) The state of the remote operation indicated by the first information is, A first state in the information processing device, which is the state before the operation to initiate the remote operation is performed, or the state after receiving a response from the image forming apparatus indicating that the remote operation is not permitted. A second state, which is the state after the information processing device has transmitted the instruction to start the remote operation to the image forming apparatus, but before receiving a response from the image forming apparatus indicating whether or not the remote operation is permitted, and The information processing apparatus according to any one of configurations 1 to 8, characterized in that it includes at least one of the third states in which the remote operation becomes possible as a result of the start instruction and the response.

[0221] (Composition 10) If the state of the remote operation indicated by the first information is either the first state or the second state, and the second information indicates that the image forming apparatus is being used, The information processing device according to configuration 9, characterized in that the notification information further includes information waiting for permission to use from a user logged into the image forming apparatus.

[0222] (Composition 11) If the state of the remote operation indicated by the first information is the third state, and the second information indicates that the image forming apparatus is being used, The information processing device according to configuration 9, characterized in that the notification information further includes information indicating that the operation screen of the image forming apparatus is shared with users logged into the image forming apparatus.

[0223] (Composition 12) The notification means is an information processing device according to any one of configurations 1 to 10, characterized in that it displays the notification information in a viewable format using a web browser.

[0224] (Composition 13) The information processing apparatus according to any one of configurations 1 to 12, characterized in that the RFB (Remote Framebuffer) protocol is used for remote connection between the image forming apparatus and the information processing apparatus.

[0225] (Composition 14) An information processing device that enables remote operation of an image forming apparatus, It includes a notification mechanism for notifying information regarding the progress of remote connections, The information processing device is characterized in that the notification information includes first information indicating the status of the remote operation between the information processing device and the image forming apparatus, and second information indicating the usage status of the image forming apparatus.

[0226] (Composition 15) An information processing system that enables remote operation of an image forming apparatus using an information processing device, The aforementioned information processing device is The system includes a notification means for notifying notification information regarding the status of the remote operation, The aforementioned notification information is, An information processing system characterized by being determined based on first information indicating the status of the remote operation between the information processing device and the image forming apparatus, and second information indicating the usage status of the image forming apparatus.

[0227] (Composition 16) An information processing method performed by an information processing device capable of remote operation of an image forming apparatus, This includes a step of notifying information regarding the progress of the remote connection, The aforementioned notification information is, An information processing method characterized by being determined based on first information indicating the status of the remote operation between the information processing device and the image forming apparatus, and second information indicating the usage status of the image forming apparatus.

[0228] (Composition 17) A program to be executed on a computer of an information processing device capable of remote operation of an image forming apparatus, This includes a step of notifying information regarding the progress of the remote connection, The aforementioned notification information is, A program characterized by being determined based on first information indicating the status of the remote operation between the information processing device and the image forming apparatus, and second information indicating the usage status of the image forming apparatus.

Claims

1. An information processing device that enables remote operation of an image forming apparatus, It includes a notification mechanism for notifying information regarding the progress of remote connections, The notification information is determined based on first information indicating the status of the remote operation between the information processing device and the image forming apparatus, and second information indicating the usage status of the image forming apparatus.

2. An acquisition means for acquiring the first information and the second information, A determination means that determines the notification information based on the first information and the second information acquired by the acquisition means, The information processing apparatus according to claim 1, characterized by comprising:

3. The system further comprises receiving means for receiving the notification information determined by the image forming apparatus, The information processing apparatus according to claim 1, characterized in that the notification means notifies the notification information received by the receiving means.

4. The information processing apparatus according to claim 1, characterized in that the notification means notifies the notification information before the operation of the remote operation start instruction is performed.

5. The information processing apparatus according to claim 4, further characterized in that the notification means updates the notification information when at least one of the first information and the second information is changed.

6. The information processing apparatus according to claim 5, characterized in that the notification means updates the notification information when the instruction to start the remote operation is transmitted to the image forming apparatus.

7. The information processing apparatus according to claim 5, characterized in that the notification means updates the notification information when the remote operation becomes possible to perform.

8. The information processing apparatus according to claim 1, characterized in that the first information is determined based on whether or not an instruction to start the remote operation is transmitted from the information processing apparatus to the image forming apparatus, or the response from the image forming apparatus to the instruction to start.

9. The state of the remote operation indicated by the first information is, A first state in the information processing device, which is the state before the operation to initiate the remote operation is performed, or the state after receiving a response from the image forming apparatus indicating that the remote operation is not permitted. A second state, which is the state after the information processing device has transmitted the instruction to start the remote operation to the image forming apparatus, but before receiving a response from the image forming apparatus indicating whether or not the remote operation is permitted, and The information processing apparatus according to claim 1, characterized in that it includes at least one of the following: a third state in which the remote operation becomes possible as a result of the start instruction and the response.

10. If the state of the remote operation indicated by the first information is either the first state or the second state, and the second information indicates that the image forming apparatus is being used, The information processing apparatus according to claim 9, characterized in that the notification information further includes information waiting for permission to use from a user logged into the image forming apparatus.

11. If the state of the remote operation indicated by the first information is the third state, and the second information indicates that the image forming apparatus is being used, The information processing device according to claim 9, characterized in that the notification information further includes information indicating that the operation screen of the image forming apparatus is shared with users logged into the image forming apparatus.

12. The information processing device according to claim 1, characterized in that the notification means displays the notification information in a viewable format using a web browser.

13. The information processing apparatus according to claim 1, characterized in that the RFB (Remote Framebuffer) protocol is used for the remote connection between the image forming apparatus and the information processing apparatus.

14. An information processing device that enables remote operation of an image forming apparatus, It includes a notification mechanism for notifying information regarding the progress of remote connections, The information processing device is characterized in that the notification information includes first information indicating the status of the remote operation between the information processing device and the image forming apparatus, and second information indicating the usage status of the image forming apparatus.

15. An information processing system that enables remote operation of an image forming apparatus using an information processing device, The aforementioned information processing device is The system includes a notification means for notifying notification information regarding the status of the remote operation, The aforementioned notification information is, An information processing system characterized by being determined based on first information indicating the status of the remote operation between the information processing device and the image forming apparatus, and second information indicating the usage status of the image forming apparatus.

16. An information processing method performed by an information processing device capable of remote operation of an image forming apparatus, This includes a step of notifying information regarding the progress of the remote connection, The aforementioned notification information is, An information processing method characterized by being determined based on first information indicating the status of the remote operation between the information processing device and the image forming apparatus, and second information indicating the usage status of the image forming apparatus.

17. A program to be executed on a computer of an information processing device capable of remote operation of an image forming apparatus, This includes a step of notifying information regarding the progress of the remote connection, The aforementioned notification information is, A program characterized by being determined based on first information indicating the status of the remote operation between the information processing device and the image forming apparatus, and second information indicating the usage status of the image forming apparatus.

Images