Information processing device, method for controlling the information processing device, and program

Abstract

This invention provides an information processing device, a control method for the information processing device, and a program that further enhance security in remote operations. [Solution] In the communication system, the MFP120, which is an information processing device, has an imaging control unit 530 that controls the imaging of images of the area around the information processing device, a detection unit 520 that detects an event to start imaging, and an image display unit 510 that displays the captured image, which is an image captured according to the control of the imaging control means, on a display screen. Furthermore, when the imaging control unit 530 receives a connection request for remote operation or when an event is detected by the detection means during a remote operation connection, it controls the imaging control unit to stop imaging or to stop the remote operation connection.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

Background Art

[0002] An image forming apparatus such as a multifunction peripheral is a type of information processing apparatus. There is known a function called remote operation in which the screen of the operation panel of an image forming apparatus is displayed on a remote client using VNC (Virtual Network Computing) or the like, and operations are performed from a remote location.

[0003] When operating an image forming apparatus remotely, it is necessary to consider security. For example, when a specific screen such as a login screen is displayed on the operation panel of an image forming apparatus, there is known a technique for restricting display in remote operation (Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Further improvement in security in remote operation is required.

Means for Solving the Problems

[0006] An information processing device according to one aspect of the present disclosure is characterized by comprising: an imaging control means for controlling the imaging of an image of the area surrounding the information processing device; a detection means for detecting an event that initiates imaging; a display control means for displaying an image captured in accordance with the control of the imaging control means on a display screen; a connection means for connecting a communication device to a communication device for displaying the image displayed on the display screen and enabling operations from the communication device to the information processing device; and a control means for controlling the imaging control means to cancel imaging or cancel the connection of the remote operation when the connection means receives a request to connect to the remote operation or when the detection means detects the event during the connection of the remote operation. [Effects of the Invention]

[0007] This disclosure suggests that security in remote operations can be improved. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing an example of a communication system configuration. [Figure 2] This is a block diagram showing an example of the hardware configuration of a communication device. [Figure 3] This is a block diagram showing an example of the software configuration of a communication device. [Figure 4] This is a block diagram showing an example of an MFP hardware configuration. [Figure 5] This is a diagram showing an example of an MFP software configuration. [Figure 6] This flowchart shows an example of the processes performed in MFP. [Figure 7] This is an example of a camera image capture error screen. [Figure 8] This figure shows an example of the software configuration of an MFP. [Figure 9]This flowchart shows an example of the process for sending screen images during a remote operation connection in an MFP. [Figure 10] This is a diagram illustrating the transmitted image. [Figure 11] This figure shows an example of the software configuration of an MFP. [Figure 12] This figure shows an example of the software configuration of a communication device. [Figure 13] This flowchart shows an example of the process of sending and receiving screen information during a remote operation connection. [Figure 14] This diagram shows the screen information when the camera is not capturing images. [Figure 15] This flowchart shows an example of the processes performed in MFP. [Modes for carrying out the invention]

[0009] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. Note that the following embodiments are not limiting to the scope of this disclosure, and not all combinations of features described in the following embodiments are essential to the solutions of this disclosure. The same reference numerals are used for identical components.

[0010] <<First Embodiment>> <Overview> In image forming apparatuses, a camera is sometimes installed to read a predetermined code (hereinafter referred to as a 2D code), such as a QR code (registered trademark), and authentication is performed using the read image. In such configurations, the image captured by the camera is displayed on the control panel to confirm whether the subject is within the camera's view.

[0011] Here, when a remote operation connection is made during imaging using a camera, there is a risk that the captured images around the image forming apparatus may be disclosed to the user making the remote operation connection under circumstances unintended by the user who uses the image forming apparatus. As a result, security problems may occur. In the embodiment described below, control is performed so that the remote operation connection and imaging by the camera are not performed simultaneously. An example of preventing the captured images from being disclosed to a client making a remote operation connection (hereinafter referred to as a remote client) and improving security will be described.

[0012] <Configuration> FIG. 1 is a diagram showing an example of the configuration of a communication system according to the present embodiment. The communication system includes a communication device 110 and a multifunction peripheral (MFP) 120. The communication device 110 and the MFP 120 communicate with each other via a network 100. In the example of FIG. 1, the communication device 110 and the MFP 120 connected to the network 100 are each assumed to be one unit, but may be a plurality of units. The communication device 110 is configured to be able to install an application.

[0013] FIG. 2 is a block diagram showing an example of the hardware configuration of the communication device 110. The communication device 110 of the present embodiment assumes an information processing device such as a PC, a smartphone, or a tablet, but may be other devices as long as they can communicate with the network 100. The communication device 110 includes a CPU 201, a ROM 202, a RAM 203, a HDD 204, an input unit 205, a display unit 206, and a communication unit 207.

[0014] The CPU 201 reads out the control program stored in the ROM 202 and executes various processes for controlling the operation of the communication device 110. The ROM 202 stores the control program. The RAM 203 is used as the main memory of the CPU 201 and a temporary storage area such as a work area. The HDD 204 stores an operating system, system software, applications, and various data such as photos and electronic documents. The HDD 204 can use any storage device as long as it can store data. For example, as the HDD 204, other storage devices such as an SSD (Solid State Drive), an SD memory card, or an eMMC (embedded Multi Media Card) may be used.

[0015] The input unit 205 receives an input to the communication device 110. In the present embodiment, the communication device 110 includes the input unit 205, but the input unit 205 may be provided in an external terminal of the communication device 110. The display unit 206 displays various screens. In the present embodiment, the input unit 205 and the display unit 206 are described as separate configurations, but the input unit 205 and the display unit 206 may be integrally configured, such as a touch panel capable of detecting a touch operation of the user. The communication unit 207 executes communication using the network 100. The configuration shown in FIG. 2 is merely an example, and it does not have to have a part of the configuration shown in FIG. 2, and it may have a configuration other than the configuration shown in FIG. 2.

[0016] Figure 3 is a block diagram showing an example of the software configuration of the communication device 110. Figure 3 is a functional block diagram of the software realized by the CPU 201 reading the control program stored in the ROM 202 or HDD 204. The communication device 110 has a remote client 301. The remote client 301 is an application that establishes a remote operation connection with the MFP 120 via the network 100 and performs remote operation of the MFP 120. In this embodiment, the explanation will be based on the assumption that the RFB (Remote FrameBuffer) protocol is used as the remote operation connection, but HTTP communication may also be used. Note that the RFB protocol has functions to reduce the amount of data by dividing the screen into multiple areas, transmitting only the data of the updated area, and compressing the data, but in this embodiment, the explanation of such functions will be omitted.

[0017] While establishing a remote operation connection, the remote client 301 displays the screen image received from the MFP 120 on the display unit 206. The remote client 301 also accepts user operations on the screen image via the input unit 205. Furthermore, the remote client 301 transmits the received operations to the MFP 120 via the network 100. Various other applications are available for the communication device 110, but their explanation is omitted here.

[0018] Figure 4 is a block diagram showing an example of the hardware configuration of the MFP120. In this embodiment, the MFP120 is a multifunction device having a printer 421 and a scanner 422, but it may also be a device having either the printer 421 or the scanner 422. Furthermore, in this embodiment, the MFP120 is used as an example of an information processing device for explanation, but the information processing device in this embodiment does not necessarily have to be an image processing device like the MFP120. Any device that can connect to a camera 423 and allows remote operation connection by a remote client 301 is acceptable.

[0019] The MFP120 includes a control unit 410, a display unit 420, a printer 421, and a scanner 422. The control unit 410 includes a CPU 411, ROM 412, RAM 413, HDD 414, display unit I / F 415, printer I / F 416, scanner I / F 417, USB I / F 418, and network I / F 419.

[0020] The control unit 410, including the CPU 411, controls the operation of the entire device. The CPU 411 reads control programs stored in the ROM 412 or HDD 414 and executes various control processes. The RAM 413 is used as the main memory for the CPU 411 and as a temporary storage area such as the work area. The HDD 414 stores various programs and data.

[0021] The display interface 415 connects the display unit 420 and the control unit 410. The display unit 420 displays screen information output from the control unit 410 and accepts user input. In Figure 4, the display unit 420 is shown as an integrated unit such as a touch panel capable of detecting user touch operations, but the display unit 420 and the input unit may be configured separately. The printer interface 416 connects the printer 421 and the control unit 410. The scanner interface 417 connects the scanner 422 and the control unit 410.

[0022] The USB I / F 418 connects the camera 423, which is connected externally to the MFP120, to the control unit 410. The camera 423 is a webcam, network camera, or smartphone camera, and captures images and videos (hereinafter collectively referred to as "peripheral images") of the area around the MFP120. In this embodiment, the explanation assumes that a USB device conforming to the UVC (USB Video Class) standard is used as the camera 423. UVC is a camera driver provided by the OS and has an interface for using a webcam via system calls. In the example shown in Figure 4, the MFP120 and camera 423 are shown as separate devices, but the camera 423 may also be built into the MFP120. The network I / F 419 connects the control unit 410 to the network 100.

[0023] Figure 5 is a block diagram showing an example of the software configuration of the MFP120. The CPU 411 reads the program stored in the MFP120's HDD 414 into the RAM 413 and executes it, so that the CPU 411 functions as the various parts shown in Figure 5 and performs the processing described later. The MFP120 has a screen display unit 510, a detection unit 520, an imaging control unit 530, and a remote server 540. The remote server 540 has a screen image acquisition unit 541 and a transmission unit 542.

[0024] The screen display unit 510 displays various screens on the display unit 420. The screen display unit 510 also displays buttons (also called app buttons) for calling up applications on the menu screen. App buttons are buttons for calling up various functions such as applications installed on the MFP120. The detection unit 520 detects the start event of imaging by the camera 423. The imaging control unit 530 controls the imaging of the surrounding image of the MFP120 using the camera 423.

[0025] The remote server 540 is an application that establishes a remote operation connection with the remote client 301 via the network 100. The screen image acquisition unit 541 acquires a screen image, which is an image showing the screen of the screen display unit 510. The transmission unit 542 transmits the screen image to the remote client 301 and receives operations performed on the screen image by the remote client 301, and notifies the control unit 410 of the received operations.

[0026] Figure 6 is a flowchart showing an example of the processing performed in the MFP120 in this embodiment. The processing shown in Figure 6 is realized by the CPU 411 of the MFP120 functioning as the various functional units shown in Figure 5, as described above. That is, the flowchart in Figure 6 is realized by the CPU 411 reading a program stored in the ROM 412 or HDD 414, etc., into the RAM 413 and executing it. Note that some or all of the functions of the steps in Figure 6 may be realized by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process means that it is a step in the flowchart (the same applies to flowcharts in this specification hereafter).

[0027] Figure 6(a) is a flowchart showing an example of the processing that occurs when a remote operation connection is initiated in the MFP120. The processing in this flowchart is initiated when the MFP120's remote server 540 receives a connection request, which is a request for a remote operation connection from the remote client 301. Figure 7 is an example of a camera imaging error screen displayed on the MFP120's display screen. The processing that occurs when a remote operation connection is initiated in the MFP120 will be explained using Figures 6 and 7.

[0028] When the remote server 540 receives a connection request from the remote client 301, in S601 the remote server 540 determines whether or not imaging is in progress. Specifically, the remote server 540 queries the imaging control unit 530 to determine whether or not imaging is in progress. The imaging control unit 530 queries the control unit 410 to determine whether or not imaging is being performed by the camera 423. If it is determined that imaging is in progress, the remote server 540 proceeds to S602; if it is determined that imaging is not in progress, the remote server 540 proceeds to S603.

[0029] In S602, the remote server 540 stops imaging with the camera 423. That is, the remote server 540 sends an instruction to the imaging control unit 530 to stop camera imaging. When the imaging control unit 530 stops camera imaging, it displays a message on the display unit 402 indicating that camera imaging has been stopped. For example, in S602, the display screen of the display unit 402 displays a message 710 indicating that camera imaging has been stopped, as shown in Figure 7(a). Message 710 indicates that camera imaging will be stopped because a remote operation connection has been started. After S602, the remote server 540 proceeds to S603. As mentioned above, if imaging is not in progress, the remote server 540 proceeds to S603 without going through S602.

[0030] In S603, the remote server 540 establishes a remote operation connection with the remote client 301. That is, the remote server 540 establishes a remote operation connection in response to the connection request that triggered the processing shown in the flowchart in Figure 6(a). Then, the remote server 540 completes the processing shown in the flowchart in Figure 6(a).

[0031] Figure 6(b) is a flowchart showing an example of the processing performed when the detection unit 520 of the MFP120 detects an image capture start event of the camera 423. The processing shown in the flowchart in Figure 6(b) is initiated when the detection unit 520 of the MFP120 detects an image capture start event of the camera 423 due to user operation on the screen display unit 501.

[0032] In S611, the imaging control unit 530 determines whether the MFP120 is connected to a remote operation system. For example, the imaging control unit 530 determines whether a remote operation connection (VNC connection) is in place by querying the remote server 540. If it determines that a remote operation connection is in place, the imaging control unit 530 proceeds to S612. If it determines that a remote operation connection is not in place, the imaging control unit 530 proceeds to S613.

[0033] In S612, the imaging control unit 530 cancels camera imaging. That is, even if the imaging control unit 530 detects the camera 423's imaging start event as the start trigger in the flowchart of Figure 6(b), it does not perform camera imaging. In S612, the imaging control unit 530 displays a message on the display unit 402 indicating that a remote operation connection is in progress. For example, in S612, the display screen of the display unit 402 displays message 720, as shown in Figure 7(b), indicating that camera imaging will not be performed. Message 720 indicates that the camera cannot be used because a remote operation connection is in progress. After the processing in S612 is completed, the imaging control unit 530 terminates the processing in the flowchart shown in Figure 6(b).

[0034] In S613, the imaging control unit 530 starts imaging with the camera 423. That is, the imaging control unit 530 starts imaging with the camera 423 in response to the detection of the camera 423 imaging start event, which is the start trigger for the flowchart in Figure 6(b). Then, the imaging control unit 530 finishes the processing shown in the flowchart in Figure 6(b).

[0035] In this embodiment, the image capture start event detected by the detection unit 520 is described as a user operation on the display unit 420, but the system is not limited to this example. The image capture start event may also be detected by a sensor separately attached to the MFP 120. Furthermore, the detection unit 520 may detect human body movement using a camera with an integrated human motion sensor as the image capture start event, or it may detect operation instructions via voice input.

[0036] Furthermore, in this embodiment, an example has been described in which, when a camera imaging request and a remote operation connection request are made in parallel, the remote operation connection is prioritized and camera imaging is canceled. However, this is not limited to this example. When a camera imaging request and a remote operation connection request are made in parallel, the system may be configured to prioritize camera imaging and cancel the remote operation connection. Alternatively, the remote server 540 may be configured so that the user can choose whether to prioritize the camera imaging request or the remote operation connection request.

[0037] As described above, this embodiment improves security in remote operations. Specifically, in this embodiment, if a camera imaging request is made while a remote operation connection is established, or if a remote operation connection request is made while camera imaging is being performed, only one of the two processes, remote operation connection or camera imaging, is enabled. The other process is then canceled. This control prevents remote operation connection and camera imaging from being performed simultaneously. As a result, it is possible to prevent the captured images from being exposed to the remote client, thereby reducing security risks.

[0038] <<Second Embodiment>> In the first embodiment, an example was described in which camera imaging is stopped so that both remote operation connection and camera imaging are not performed simultaneously when a remote operation connection request is received or when a camera imaging start event is detected. Alternatively, an example was described in which only the remote operation connection is stopped, or the priority between camera imaging and remote operation connection can be changed by user settings. Although these processes can improve security, they may reduce user operability because one of the operations cannot be performed on the MFP120.

[0039] This embodiment describes an example that enables simultaneous execution of camera imaging and remote operation connection while protecting security. This embodiment describes an example in which the MFP120 overwrites (hereinafter referred to as masking) the captured image displayed on the remote client 301.

[0040] The system configuration, hardware configuration of the communication device, software configuration of the communication device, and hardware configuration of the MFP in this embodiment can be the same as those described in the example in the first embodiment, so their description is omitted.

[0041] Figure 8 shows an example of the software configuration of the MFP120 in this embodiment. The MFP120 includes a screen display unit 510, a detection unit 520, an imaging unit 810, and a remote server 820. The remote server 820 includes a screen image acquisition unit 541, a transmission unit 542, and a transmission image generation unit 821.

[0042] The screen display unit 510 and the detection unit 520 have the same configuration as described in the first embodiment. The imaging unit 810 uses the camera 423 to capture images of the area around the MFP 120. The imaging control unit 530 in the first embodiment is a processing unit that controls the stopping of imaging depending on the status of the remote operation connection. Unlike the imaging control unit 530 in the first embodiment, the imaging unit 810 in this embodiment is a processing unit that does not control the stopping of imaging depending on the status of the remote operation connection.

[0043] The remote server 820 establishes a remote operation connection with the remote client 301 via the network 100. The transmission image generation unit 821 within the remote server 820 generates a transmission image for transmission to the remote client 301. The screen image acquisition unit 541 and transmission unit 542 within the remote server 820 are the same as those described in the first embodiment.

[0044] Figure 9 is a flowchart showing an example of the screen image transmission process during a remote operation connection in the MFP120. Similar to the example described in Figure 6, the process shown in Figure 9 is realized by the MFP120's CPU 411 functioning as the various functional units shown in Figure 8, as previously mentioned. The flowchart in Figure 9 begins based on the remote server 820 responding to a remote operation connection request from the remote client 301 and establishing a remote operation connection.

[0045] S901 indicates a loop process. While the remote server 820 is connected to the remote client 301 for remote operation (VNC connection), it repeats the processes from S902 to S906. When the remote operation connection with the remote client 301 is terminated, the remote server 820 terminates the process shown in the flowchart in Figure 9.

[0046] In S902, the screen image acquisition unit 541 captures the screen displayed on the display unit 420. For example, the screen image acquisition unit 541 acquires the screen image of the MFP120's display unit 510. Next, in S903, the remote server 820 determines whether the imaging unit 810 is taking images with the camera 423, that is, whether it is taking images or not. For example, the remote server 820 makes this determination by querying the imaging unit 810 whether it is taking images. If it determines that it is taking images, the remote server 820 proceeds to S904; if it determines that it is not taking images, the remote server 820 proceeds to S906.

[0047] In S904, the remote server 820 instructs the transmission image generation unit 821 to generate a transmission image. The transmission image is an image in which the area of ​​the captured image in the screen image is masked. The transmission image generation unit 821 acquires the screen image from the screen display unit 510. The transmission image generation unit 821 also acquires area information indicating the area of ​​the captured image in the screen image from the RAM 413. Then, it generates the transmission image by applying a masking process to the area indicated by the area information in the screen image. Details will be described later using Figure 10. Next, in S905, the remote server 820 instructs the transmission unit 542 to send the transmission image generated in S904. Then, it returns to the loop start process in S901. Meanwhile, in S906, the remote server 820 instructs the transmission unit 542 to send the screen image displayed by the screen display unit 510. Then, it returns to the loop start process in S901.

[0048] Figure 10 is a diagram illustrating the transmitted image. Figure 10(a) shows an example of a screen image 1010. Figure 10(b) shows an example of coordinate information 1030, which is an example of region information. Figure 10(c) shows an example of a masked transmitted image 1020. Coordinate information 1030 is information indicating the position coordinates of the captured image 1011 within the screen image 1010 acquired in S904. Coordinate information 1030 includes the vertical distance 1031 and horizontal distance 1032 from the top left corner (P1013) of the screen image 1010 to the top left corner (P1014) of the captured image 1011, as well as the width 1033 and height 1034 of the captured image 1011. By using this information, the transmitted image generation unit 821 can identify the region of the captured image 1011 within the screen image 1010. The transmission image generation unit 821 generates the transmission image 1020 by applying the mask 1021 to the region of the captured image 1011 within the screen image 1010 identified in this way.

[0049] In this embodiment, an example of not displaying the captured image in the image transmitted during a remote operation connection was described in which the captured image is masked by filling it with a predetermined pattern, but this is not the only example. For example, the captured image may be masked by overlaying it with another image, or any other arbitrary method may be used to prevent the captured image from being displayed.

[0050] As explained above, according to this embodiment, by masking the area of ​​the captured image in the image transmitted to the remote client, it is possible to prevent the captured image from being exposed to the remote client, thereby reducing security risks.

[0051] <<Third Embodiment>> In the second embodiment, an example was described in which the masking of the captured image is performed by a remote server. In this embodiment, an example is described in which the masking of the captured image is performed on the remote client side.

[0052] The system configuration, the hardware configuration of the communication device, and the hardware configuration of the MFP in this embodiment can be the same as those described in the first embodiment, so their description is omitted here.

[0053] Figure 11 shows an example of the software configuration of the MFP120 in this embodiment. The MFP120 includes a screen display unit 510, a detection unit 520, an imaging unit 810, and a remote server 1110. The remote server 1110 includes a screen image acquisition unit 541, a transmission unit 1112, and a screen information generation unit 1111.

[0054] The screen display unit 510 and the detection unit 520 have the same configuration as described in the first embodiment. The imaging unit 810 has the same configuration as described in the second embodiment.

[0055] The remote server 1110 establishes a remote operation connection with the communication device 110 via the network 100. The screen information generation unit 1111 within the remote server 1110 generates screen information to be sent to the remote client 301. In this embodiment, the data (information) sent from the remote server 1110 to the remote client 301 is screen information. The screen information may consist only of a screen image, as will be described later, or it may include a screen image and coordinate information as described in the second embodiment. The transmission unit 1112 within the remote server 1110 transmits the screen information to the remote client 301. The screen image acquisition unit 541 within the remote server 1110 is the same as in the example described in the first embodiment.

[0056] Figure 12 shows an example of the software configuration of the communication device 110 in this embodiment. The communication device 110 has a remote client 1210. The remote client 1210 is an application that establishes a remote operation connection with the remote server 1110 of the MFP 120 via the network 100. The remote client 1210 has a transmitting / receiving unit 1211 and a display image generation unit 1212.

[0057] The display image generation unit 1212 in the remote client 1210 generates a display image for display on the display unit 206 of the communication device 110. The transmitting / receiving unit 1211 receives screen information from the remote server 1110 and sends it to the display image generation unit 1212.

[0058] Figure 13 is a flowchart showing an example of screen information transmission and reception processing during remote operation connection in this embodiment. Figure 13(a) is a flowchart showing an example of screen information transmission processing performed in the MFP120. Figure 13(b) is a flowchart showing an example of screen information reception processing in the communication device 110. The processing shown in Figure 13(a) is realized by the CPU 411 of the MFP120 functioning as the various functional units shown in Figure 11, as described above, similar to the example described in Figure 6. The processing shown in Figure 13(b) is realized by the CPU 201 of the communication device 110 functioning as the various functional units shown in Figure 12, as described above. That is, the flowchart in Figure 13(b) is realized by the CPU 201 reading a program stored in ROM 202 or HDD 204, etc., into RAM 203 and executing it. Note that some or all of the functions of the steps in Figure 13(b) may be realized by hardware such as an ASIC or electronic circuit.

[0059] The processing steps in each flowchart shown in Figures 13(a) and (b) are initiated based on the establishment of a remote operation connection between the remote client 1210 and the remote server 1110. Figure 14 shows the screen information when the camera 423 is not capturing images.

[0060] First, the screen information transmission process performed in the MFP120 will be explained using Figure 13(a). S1301 indicates a loop process. The remote server 1110 repeats the processes from S1302 to S1306 while it is connected to the remote client 1210 for remote operation (VNC connection). When the remote operation connection with the remote client 1210 is terminated, the remote server 1110 terminates the process shown in the flowchart in Figure 13(a).

[0061] In S1302, the screen image acquisition unit 541 captures the screen displayed on the display unit 420. For example, the screen image acquisition unit 541 acquires the screen image of the MFP 120's display unit 510. Next, in S1303, the remote server 1110 determines whether the imaging unit 810 is taking images with the camera 423, that is, whether it is taking images or not. For example, the remote server 1110 makes this determination by querying the imaging unit 810 to see if it is taking images. If it determines that it is taking images, the remote server 1110 proceeds to S1304; if it determines that it is not taking images, the remote server 1110 proceeds to S1305.

[0062] In S1304, the remote server 1110 instructs the screen information generation unit 1111 to generate screen information including the screen image 1010 and the coordinate information 1030 of the captured image 1011 within the screen image 1010, as explained in Figure 10. Then, the remote server 1110 proceeds to S1306.

[0063] In S1305, the remote server 1110 instructs the screen information generation unit 1111 to generate screen information including the screen image 1410 shown in Figure 14. That is, unlike in S1304, it generates screen information that includes screen information but does not include coordinate information. Then, the remote server 1110 proceeds to S1306. The screen image 1410 in Figure 14 is the screen image 1410 when the camera 423 is not capturing images, so unlike the example shown in Figure 10(a), it does not include the captured image 1011.

[0064] In S1306, the remote server 1110 sends the screen information generated in S1305 or S1306 to the remote client 1210. Then, it returns to the loop start process in S1301.

[0065] Next, the screen information reception process performed in the communication device 110 will be explained using Figure 13(b). S1311 indicates a loop process. The remote client 1210 repeats the processes from S1312 to S1316 while it is connected to the remote server 1110 for remote operation (VNC connection). When the remote operation connection with the remote server 1110 is terminated, the remote client 1210 terminates the process shown in the flowchart in Figure 13(b).

[0066] In S1312, the transmitting / receiving unit 1211 of the remote client 1210 receives screen information from the remote server 1110. In S1313, the display image generation unit 1212 of the remote client 1210 determines whether the screen information received in S1312 includes coordinate information 1030. If it determines that the screen information includes coordinate information, the display image generation unit 1212 proceeds to S1314. If it determines that the screen information does not include coordinate information, the display image generation unit 1212 proceeds to S1316.

[0067] In S1314, the display image generation unit 1212 generates a display image by applying a mask 1021 to the area indicated by the coordinate information 1030 within the screen image 1010, as explained in Figure 10. The example of the masking process is the same as the example explained in the third embodiment. Then, the display image generation unit 1212 proceeds to S1315.

[0068] In S1315, the display image generation unit 1212 displays the display image generated in S1314 on the display unit 206. Then, the process returns to the loop start process in S1311.

[0069] In S1316, the display image generation unit 1212 displays the screen image received in S1312 on the display unit 206. Then, it returns to the loop start process in S1311.

[0070] As explained above, according to this embodiment, by performing a masking process on the captured image in the remote client 1210, it is possible to prevent the captured image from being exposed to the remote client, thereby reducing security risks.

[0071] <<Fourth Embodiment>> In the first embodiment, an example was described in which camera imaging is stopped when a remote operation connection request is received and when a camera imaging start event is detected, so that the remote operation connection and camera imaging processes are not performed simultaneously. Although this can improve security, stopping camera imaging prevents operation of the MFP120, which may reduce user usability. For example, the image captured by camera 423 may be used not only for facial recognition for logging into the MFP120, but also for reading 2D codes for submitting print jobs. In that case, even though there is no problem in exposing the 2D code to the remote client 301, camera imaging would be stopped.

[0072] Therefore, in this embodiment, we will describe an example in which camera imaging and remote operation connection are controlled after confirming the authentication settings and login status of the MFP120. The system configuration, hardware configuration of the communication device, software configuration of the communication device, hardware configuration of the MFP, and software configuration of the MFP in this embodiment can be the same as in the example described in the first embodiment, so we will omit their explanation.

[0073] Figure 15 is a flowchart showing an example of the processing performed in the MFP120 in this embodiment. The processing shown in Figure 15 is realized by the CPU 411 of the MFP120 functioning as the various functional units shown in Figure 5, as described above.

[0074] Figure 15(a) is a flowchart showing the processing when a remote operation connection is initiated in the MFP120. The processing shown in the flowchart in Figure 15(a) is initiated when the remote server 540 receives a connection request from the remote client 301. The difference between Figure 15(a) and Figure 6(a) is the addition of S1502 and S1503. S1501, S1504, and S1505 are the same as S601, S602, and S603 in Figure 6(a), so their explanation is omitted.

[0075] In S1502, the remote server 540 requests the control unit 410 to retrieve the authentication settings for the MFP120 stored in the HDD 414 and determines whether the authentication settings are ON. If it determines that the authentication settings are ON, the remote server 540 proceeds to S1503. If it determines that the authentication settings are not ON, the remote server 540 proceeds to S1505.

[0076] In S1503, the remote server 540 queries the control unit 410 to determine whether a user has logged in to the MFP120 and whether or not the user has not yet logged in. If it determines that the user has not yet logged in, the remote server 540 proceeds to S1504; otherwise, it proceeds to S1505.

[0077] In other words, in the process shown in Figure 15(a), if the authentication setting is ON and a remote operation connection request is received before login, the imaging is stopped and the remote operation connection is established. This is because the images captured by camera 423 are highly likely to be used for facial recognition to log in to MFP120, and therefore, the imaging is stopped for security reasons. On the other hand, if the authentication setting is OFF, or if the authentication setting is ON and the user is already logged in, the images captured by camera 423 are highly unlikely to be used for facial recognition to log in to MFP120. In this case, since there are fewer security concerns, the process of establishing the remote operation connection is performed without stopping the imaging.

[0078] Next, using Figure 15(b), we will explain the processing that occurs when the detection unit 520 of the MFP120 detects an image capture start event for the camera 423. The flowchart shown in Figure 15(b) is initiated when the detection unit 520 of the MFP120 detects an image capture start event for the camera 423 due to user operation on the screen display unit 501. The difference between Figure 15(b) and Figure 6(b) is the addition of S1512 and S1513. The other steps, S1511, S1514, and S1515, are the same as S611, S612, and S613 in Figure 6(b), so their explanation is omitted.

[0079] In S1512, the imaging control unit 530 requests the control unit 410 to retrieve the authentication settings for the MFP120 stored in the HDD 414 and determines whether the authentication settings are ON. If it determines that the authentication settings are ON, the imaging control unit 530 proceeds to S1513. If it determines that the authentication settings are not ON, the imaging control unit 530 proceeds to S1515. In S1513, the imaging control unit 530 inquires with the control unit 410 whether a user has logged in to the MFP120 and determines whether the login has not yet occurred. If it determines that the login has not yet occurred, the imaging control unit 530 proceeds to S1514. If it determines that the login has not yet occurred, the imaging control unit 530 proceeds to S1515.

[0080] In other words, in the process shown in Figure 15(b), if a remote operation connection is established, authentication is enabled, and the user has not yet logged in, and an image capture start event is detected, the image capture process is stopped. This is because the image captured by camera 423 is likely to be used for facial recognition to log in to the MFP120, and therefore, for security reasons, the image capture process is stopped. On the other hand, if the authentication setting is disabled while a remote operation connection is established, or if the authentication setting is enabled and the user has already logged in, the image captured by camera 423 is unlikely to be used for facial recognition to log in to the MFP120. In this case, since there are fewer security concerns, the image capture process will start in response to the image capture start event.

[0081] In this embodiment, the login state when the authentication setting of the MFP120 is ON was used as an example to explain the use of imaging when controlling the system to prevent simultaneous remote operation connection and camera imaging. However, instead of controlling the system to prevent camera imaging, it is also possible to employ a process that controls the display so that human faces are not displayed, for example, by setting the camera image to a state where no faces are recognized.

[0082] Furthermore, when a service technician connects for remote operation, such as when camera 423 cannot capture images or when user login using captured images is not possible, it is more efficient to be able to work while checking the images captured by camera 423. For this reason, the system may be configured to add specific parameters to connection requests from specific remote clients, and to allow simultaneous execution of remote operation connection and camera imaging depending on the presence or absence of these specific parameters in the request parameters.

[0083] Furthermore, under a policy that captured images may be made public within the same network, such as a company LAN, the system may be configured to allow simultaneous execution of remote operation connection and camera imaging if they belong to the same network. In that case, for example, instead of S1502-S1503 and S1512-S1513, the following processing may be performed. That is, the system may perform a process to determine whether the result of a logical AND operation performed on the IP address of the remote client and the IP address of the image processing device using the subnet mask of the image processing device is the same. If they are within the same network, the process of S1505 or S1515 may be executed, and if it is determined that they are not on the same network, the process of S1504 or S1514 may be performed. This makes it possible to allow simultaneous execution of remote operation connection and camera imaging within the same network, such as a company LAN.

[0084] As described above, according to this embodiment, simultaneous execution of remote operation connection and camera imaging can be permitted when certain conditions for low security risk are met. This improves usability. On the other hand, under conditions of high security risk, security risk can be reduced by implementing control that prohibits simultaneous execution of remote operation connection and camera imaging, as described in the first embodiment.

[0085] <<Other Embodiments>> In the fourth embodiment, an example was described in which, when certain conditions for low security risk are met, simultaneous execution of remote operation connection and camera imaging, as described in the first and fourth embodiments, may be permitted rather than prohibited. The processing when these predetermined conditions are met can also be applied to the examples described in the second and third embodiments. That is, the system may be configured not to perform the masking process when the aforementioned predetermined conditions are met. For example, after the YES determination in S903, it may be determined whether the predetermined conditions are met, and if the predetermined conditions are met, the system may proceed to S906, and if the predetermined conditions are not met, the system may be configured to perform the processing in S904. Similarly, after the YES determination in S1303, it may be determined whether the predetermined conditions are met, and if the predetermined conditions are met, the system may proceed to S1305, and if the predetermined conditions are not met, the system may be configured to perform the processing in S1304.

[0086] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0087] The disclosure of this embodiment includes configurations represented by the following examples of information processing devices, control methods for information processing devices, and programs.

[0088] <Configuration 1> An information processing device, The information processing device includes an imaging control means for controlling the acquisition of images of the surrounding area, A detection means for detecting the event that initiates the imaging, A display control means that displays the captured image, which is an image captured according to the control of the aforementioned imaging control means, on a display screen. A connection means for connecting the communication device to the communication device to enable remote operation, which displays the image shown on the display screen and enables operation from the communication device to the information processing device. When the connection means receives a connection request for the remote operation, or when the detection means detects the event while the remote operation is connected, the control means controls the imaging control means to stop imaging or to stop the connection of the remote operation. An information processing device characterized by having the following features.

[0089] <Configuration 2> The information processing apparatus according to Configuration 1, characterized in that the control means controls the imaging control means to stop imaging when it receives a connection request for the remote operation in the connection means, and controls the imaging control means to stop imaging when the event is detected by the detection means while the remote operation is connected.

[0090] <Structure 3> The information processing apparatus according to configuration 1 or 2, characterized in that the display control means displays a message indicating cancellation on the display screen when the imaging control means performs control to cancel imaging.

[0091] <Structure 4> The information processing apparatus according to any one of configurations 1 to 3, characterized in that, when a predetermined condition is met, the control means controls the connection means to receive a connection request for the remote operation, or when the detection means detects the event while the remote operation is connected, so as not to cancel the imaging by the imaging control means and not to cancel the connection for the remote operation.

[0092] <Composition 5> An information processing device, The information processing device includes an imaging control means for controlling the acquisition of images of the surrounding area, A display control means that displays the captured image, which is an image captured according to the control of the aforementioned imaging control means, on a display screen. A connection means for connecting the communication device to the communication device to enable remote operation, which displays the image shown on the display screen and enables operation from the communication device to the information processing device. When imaging is performed by the imaging control means while the remote operation is connected, a generation means generates an image with the area of ​​the captured image on the display screen masked, A transmission means for transmitting the generated image to the communication device via the remote operation, An information processing device characterized by having the following features.

[0093] <Composition 6> The generation means, when certain conditions are met, does not generate an image that masks the area of ​​the captured image on the display screen, even when imaging is performed by the imaging control means while the remote operation is connected. The information processing apparatus according to configuration 5, characterized in that, if the image is not generated by the generation means, the transmission means transmits the image displayed on the display screen to the communication device by remote operation.

[0094] <Composition 7> An information processing device, The information processing device includes an imaging control means for controlling the acquisition of images of the surrounding area, A display control means that displays the captured image, which is an image captured according to the control of the aforementioned imaging control means, on a display screen. A connection means for connecting the communication device to the communication device to enable remote operation, which displays the image shown on the display screen and enables operation from the communication device to the information processing device. When imaging is performed by the imaging control means while the remote operation is connected, a transmission means transmits to the communication device information indicating the region of the image on the display screen and the image displayed on the display screen. An information processing device characterized by having the following features.

[0095] <Structure 8> The information processing apparatus according to configuration 7, characterized in that, when predetermined conditions are met, even when imaging is performed by the imaging control means while the remote operation is connected, the transmitting means does not transmit information indicating the area of ​​the captured image on the display screen, but transmits the image displayed on the display screen to the communication device.

[0096] <Composition 9> The information processing device according to any one of configurations 4, 6, and 8, characterized in that the predetermined condition includes the authentication setting of the information processing device not being turned ON.

[0097] <Composition 10> The information processing device according to configuration 9, characterized in that the predetermined conditions include being logged in even when the authentication setting is turned ON.

[0098] <Composition 11> The information processing device according to any one of configurations 4, 6, and 8 to 10, characterized in that the predetermined conditions include the provision of specific parameters when a connection request for the remote operation is made from the communication device.

[0099] <Composition 12> The information processing apparatus according to any one of configurations 4, 6, and 8 to 11, characterized in that the predetermined conditions include the information processing apparatus and the communication apparatus belonging to the same network.

[0100] <Composition 13> A communication device, A connection means for connecting to the information processing device and the information processing device to display an image on the display screen of the information processing device on the communication device, and to enable remote operation of the information processing device by the communication device, When the information processing device receives an image displayed on the display screen and information indicating the region of the captured image on the display screen during the connection of the remote operation, the generation means generates an image with the region of the captured image on the display screen masked. When the image is generated by the generation means, the display control means displays the image generated by the generation means, and when the information processing device does not receive information indicating the area of ​​the captured image on the display screen during the connection of the remote operation, the display control means displays the image that was received and displayed on the display screen. A communication device characterized by having the following features.

[0101] <Composition 14> A method for controlling an information processing device, The steps include controlling the acquisition of an image of the surroundings of the information processing device, The steps include detecting an event that initiates the aforementioned imaging, The steps include: displaying the captured image, which is an image captured according to the control described above, on a display screen; The steps include: displaying the image shown on the display screen on the communication device, and establishing a remote operation connection with the communication device that enables operation from the communication device to the information processing device; When a connection request for the remote operation is received, or when the event is detected while the remote operation is connected, the system controls the system to either stop the imaging or to cancel the connection to the remote operation. A control method for an information processing device, characterized by having the following features.

[0102] <Composition 15> A method for controlling an information processing device, The steps include controlling the acquisition of an image of the surroundings of the information processing device, The steps include: displaying the captured image, which is an image captured according to the control described above, on a display screen; The steps include: displaying the image shown on the display screen on the communication device, and establishing a remote operation connection with the communication device that enables operation from the communication device to the information processing device; If the image is taken while the remote operation is connected, the steps include generating an image with the area of ​​the captured image on the display screen masked, The steps include transmitting the generated image to the communication device via the remote operation, A control method for an information processing device, characterized by having the following features.

[0103] <Composition 16> A method for controlling an information processing device, The steps include controlling the acquisition of an image of the surroundings of the information processing device, The steps include: displaying the captured image, which is an image captured according to the control described above, on a display screen; The steps include: displaying the image shown on the display screen on the communication device, and establishing a remote operation connection with the communication device that enables operation from the communication device to the information processing device; If the image is taken while the remote operation is connected, the communication device is sent information indicating the region of the captured image on the display screen and the image displayed on the display screen. A control method for an information processing device, characterized by having the following features.

[0104] <Composition 17> A program for causing a computer to execute the control method of the information processing device described in any one of items 14 to 16 of the configuration.

[0105] <Composition 18> A method for controlling a communication device, The steps include: displaying an image on the display screen of the information processing device on the communication device, and establishing a remote operation connection with the information processing device that enables the communication device to operate the information processing device; When the information processing device receives an image displayed on the display screen and information indicating the region of the captured image on the display screen during the connection of the remote operation, the step of generating an image with the region of the captured image on the display screen masked, If the aforementioned image is generated, the generated image is displayed, and if information indicating the region of the captured image on the display screen is not received from the information processing device during the connection of the remote operation, the image displayed on the received display screen is displayed. A method for controlling a communication device, characterized by having the following features.

[0106] <Composition 19> A program to cause a computer to execute the control method for the communication device described in Configuration 18. [Explanation of Symbols]

[0107] 110 Communication equipment 120 MFP 410 Control Unit 411 CPU

Claims

1. An information processing device, The information processing device includes an imaging control means for controlling the acquisition of images of the surrounding area, A detection means for detecting the event that initiates the imaging, A display control means that displays the captured image, which is an image captured according to the control of the aforementioned imaging control means, on a display screen. A connection means for connecting the communication device to the communication device to enable remote operation, which displays the image shown on the display screen and enables operation from the communication device to the information processing device. When the connection means receives a connection request for the remote operation, or when the detection means detects the event while the remote operation is connected, the control means controls the imaging control means to stop imaging or to stop the connection of the remote operation. An information processing device characterized by having the following features.

2. The information processing apparatus according to claim 1, characterized in that the control means controls the imaging control means to stop imaging when it receives a connection request for the remote operation in the connection means, and controls the imaging control means to stop imaging when the event is detected by the detection means while the remote operation is connected.

3. The information processing apparatus according to claim 1 or 2, characterized in that the display control means displays a message indicating cancellation on the display screen when the imaging control means performs control to cancel imaging.

4. The information processing apparatus according to claim 1, characterized in that, when predetermined conditions are met, the control means controls the connection means to receive a connection request for the remote operation, or when the detection means detects the event while the remote operation is connected, so as not to cancel the imaging by the imaging control means and not to cancel the connection for the remote operation.

5. An information processing device, The information processing device includes an imaging control means for controlling the acquisition of images of the surrounding area, A display control means that displays the captured image, which is an image captured according to the control of the aforementioned imaging control means, on a display screen. A connection means for connecting the communication device to the communication device to enable remote operation, which displays the image shown on the display screen and enables operation from the communication device to the information processing device. When imaging is performed by the imaging control means while the remote operation is connected, a generation means generates an image with the area of ​​the captured image on the display screen masked, A transmission means for transmitting the generated image to the communication device via the remote operation, An information processing device characterized by having the following features.

6. The generation means, when certain conditions are met, does not generate an image that masks the area of ​​the captured image on the display screen, even when imaging is performed by the imaging control means while the remote operation is connected. The information processing apparatus according to claim 5, characterized in that, if the image is not generated by the generation means, the transmission means transmits the image displayed on the display screen to the communication device by remote operation.

7. An information processing device, The information processing device includes an imaging control means for controlling the acquisition of images of the surrounding area, A display control means that displays the captured image, which is an image captured according to the control of the aforementioned imaging control means, on a display screen. A connection means for connecting the communication device to the communication device to enable remote operation, which displays the image shown on the display screen and enables operation from the communication device to the information processing device. When imaging is performed by the imaging control means while the remote operation is connected, a transmission means transmits to the communication device information indicating the region of the image on the display screen and the image displayed on the display screen. An information processing device characterized by having the following features.

8. The information processing apparatus according to claim 7, characterized in that, when predetermined conditions are met, even when imaging is performed by the imaging control means while the remote operation is connected, the transmitting means does not transmit information indicating the area of ​​the captured image on the display screen, but transmits the image displayed on the display screen to the communication device.

9. The information processing device according to any one of claims 4, 6, and 8, characterized in that the predetermined condition includes the authentication setting of the information processing device not being turned ON.

10. The information processing device according to claim 9, characterized in that the predetermined conditions include being logged in even when the authentication setting is turned ON.

11. The information processing apparatus according to any one of claims 4, 6, and 8, characterized in that the predetermined conditions include the provision of specific parameters when a connection request for the remote operation is made from the communication device.

12. The information processing apparatus according to any one of claims 4, 6, and 8, characterized in that the predetermined conditions include the information processing apparatus and the communication apparatus belonging to the same network.

13. A communication device, A connection means for connecting to the information processing device and the information processing device to display an image on the display screen of the information processing device on the communication device, and to enable remote operation of the information processing device by the communication device, When the information processing device receives an image displayed on the display screen and information indicating the region of the captured image on the display screen during the connection of the remote operation, the generation means generates an image with the region of the captured image on the display screen masked. When the image is generated by the generation means, the display control means displays the image generated by the generation means, and when the information processing device does not receive information indicating the area of ​​the captured image on the display screen during the connection of the remote operation, the display control means displays the image that was received and displayed on the display screen. A communication device characterized by having the following features.

14. A method for controlling an information processing device, The steps include controlling the acquisition of an image of the surroundings of the information processing device, The steps include detecting an event that initiates the aforementioned imaging, The steps include: displaying the captured image, which is an image captured according to the control described above, on a display screen; The steps include: displaying the image shown on the display screen on the communication device, and establishing a remote operation connection with the communication device that enables operation from the communication device to the information processing device; When a connection request for the remote operation is received, or when the event is detected while the remote operation is connected, the system controls the system to either stop the imaging or to cancel the connection to the remote operation. A control method for an information processing device, characterized by having the following features.

15. A method for controlling an information processing device, The steps include controlling the acquisition of an image of the surroundings of the information processing device, The steps include: displaying the captured image, which is an image captured according to the control described above, on a display screen; The steps include: displaying the image shown on the display screen on the communication device, and establishing a remote operation connection with the communication device that enables operation from the communication device to the information processing device; If the image is taken while the remote operation is connected, the steps include generating an image with the area of ​​the captured image on the display screen masked, The steps include transmitting the generated image to the communication device via the remote operation, A control method for an information processing device, characterized by having the following features.

16. A method for controlling an information processing device, The steps include controlling the acquisition of an image of the surroundings of the information processing device, The steps include: displaying the captured image, which is an image captured according to the control described above, on a display screen; The steps include: displaying the image shown on the display screen on the communication device, and establishing a remote operation connection with the communication device that enables operation from the communication device to the information processing device; If the image is taken while the remote operation is connected, the communication device is sent information indicating the region of the captured image on the display screen and the image displayed on the display screen. A control method for an information processing device, characterized by having the following features.

17. A program for causing a computer to execute the control method of an information processing device according to any one of claims 14 to 16.

18. A method for controlling a communication device, The steps include: displaying an image on the display screen of the information processing device on the communication device, and establishing a remote operation connection with the information processing device that enables the communication device to operate the information processing device; When the information processing device receives an image displayed on the display screen and information indicating the region of the captured image on the display screen during the connection of the remote operation, the step of generating an image with the region of the captured image on the display screen masked, If the aforementioned image is generated, the generated image is displayed, and if information indicating the region of the captured image on the display screen is not received from the information processing device during the connection of the remote operation, the image displayed on the received display screen is displayed. A method for controlling a communication device, characterized by having the following features.

19. A program for causing a computer to execute the control method of the communication device described in claim 18.

Images