Electronic device, information processing system, and method for controlling electronic device

Abstract

To provide a technology for more appropriately controlling the propriety of the direct operation of a remote device when the remote access of the remote device is possible.SOLUTION: An electronic apparatus that is a first device includes a reception unit configured to receive an operation from a user, and a control unit configured to control an operation of a second device in response to the operation from the user on the reception unit, in which when the control unit controls the operation of the second device in response to the operation from the user on the reception unit, the control unit performs first control to suppress an operation on the second device different from the operation from the user on the reception unit in a state in which the user cannot operate the second device.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an electronic device, an information processing system, and a method for controlling an electronic device, and more particularly to an information processing system capable of controlling a remotely accessed device. [Background technology]

[0002] Conventionally, there are technologies related to direct operation of a remote device that is the target of remote access (operations for controlling the remote device that are detected by the remote device itself). Such technologies include a technology that displays a lock screen on the remote device to make direct operation difficult, and a technology that enables remote operation and direct operation simultaneously. A remote operation is an operation for controlling a remote device, but is an operation on another device rather than the remote device.

[0003] Patent Document 1 discloses a technique for disabling direct manipulation of a remote device and notifying the user that the direct manipulation has been disabled by displaying a screen. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-360520 Summary of the Invention [Problem to be solved by the invention]

[0005] However, none of the above-mentioned technologies mentions making it easy for only the user using remote access (control of a remote device using remote operation) to directly operate the remote device.

[0006] For example, if remote access to a remote device is made directly operable, the remote access user can operate it directly while remaining in remote access mode, but it also allows a third party to operate it directly. This can be particularly inconvenient if unintended third parties operate the payment screen or online banking screen.

[0007] Therefore, an object of the present invention is to provide a technique for more appropriately controlling whether or not a remote device can be directly operated when the remote device is remotely accessible. [Means for solving the problem]

[0008] The electronic device of the present invention is an electronic device that is a first device and has a receiving means for receiving operations from a user and a control means for controlling the operation of a second device in accordance with the operation from the user to the receiving means, and is characterized in that when the control means controls the operation of the second device in accordance with the operation from the user to the receiving means, it performs a first control that suppresses operations on the second device that are different from the operation from the user to the receiving means when the user is not able to operate the second device. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a technique for more appropriately controlling whether or not a remote device can be directly operated when the remote device is remotely accessible. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is an overall diagram showing the configuration of an information processing system. [Figure 2] FIG. 1 is a block diagram showing a configuration of an information processing system. [Figure 3] FIG. 10 is a diagram showing a state in which an overlay display is performed in a virtual space. [Figure 4]FIG. 10 is a diagram showing a state in which an overlay display is enlarged in the virtual space. [Figure 5] 10 is a flowchart showing control for suppressing a direct operation. [Figure 6] 10 is a setting screen for a control method relating to direct manipulation. [Figure 7] This is a screen for detailed settings of the control method for direct manipulation. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described in detail by way of example with reference to the accompanying drawings. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the present invention.

[0012] <Embodiment 1> FIG. 1 is an overall diagram showing the configuration of an information processing system 1. FIG. 2 is a block diagram showing the functional configuration of the information processing system 1. The information processing system 1 according to the first embodiment presents a mixed reality space, which is a fusion of a real space and a virtual space, to a user of a display device 101 who is experiencing the system. The mixed reality space is also called an MR (Mixed Reality) space. In the first embodiment, the information processing system 1 displays a composite image obtained by combining an image of a virtual space rendered by CG (Computer Graphics) with an image of a real space. In this way, the information processing system 1 presents an MR space (such as the MR space 301 shown in FIG. 3) to the user.

[0013] The information processing system 1 includes a local device 10, a remote device 104, and a device 105. The local device 10 is an electronic device including a display device 101, an information processing device 102, and an operation device 103.

[0014] The display device 101 is a video see-through display device, and is a head-mounted display device (HMD). However, the display device 101 is not limited to such an HMD, and may also be a handheld display (HHD) or the like. An HHD is a handheld display. In other words, the display device 101 may be a display that a user holds in their hand and peers into like binoculars to observe an image. The display device 101 may also be a display terminal such as a tablet or a smartphone.

[0015] The information processing device 102 generates a composite image by combining an image of real space captured from the display device 101 with an image of virtual space generated in the information processing device 102. The information processing device 102 outputs the composite image to the display device 101 as an MR image (mixed reality image).

[0016] The operation device 103 is capable of receiving operations from a user. The operation device 103 is, for example, an operation device for video games that can detect values ​​corresponding to the tilt direction of a stick and values ​​indicating the pressed state of a button.

[0017] The operation device 103 is not limited to an operation device held by the user's hand (such as an operation device for a video game), but may also be an operation device worn on the user's body or hand. The operation device 103 may be, for example, a ring-shaped device that can be worn on the user's finger.

[0018] The remote device 104 is an electronic device that can be remotely accessed from the local device 10. The remote device 104 is a terminal capable of accessing the local device 10 (controlling the local device 10 in response to an operation on the local device 10). The remote device 104 may be a terminal such as a personal computer or a tablet.

[0019] The device 105 is a terminal held or worn by a person (hereinafter referred to as "other person") other than the user (remote access user) operating the local device 10. The device 105 may be, for example, a smartphone, a smart watch, smart glasses, or other device held or worn by the other person.

[0020] The information processing device 102 and the display device 101, the information processing device 102 and the operation device 103, and the information processing device 102 and the remote device 104 are connected to each other so that data communication is possible. Note that the data communication may be wired communication or wireless communication. The display device 101 may have the information processing device 102 built therein. Furthermore, when a camera or the like connected to the information processing device 102 is used, the camera and the information processing device 102 are connected to each other so that data communication is possible. Alternatively, the camera and the display device 101 may be connected to each other so that data communication is possible.

[0021] However, the information processing system 1 according to the first embodiment is not limited to an MR system that displays MR images. For example, the information processing system 1 may be an AR (Augmented Reality) system that presents an image of a virtual space to a user by making the real space transparent. Alternatively, the information processing system 1 may be a VR (Virtual Reality) system that presents only an image of a virtual space to a user. Alternatively, the information processing system 1 may be a so-called XR (Extended Reality) system, such as those exemplified above.

[0022] If the information processing system 1 is a VR system, the user must be able to use an operation unit to directly operate the remote device 104 in real space. If the information processing system 1 is a VR system, direct operation may be possible by, for example, superimposing CG on the operation unit. Furthermore, the information processing system 1 does not need to be an XR system. For example, the information processing system 1 may simply be a system that allows remote access from the local device 10 to the remote device 104. In this case, the local device 10 may be a mobile terminal including a display device 101 and an information processing device 102. Furthermore, the remote device 104 is not limited to mobile terminals such as smartphones and laptops, but may also be information processing terminals such as servers and desktop personal computers. Furthermore, remote access in this embodiment refers to, for example, accessing the remote device 104 so as to operate it remotely based on an operation from the user to the local device 10.

[0023] (Display device configuration) The display device 101 includes a sound recording unit 210 , a sound output unit 211 , an imaging unit 212 , and a display unit 213 .

[0024] The recording unit 210 captures sounds around the display device 101 and the user's own voice as audio data, and outputs the data to the recording processing unit 222. Depending on the application, the recording unit 210 may be a recording device such as a directional microphone or a movable pin microphone. The recording unit 210 may also be equipped with a plurality of these recording devices.

[0025] The audio output unit 211 outputs audio in accordance with audio data output from the information processing device 102. The audio output unit 211 may be an audio output device such as a speaker, earphones, or headphones. The audio output unit 211 may also include a plurality of these audio output devices. The audio output device is connected to the display device 101 or the information processing device 102 via a wired connection. Alternatively, a device including both the recording unit 210 and the audio output unit 211 (for example, a wireless earphone with a microphone function) may communicate with the display device 101 or the information processing device 102.

[0026] The imaging unit 212 captures images of the real space continuously in time series and outputs the captured images of the real space (captured images) to the information processing device 102. The imaging unit 212 may include a stereo camera. The stereo camera has two cameras so as to be able to capture images of the real space existing in the line of sight of the user of the display device 101. The two cameras capture images on the opposite side (outside) of the user as viewed from the display device 101.

[0027] Furthermore, the imaging unit 212 may include a camera that captures an image of the user side (inside) as viewed from the display device 101 so as to be able to detect the user's expressions (body movements), such as the user's facial expressions and body language. Furthermore, the imaging unit 212 may also use a camera or the like provided externally to the display device 101 in order to detect the user's expressions. The external camera may be, for example, a camera fixed to a tripod or a wall of a building. In this case, the external camera may transmit the captured image to the information processing device 102 via the display device 101, or may transmit the captured image directly to the information processing device 102. The captured image is stored in the data storage unit 225 by the control unit 220.

[0028] In this way, the imaging unit 212 is a camera for detecting the user's expression. However, the configuration for detecting the user's expression is not limited to this, and other configurations capable of detecting the user's expression may be used instead of the imaging unit 212. For example, the imaging unit 212 may have a sensor and a detection unit instead of a camera. Multiple sensors may be arranged on the user's body, including the face, hands, and feet, and the detection unit may detect the user's expression based on values ​​acquired by the sensors.

[0029] The display unit 213 displays the MR image output from the information processing device 102. The display unit 213 may include two displays arranged to correspond to the left and right eyes of the user, respectively. In this case, the display corresponding to the user's left eye displays the MR image for the left eye, and the display corresponding to the user's right eye displays the MR image for the right eye.

[0030] (Configuration of information processing device) The information processing device 102 has a control unit 220, an audio output instruction unit 221, a sound recording processing unit 222, an image generation unit 223, an image synthesis unit 224, a data storage unit 225, a remote access processing unit 226, an operation processing unit 227, a vibration instruction unit 228, and a position and orientation calculation unit 229.

[0031] The control unit 220 controls the entire information processing device 102. The control unit 220 can transmit and receive signals and data to and from the image generation unit 223, data storage unit 225, remote access processing unit 226, position and orientation calculation unit 229, etc. The control unit 220 can also receive data and signals from the imaging unit 212, sound recording processing unit 222, operation processing unit 227, etc. The control unit 220 can also transmit signals and data to the audio output instruction unit 221, image synthesis unit 224, vibration instruction unit 228, etc.

[0032] The control unit 220 controls the position and orientation of a GUI (Graphical User Interface) such as a pointer displayed on the display unit 213, based on, for example, information on the position and orientation (position and orientation) of the operation device 103 stored in the data storage unit 225. The position of the GUI is indicated by, for example, three-dimensional coordinate information according to a Cartesian coordinate system of three axes, the X axis, the Y axis, and the Z axis. The orientation of the GUI corresponds to the orientation of the GUI in a three-dimensional virtual space. However, the three-dimensional coordinate information is not limited to information according to a Cartesian coordinate system of three axes, and may be information according to, for example, polar coordinates. It may also be three-dimensional coordinate information according to a reference system. Here, consider the case where the GUI is a virtual ray emitted from the tip of a user's finger, for example. When the ray indicates a certain point, the position of the GUI is, for example, the start position or end position of the ray. Furthermore, when the GUI is a ray, the orientation of the GUI corresponds to, for example, the direction in which the ray extends in a three-dimensional virtual space.

[0033] The control unit 220 transmits a signal of an instruction related to the direct operation suppression control to the remote access processing unit 243. The instruction related to the direct operation suppression control may be, for example, an instruction related to enabling or disabling direct operation of the remote device 104. Furthermore, the instruction related to the direct operation suppression control may be, for example, an instruction related to starting or ending the direct operation suppression control.

[0034] Here, direct operation suppression control (suppression control) will be described. In the first embodiment, when the control unit 220 is performing remote access, only the user can directly operate the remote device 104. Therefore, during remote access, the control unit 220 controls to suppress direct operation of the remote device 104 by others other than the user, or to prevent others from directly operating the remote device 104. In the first embodiment, such control by the control unit 220 is referred to as "direct operation suppression control."

[0035] The audio output instruction unit 221 instructs the audio output unit 211 to perform audio output. For example, the control unit 220 acquires multiple pieces of sound source data stored in the data storage unit 225 and synthesizes the multiple pieces of sound source data as necessary. In this way, the control unit 220 generates sound source data that the user can hear and passes it to the audio output instruction unit 221. In this case, the audio output instruction unit 221 instructs the audio output unit 211 to perform audio output using the sound source data, based on an instruction from the control unit 220.

[0036] The recording processing unit 222 processes the voice data acquired from the recording unit 210. The recording processing unit 222 transmits the user's voice data acquired from the recording unit 210 to the control unit 220. The recording processing unit 222 also analyzes the user's voice data acquired from the recording unit 210. In this way, the control unit 220 may perform various controls in accordance with the voice commands uttered by the user obtained from the analysis of the voice data.

[0037] The image generation unit 223 constructs a virtual space based on the virtual space data stored in the data storage unit 225. The virtual space data includes data relating to each virtual object that constitutes the virtual space, data relating to a light source that illuminates the virtual space, and information about the user's appearance.

[0038] The image generation unit 223 acquires information about the position and orientation of the imaging unit 212 calculated by the position and orientation calculation unit 229 from the data storage unit 225 via the control unit 220. The image generation unit 223 also acquires information about the position and orientation of the GUI controlled by the control unit 220 from the data storage unit 225 via the control unit 220. The image generation unit 223 generates an image of the virtual space (virtual space image) according to the position and orientation of the imaging unit 212.

[0039] The user's appearance information in the first embodiment is expressed in a virtual space constructed by the image generation unit 223 according to instructions from the control unit 220. The user's appearance information may be information that faithfully reproduces the appearance of the real user, or may be information that has been processed based on the appearance of the real user. The user's appearance information may be changeable, for example, by changing the skin color including skin beautification, changing the appearance exemplified by the color of the hair, eyelashes, eyebrows and eyes, changing the clothing, changing the height and proportions, etc. Furthermore, the user's appearance information may be changed by creating a 3D avatar or the like that is not based on the appearance of the real user. It may also be information.

[0040] The image synthesis unit 224 synthesizes the image of the virtual space generated by the image generation unit 223 with the image of the real space captured by the imaging unit 212 to generate an MR image. At this time, the image of the virtual space generated by the image generation unit 223 may be an image representing the entire virtual space or an image representing a part of the virtual space. Furthermore, the image synthesis unit 224 may generate an MR image by performing an affine transformation on the image, or may generate an MR image by assigning the image to a parametric curved surface.

[0041] The image synthesis unit 224 outputs the generated MR image to the display unit 213. The image synthesis unit 224 is not limited to a configuration that outputs the MR image to the display unit 213. For example, the image synthesis unit 224 may transmit the MR image to an external device. Furthermore, the image synthesis unit 224 may store the MR image as a 3D video in a storage medium so that it can be later displayed on the display unit 213. For example, if the display device 101 is an optical see-through HMD, the user can see the real space even without displaying the virtual space on the display unit 213. For this reason, an image displayed on the remote device 104 may be superimposed on the external monitor and the recorded data in a picture-in-picture manner. If the user's operation is not impaired even if the MR image is not displayed on the display unit 213, the image synthesis unit 224 may output the MR image to a unit other than the display unit 213. Furthermore, the MR image generated by the image synthesis unit 224 may be output simultaneously to the "display unit 213" and "display unit other than the display unit 213."

[0042] The data storage unit 225 can store various information for operating the information processing device 102. The data storage unit 225 includes at least one of storage media such as a RAM (Random Access Memory) and a hard disk drive device. In addition to the information described above as information to be stored in the data storage unit 225, the data storage unit 225 can also store information such as programs related to the operation of the information processing system 1. The data storage unit 225 can store sound source data that can be heard by the user (for example, synthesized voices and notification sounds).

[0043] The remote access processing unit 226 performs processing based on instructions from the control unit 220. The remote access processing unit 226 controls the remote access processing unit 243, for example, so that the information processing device 102 can access the remote device 104. The remote access processing unit 226 communicates with the remote access processing unit 243 of the remote device 104, which is the remote access destination, thereby realizing remote access between the information processing device 102 and the remote device 104. During remote access, the information processing device 102 generates an image to be displayed on the display unit 213 (hereinafter referred to as a "display image"). The display image may be generated by the remote access processing unit 226. At this time, the remote access processing unit 226 may receive information stored in the remote device 104 via the remote access processing unit 243.

[0044] The remote access processing unit 226 may generate a display image based on information received from the remote device 104. For example, a template is stored in advance in the data storage unit 225, and the control unit 220 reads out the template. The remote access processing unit 226 may process the template based on information received from the remote access processing unit 243 to generate a display image. The generated display image may be composited with another image by the image composition unit 224 and displayed to the user as a remote image by the display unit 213. The display image may also be a virtual object corresponding to the remote device 104 generated in a virtual space. The user may be able to control the remote device 104 by controlling the virtual object in the virtual space displayed on the display unit 213.

[0045] The remote access processing unit 226 also performs processing related to the termination of remote access based on instructions from the control unit 220. When the remote access is to be terminated, the remote access processing unit 226 transmits an instruction to terminate the remote access to the remote access processing unit 243. In this way, the remote access processing unit 226 terminates the remote access from the local device 10 to the remote device 104, and also hides the remote screen being displayed on the display unit 213.

[0046] However, the timing of hiding or terminating remote access does not matter. In some cases, hiding and terminating remote access may be performed simultaneously through parallel processing. Furthermore, the instruction to terminate remote access may also serve as an instruction regarding the suppression control of the remote device 104. This is because the remote device 104 needs to release the suppression control when the remote access is terminated. Note that the remote access processing unit 226 does not need to specifically instruct the remote device 104 to release the suppression of direct operations when the remote access is terminated. For example, if the remote device 104 performs processing to automatically release the suppression of direct operations when the remote access is terminated, the remote access processing unit 226 does not need to send a specific instruction signal regarding the termination of the suppression control.

[0047] The operation processing unit 227 receives operation information related to an operation performed by the user on the operation instruction unit 231. Based on the operation information received by the operation processing unit 227, the control unit 220 instructs the image generation unit 223 to construct a virtual space that reflects the operation related to the virtual object, as necessary. In this way, the operation processing unit 227 realizes control of the virtual object corresponding to the operation performed by the user. Note that the virtual object controlled by the user does not have to be a single object, but may be, for example, a group consisting of multiple objects. Also, for example, when an operation related to confirmation or cancellation is performed, the control unit 220 performs control related to confirmation or cancellation based on the user's operation.

[0048] The operation processing unit 227 changes data related to each virtual object stored in the data storage unit 225 based on operation information acquired from the operation instruction unit 231, and overwrites the data in the data storage unit 225. The operation processing unit 227 transmits data related to each virtual object and information on the position and orientation of the imaging unit 212 to the image generation unit 223. The operation processing unit 227 then controls the virtual objects by causing the image generation unit 223 to construct a virtual space that takes into account the position and orientation of the imaging unit 212. In this case, the virtual object may be an object related to the start and end of remote access, or may be a virtual object corresponding to the remote device 104 that is currently remotely accessing the virtual object. For example, the image generation unit 223 generates a virtual object of the remote device 104 based on data received from the remote access processing unit 226 via the control unit 220. However, if the data received by the control unit 220 is the virtual object itself, the image generation unit 223 does not need to generate a virtual object.

[0049] The vibration instruction unit 228 transmits a signal of a vibration instruction (an instruction requesting vibration) to the vibration unit 230 based on an instruction from the control unit 220. The vibration unit 230 vibrates in response to this signal, and therefore the vibration instruction unit 228 can give the user a vibration sensation.

[0050] The position and orientation calculation unit 229 calculates (acquires) the position and orientation (at least one of the position and orientation) of the image capture unit 212 in the world coordinate system. The position and orientation calculation unit 229 extracts markers assigned to the world coordinate system from the image of real space captured by the image capture unit 212. Then, the position and orientation calculation unit 229 calculates the position and orientation of the image capture unit 212 in the world coordinate system based on the position and orientation of the extracted markers. Here, the position and orientation calculation unit 229 transfers information indicating the calculated position and orientation of the image capture unit 212 (position and orientation information) to the digital camera via the control unit 220. The data is stored in the data storage unit 225.

[0051] The position and orientation calculation unit 229 may calculate the position and orientation of the operation device 103 in the world coordinate system using the position and orientation information of the operation device 103 acquired from the position and orientation calculation unit 232. Note that the position and orientation of the operation device 103 in the world coordinate system may be calculated using, for example, an image of real space captured by the imaging unit 212. The position and orientation calculation unit 229 may store the calculated position and orientation information of the operation device 103 in the data storage unit 225 via the control unit 220. In this case, the control unit 220 may control the position of the GUI based on the position and orientation information of the imaging unit 212 stored in the data storage unit 225. Furthermore, the control unit 220 may control the position of the GUI based on either or both of the position and orientation information of the operation device 103 and the position and orientation information of the imaging unit 212.

[0052] Note that detailed description will be omitted as it is a well-known technique for generating an image in a virtual space in accordance with information on the position and orientation of the image capture unit 212. Furthermore, the information processing device 102 does not need to include the position and orientation calculation unit 229 when the position and orientation of the image capture unit 212 in the world coordinate system is calculated by the control unit 220.

[0053] (Configuration of the operation device) The operation device 103 includes a vibration unit 230 , an operation instruction unit 231 , and a position and orientation calculation unit 232 .

[0054] The vibration unit 230 vibrates to give the user a vibration sensation in accordance with a vibration instruction from the vibration instruction unit 228. The vibration unit 230 has a vibration element with a short response time to give the vibration sensation. For example, a piezoelectric element can be used as the vibration element. The vibration unit 230 may also have other vibration elements such as a linear vibrator (LRA: Linear Resonant Actuator) or an eccentric rotating mass (ERM: Eccentric Rotating Mass).

[0055] The operation instruction unit 231 outputs values ​​according to the user's operation (such as a value according to the tilt direction of the stick and a value indicating the pressed state of a button) as operation information to the operation processing unit 227. Note that the operation referred to here may be an operation to instruct selection, determination, or movement of a virtual object. Furthermore, the movement of a virtual object is not limited to simple translation, and may include various affine transformations such as rotation, enlargement, and reduction.

[0056] The operation instruction unit 231 may have multiple buttons or may incorporate an optical trackpad (OTP). When the operation device 103 incorporates an OTP, the user can place a finger on the OTP and rub it in any direction to move the pointer to a desired item. The user can then perform a confirmation action to confirm the selection of a virtual object such as a menu item by pressing a button on the OTP. The user can also point to another location while keeping the virtual object selected. While selecting a virtual object, the user can also move the virtual object by performing a predetermined operation, such as twisting their arm or finger.

[0057] The operation device 103 may not have the operation instruction unit 231. In this case, the control unit 220 may recognize the user's hand movement (gesture) from the image of the real space captured by the imaging unit 212, and store information about the recognized gesture in the data storage unit 225.

[0058] The position and orientation calculation unit 232 calculates the position and orientation (position and The position and orientation calculation unit 232 calculates (acquires) the position and orientation (at least one of the orientations). Then, the position and orientation calculation unit 232 outputs information on the calculated position and orientation to the operation processing unit 227. The position and orientation calculation unit 232 has, for example, a sensor (such as an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor) for calculating the position and orientation of the operation device 103. Note that the position and orientation calculation unit 232 may have multiple sensors.

[0059] The position and orientation calculation unit 232 may have an imaging unit and an imaging processing unit, and may calculate the position and orientation through processing by the imaging unit and the imaging processing unit. Specifically, the imaging processing unit may calculate the position and orientation by using SLAM (Simultaneous Localization and Mapping) based on feature points of the captured image acquired by the imaging unit. The position and orientation calculation unit 232 may calculate the position and orientation in conjunction with an optical sensor installed in real space.

[0060] Furthermore, the control unit 220 may control the position of the GUI based on the information about the position and orientation of the operation device 103 acquired by the position and orientation calculation unit 232.

[0061] Note that when the position and orientation calculation unit 232 calculates the position and orientation, an error may occur between the calculated position and orientation of the operation device 103 and the actual position and orientation. For example, in a method of calculating the position and orientation using a combination of an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor, errors in the sensor values ​​acquired by each sensor may accumulate, resulting in a calculated value that includes an error compared to the actual position and orientation. Also, in a method of calculating the position and orientation using an optical sensor installed in real space, the optical sensor may be blocked by another real object, resulting in a calculated value that includes an error compared to the actual position and orientation.

[0062] As described above, there are cases where the position and orientation calculation unit 232 cannot correctly calculate the position and orientation of the operation device 103. Therefore, the position and orientation calculation unit 232 may extract a marker attached to the operation device 103 from an image of real space captured by the imaging unit 212, and calculate the position and orientation of the operation device 103 based on the position and orientation of the marker. By calculating the position and orientation of the operation device 103 based on the position and orientation of the marker in this way, the accuracy of the calculation result can be improved. In this case, the position and orientation calculation unit 232 may use all or part of the calculation result of the position and orientation acquired from the position and orientation calculation unit 229.

[0063] The method of calculating the position and orientation by the position and orientation calculation unit 232 is not limited to a method using markers, and may be a method of calculating the position and orientation by SLAM processing. Also, an orientation sensor unit (not shown) of the display device 101 may detect the orientation. The orientation sensor unit has an inertial measurement unit (IMU) and outputs information on the orientation of the display device 101 (orientation information) to the information processing device 102.

[0064] Furthermore, when the operation device 103 does not appear in the image of real space captured by the imaging unit 212, the position and orientation calculation unit 229 may calculate the position and orientation of the operation device 103 based on the value acquired from the position and orientation calculation unit 232. At this time, the position and orientation calculation unit 229 may calculate the position and orientation of the operation device 103 based on the value acquired from the position and orientation calculation unit 232 and other information. For example, when the operation device 103 does not appear in the image of real space, if the operation device 103 includes an acceleration sensor or the like, the position and orientation calculation unit 229 may calculate the position and orientation of the operation device 103 based on the detection result of the acceleration sensor or the like.

[0065] However, the position and orientation of the operation device 103 does not have to be calculated based on various sensors. For example, the position and orientation calculation unit 229 adds the amount of movement of the operation device 103 from a past point in time, calculated from acceleration, to position information of the operation device 103 based on an image in which the operation device 103 appears among images in real space taken at a past point in time. In this way, the position and orientation calculation unit 229 Alternatively, the position and orientation calculation unit 229 may calculate the current position and orientation of the operation device 103. If the operation device 103 is not captured in the image in real space, the position and orientation calculation unit 229 may detect only the orientation of the operation device 103.

[0066] As described above, the position and orientation calculation unit 232 may output information (values ​​output from the sensor) or images for calculating the position and orientation of the operation device 103 to the information processing device 102. In this case, the position and orientation calculation unit 229 may calculate the position and orientation of the operation device 103 based on the information or images acquired from the position and orientation calculation unit 232. In this case, the operation device 103 does not need to have the position and orientation calculation unit 232.

[0067] The position and orientation calculation unit 232 may calculate the values ​​of either the position component or the orientation component, rather than calculating the values ​​of both.

[0068] (Remote Device Configuration) The remote device 104 includes an audio output unit 241 , a display unit 242 , a remote access processing unit 243 , a control unit 244 , a data storage unit 245 , a direct operation unit 246 , and a vibration instruction unit 247 .

[0069] The audio output unit 241 outputs audio based on instructions from the control unit 244 .

[0070] The display unit 242 performs display based on instructions from the control unit 244 .

[0071] The remote access processor 243 performs "processing related to the start and end of remote access" and "processing related to suppression control" based on information received from the information processing device 102. The information that the remote access processor 243 receives from the information processing device 102 is, for example, a notification of the start and end of remote access. However, the notification of the start and end of remote access may not be received from the information processing device 102 but from another device (such as the device 105). The processing related to the start and end of remote access itself is a well-known technique, and therefore will not be described here.

[0072] When remote access starts, the remote access processing unit 243 acquires screen information (information about the image displayed on the display unit 242) of the remote device 104 via the control unit 244, and transmits the screen information to the information processing device 102. When remote access ends, the remote access processing unit 243 stops acquiring the screen information of the remote device 104 from the control unit 244. The screen information of the remote device 104 may be image data such as bitmap and JPEG, or may be image data containing only difference information from a template screen or data in other formats.

[0073] The control unit 244 controls the entire remote device 104. The control unit 244 can transmit and receive signals and data to and from the remote access processing unit 243 and the like. The control unit 244 can receive signals from the direct operation unit 246 and the like. The control unit 244 can transmit signals and data to the audio output unit 241, the display unit 242, and the like.

[0074] When direct operation on the remote device 104 is enabled, the control unit 244 performs control based on an operation instruction signal from the direct operation unit 246. In this case, the control unit 244 instructs the display unit 242 to perform display output, or instructs the audio output unit 241 to perform audio output.

[0075] The control unit 244 issues a predetermined notification to prevent others from operating the remote device 104 in response to a request from the remote access processing unit 243 based on an instruction from the control unit 220. Control may be performed to perform the above.

[0076] The control unit 244 may transmit a "signal instructing to give a predetermined notification by voice" to the audio output unit 241. The control unit 244 may transmit a "signal instructing to give a predetermined notification by generating a specific display item related to the predetermined notification and displaying the specific display item" to the display unit 242.

[0077] When the remote device 104 is connected to the device 105, the control unit 244 may send to the vibration instruction unit 247 a signal instructing the vibration unit 250 to vibrate to give a predetermined notification.

[0078] If the remote device 104 has an audio instruction unit and the device 105 has an audio output unit, the control unit 244 may cause the audio output unit of the device 105 to make a predetermined audio notification via the audio instruction unit of the remote device 104.

[0079] If the remote device 104 has a display instruction unit and the device 105 has a display unit, the control unit 244 may cause the display unit of the device 105 to display a specific display item related to a specified notification via the display instruction unit of the remote device 104.

[0080] However, the control for issuing a predetermined notification is not limited to these, and any control for preventing others from operating the remote device 104 may be used.

[0081] The control unit 244 may perform processing to disable an operation directly input to the operation unit 246 in response to a request from the remote access processing unit 243 based on an instruction from the control unit 220. The control unit 244 may also perform processing to enable an operation directly input to the operation unit 246 in response to a request from the remote access processing unit 243 based on an instruction from the control unit 220.

[0082] Here, the enabling and disabling of operations will be explained. "Enabling an operation" means that when a user performs an operation on the direct operation unit 246, the control unit 244 controls the remote device 104 to a state in which processing (control) corresponding to the operation is performed. "Disabling an operation" means that even when a user performs an operation on the direct operation unit 246, the control unit 244 controls the remote device 104 to a state in which processing (control) corresponding to the operation is not performed.

[0083] The control unit 244 may transmit a signal of a direct operation control instruction (control instruction) to the direct operation unit 246. The control instruction is an instruction to the direct operation unit 246 to perform processing to enable or disable direct operations. The direct operation unit 246 performs processing to enable or disable direct operations based on the signal from the control unit 244. In particular, when disabling direct operations, operation information is no longer transmitted from the direct operation unit 246, so that it is expected that the amount of communication within the remote device 104 will be reduced.

[0084] However, if processing is performed to disable operations on direct operation unit 246, operation information will no longer be transmitted from direct operation unit 246, and therefore, when an attempt is made to enable an operation, components other than direct operation unit 246 will be unable to perform processing in response to the direct operation. Therefore, direct operation unit 246 itself needs to have the function of enabling and disabling direct operations.

[0085] The signal that the control unit 244 receives from the remote access processing unit 243 may be a signal of abstract instructions regarding suppression control, or a signal of specific instructions regarding suppression control. If the signal regarding suppression control is a signal of abstract instructions, the control unit 244 performs specific suppression control at its own discretion. At this time, the control unit 244 uses the data stored in the data storage unit 245. The suppression control may be performed in accordance with the default settings for direct manipulation that are set in the control panel. Specific instructions for the suppression control and abstract instructions for the suppression control will be described in detail later.

[0086] The data storage unit 245 can store various information for operating the remote device 104. The data storage unit 245 includes at least one of storage media such as a RAM (Random Access Memory) and a hard disk drive device. The data storage unit 245 can store settings related to direct operations that can be read by the control unit 244. However, the remote device 104 does not necessarily have to have the data storage unit 245.

[0087] The direct operation unit 246 is used when the user operates the remote device 104. For example, if the remote device 104 is a personal computer, the direct operation unit 246 corresponds to input devices such as a mouse and a keyboard. Even if the remote device 104 is a device other than a personal computer, the direct operation unit 246 is not particularly limited as long as it is an input device that can directly operate the remote device 104.

[0088] The vibration instruction unit 247 transmits a vibration instruction signal to the vibration unit 250 of the device 105 based on a signal from the control unit 244. Communication between the vibration instruction unit 247 and the device 105 may be established in advance. When transmitting the vibration instruction signal, the vibration instruction unit 247 may dynamically establish communication and then transmit the vibration instruction signal.

[0089] Typically, the vibration instruction unit 247 notifies the control unit 244 in advance that the device 105 is present, and based on that information, the control unit 244 transmits a vibration instruction signal to the vibration unit 250 via the vibration instruction unit 247. For this reason, it is assumed that communication between the control unit 244 and the vibration unit 250 is first established or communication possibility is confirmed. However, if communication establishment can be performed in a short time that does not affect usability, communication may be established dynamically when transmitting a vibration instruction signal as described above. However, as long as a vibration instruction signal can be transmitted from the control unit 244 to the vibration unit 250, the method and timing of establishing communication are not limited.

[0090] (Device configuration) The device 105 has a vibrating part 250 .

[0091] The vibration unit 250 vibrates based on a signal from the vibration instruction unit 247 of the remote device 104 .

[0092] Furthermore, device 105 may have a display unit (not shown). When device 105 has a display unit, remote device 104 may have a display instruction unit (not shown). The display unit may display an image to be shown to the user (display output to the user) based on a signal from the display instruction unit of remote device 104.

[0093] Furthermore, device 105 may have an audio output unit (not shown). If device 105 has an audio output unit, remote device 104 may have an audio instruction unit (not shown). The audio output unit may output audio to be heard by the user (audio output to the user) based on a signal from the audio output instruction unit of remote device 104.

[0094] The device 105 operates based on an instruction from the remote device 104, but may also operate based on an instruction signal from a terminal other than the remote device 104. For example, the information processing device 102 may transmit an instruction signal to the device 105, or another terminal (not shown) may transmit an instruction signal. may transmit an instruction signal to the device 105. When a terminal equipped with a sensor such as a camera determines that direct manipulation should be suppressed, the terminal may transmit an instruction signal to the device 105, causing the device 105 to operate.

[0095] (Direct Operation Inhibition Control Instructions) The following describes direct operation suppression control (suppression control). An instruction related to suppression control is, for example, an instruction to output sound from the sound output unit 241 of the remote device 104. An instruction related to suppression control may also be an instruction to display a specific display item related to a predetermined notification on the display unit 242. An instruction related to suppression control may also be an instruction to vibrate the vibration unit 250 of the device 105 via the remote device 104. An instruction related to suppression control may also be an instruction to display output on a display unit (not shown) of the device 105, or an instruction to output sound from the sound output unit.

[0096] In the first embodiment, an abstract instruction regarding inhibitory control is an instruction as to whether or not to perform inhibitory control. Specific instructions regarding inhibitory control include instructions regarding individual control of the method of notifying others. Specific instructions regarding inhibitory control include instructions for setting whether to enable or disable direct operation of the remote device 104 by others. In other words, even when inhibitory control is performed, others may actually be able to operate the device simply by issuing a predetermined notification, or others may actually be unable to operate the device after issuing a predetermined notification. Furthermore, others may actually be unable to operate the device without issuing a predetermined notification.

[0097] Now consider a case where the information processing device 102 and the remote device 104 communicate with each other regarding instructions related to power reduction. When the signal transmitted from the control unit 220 is a specific instruction related to power reduction, the remote access processor 226 receives the signal transmitted from the control unit 220 and transmits it to the remote access processor 243. Based on the received signal, the remote access processor 243 transmits an instruction signal related to power reduction to the control unit 244.

[0098] If the instruction transmitted from the information processing device 102 to the remote device 104 is an abstract instruction related to suppression control, the remote device 104 determines the specific suppression control (such as the method of notifying others). For example, if the signal transmitted from the control unit 220 is an abstract instruction related to suppression control, the remote access processing unit 243 may determine the specific control related to suppression control. In this case, the remote access processing unit 243 may transmit a signal containing the specific instruction related to suppression control to the control unit 244 based on the determination result. Alternatively, the signal transmitted from the remote access processing unit 243 to the control unit 244 may include only the abstract instruction related to suppression control, and the control unit 244 may determine the specific suppression control.

[0099] (MR space) Here, an example of an MR image generated by the image synthesis unit 224 is shown. Fig. 3 is a diagram showing an MR image displayed on the display unit 213, which is an image when superimposed (overlaid) on a remote device. Fig. 4 is a diagram showing an MR image displayed on the display unit 213, which is an image when the screen of the remote device is enlarged and displayed. The MR image generated by the image synthesis unit 224 includes an MR space 301 and a virtual object 202.

[0100] The MR space 301 is generated by the image generation unit 223 based on an image captured by the imaging unit 212. The image generation unit 223 generates the virtual object 202 based on information acquired from the remote device 104 (control unit 244). The remote device 104 is placed in the MR space 301, and the virtual object 302 is displayed on the screen of the remote device 104. It is configured as a corresponding virtual object.

[0101] In the state shown in FIG. 3 , a virtual object 302 is placed in the MR space 301 so as to be superimposed on the screen of the remote device 104. This allows the user to control the remote device 104 through the MR space 301 in the same way as in the real space. In this case, the user can check the screen of the remote device 104 via the display unit 213. However, in the remote device 104 in the real space, the display unit 242 may be set to hidden, or a specific display item related to a predetermined notification may be displayed on the display unit 242. In other words, the display unit 242 in the real space may be displayed in a way that makes it difficult for others to directly operate the remote device 104.

[0102] 4, a virtual object 302 is displayed in the MR space 301 at an enlarged scale larger than the screen of the remote device 104. The screen of the remote device 104 is blacked out. This allows the user to check the screen of the remote device 104 by remote access, without directly operating the remote device 104, when attempting to control the remote device 104 through the MR space 301.

[0103] However, the display method of the virtual object 302 is not limited to this. For example, the image synthesis unit 224 may place the virtual object 302 in the MR space 301 so as to be superimposed on the remote device 104. In this case, the image synthesis unit 224 may display the virtual object 302 at a position adjusted so as to face the user. Furthermore, the image synthesis unit 224 may fix the display direction of the virtual object 302 depending on the content of the MR space 301. Furthermore, the virtual object 302 does not need to be generated by the image generation unit 223. For example, the display unit 213 may display an image displayed on the screen of the remote device 104 acquired from the imaging unit 212, instead of the virtual object 302.

[0104] (Remote device direct operation suppression control) 5 is a flowchart showing suppression control over the remote device 104. In the first embodiment, the information processing apparatus 102 can be configured to "display a setting screen such as that shown in FIG. 6A on the display unit 213 to suppress direct operation during remote access." Here, a case where the setting to suppress direct operation during remote access is effective is considered. In the following, control will be described for this case in which the user directly operates the remote device 104 while suppressing direct operation of the remote device 104 by others other than the user during remote access.

[0105] In step S501, the control unit 220 determines whether or not the user has instructed the remote device 104 to start remote access. If it is determined that the user has instructed the start of remote access, the process proceeds to step S502. If it is determined that the user has not instructed the start of remote access, the process of step S501 is repeated.

[0106] For example, the user may instruct the start of remote access by selecting a virtual object corresponding to the remote device 104 displayed on the display unit 213. The control unit 220 acquires information acquired by the operation processing unit 227 (information acquired based on an operation by the user on the operation instruction unit 231).

[0107] When the user issues an instruction to start remote access, the control unit 220 transmits a signal instructing the remote access to the remote access processing unit 226, and the remote access processing unit 226 communicates with the remote access processing unit 243. The remote access processor 243 requests the control unit 244 for screen information of the remote device 104 and receives the screen information from the control unit 244. The remote access processor 243 then starts transmitting the screen information of the remote device 104 to the remote access processor 226.

[0108] However, the method of instructing the start of remote access is not particularly limited. For example, when the user's line of sight is detected using the imaging unit 212, the user may instruct the start of remote access by gazing at a virtual object corresponding to the remote device 104 for a predetermined period of time or more. In this case, the control unit 220 controls the information processing device 102 based on the user's line of sight information detected by the imaging unit 212, without via the operation processing unit 227. In the first embodiment, an example has been shown in which the start of remote access is instructed using a virtual object corresponding to the remote device 104, but an actual image of the remote device 104 (an image captured of the remote device 104) may also be used.

[0109] In step S502, control unit 220 transmits a signal from remote access processor 226 to remote access processor 243 as an instruction regarding restriction control.

[0110] When the remote access processor 243 receives an instruction signal for suppression control, it transmits the instruction signal for suppression control to the controller 244. When the controller 244 receives the instruction signal for suppression control, it transmits a control instruction signal for disabling direct manipulation to the direct manipulation unit 246. Alternatively, the controller 244 transmits a signal for issuing a predetermined notification based on a request from the remote access processor 243 to the audio output unit 241, the display unit 242, and the like of the remote device 104. When an instruction for suppression control is issued to the remote device 104, a notification that direct manipulation is not possible is issued to the display unit 242, and the like. While the control within the remote device 104 has been described so far, the controller 244 may also transmit an instruction signal for suppression control to the device 105. For example, the controller 244 may suppress direct manipulation by vibrating the vibration unit 250 via the vibration instruction unit 247.

[0111] Here, the setting of the control method for direct manipulation will be described. FIG. 6 shows a setting screen for the control method for direct manipulation. The user may be able to set the restriction and prohibition of direct manipulation by displaying setting screens such as those shown in FIGS. 6(a) and 6(b) on the display unit 213 as the settings for the above-mentioned instructions for suppression control and control instructions. The control unit 220 may transmit a setting instruction signal from the remote access processing unit 226 to the remote access processing unit 243 based on an operation (input) performed by the user on the operation instruction unit 231. Information regarding the method for suppressing direct manipulation input by the user is transmitted to the control unit 244, and the control unit 244 controls the remote device 104 based on the received information. While the example shown in FIGS. 6(a) and 6(b) shows the user setting the setting screen from the display unit 213, a similar setting screen may also be displayed on the remote device 104. In this case, the control unit 244 performs control according to the setting of the remote device 104.

[0112] 7(a) and 7(b) show screens for detailed settings of the control method for suppressing direct manipulation. When suppressing direct manipulation is selected in the settings for the control method for suppressing direct manipulation described above, a settings screen such as that shown in FIG. 7(a) is displayed on the display unit 213. This may allow the user to set detailed control of suppressing direct manipulation. For example, when the user selects "enable control for suppressing direct manipulation," each notification method, such as audio notification, vibration notification, and display notification, may be individually enabled or disabled.

[0113] Also, on the detailed settings screen, you can enable direct operation of the remote device 104. It may be possible to select whether or not to provide notification. On the screen of FIG. 7(a), audio notification and notification on a display screen are selected as notification methods. Therefore, in order to prevent others from operating the remote device 104, a predetermined audio notification is output from the audio output unit 241 and a predetermined notification is provided by a specific display item displayed on the display unit 242. Also, on the screen of FIG. 7(a), direct operation is enabled. Therefore, on the remote device 104, audio and visual notifications are provided to prevent others from operating the remote device 104, but operation by others is still possible. However, detailed control is not limited to being set individually; for example, multiple instructions may be set in a single package. This may reduce communication traffic.

[0114] As shown in FIG. 7(a), the notification method and enabling / disabling direct operation may be selectable as separate items, or the user may be able to select detailed settings for suppressing direct operation from each item as shown in FIG. 7(b). In FIG. 7(b), the items of audio notification, notification on the display screen, and disabling direct operation are selected as detailed settings for suppressing direct operation. In other words, the screen in FIG. 7(b) is set so that audio and visual notifications are given to prevent others from operating the remote device 104, and direct operation by the user is also disabled.

[0115] In step S503, the control unit 220 determines whether or not the user is in a state where he or she can directly operate the remote device 104. If it is determined that the user is in a state where he or she can directly operate the remote device 104, the process proceeds to step S504. If it is determined that the user is not in a state where he or she can directly operate the remote device 104, the process proceeds to step S505.

[0116] For example, the control unit 220 determines that the user is in a state where he or she can directly operate the remote device 104 when the distance between the user and the remote device 104 is less than a threshold. Furthermore, the control unit 220 determines that the user is not in a state where he or she can directly operate the remote device 104 when the distance between the user and the remote device 104 is equal to or greater than a threshold. One method for detecting the distance between the user and the remote device 104 is to detect the distance between the user and the remote device 104 based on an image of the remote device 104 obtained from the imaging unit 212. Alternatively, an imaging unit (not shown) provided in the remote device 104 may acquire images of the user and the display device 101, etc., and the control unit 220 may detect the distance between the remote device 104 and the user. Alternatively, the control unit 220 may detect the distance between the user and the remote device 104 based on the communication strength between the devices, such as the communication strength between the remote access processing unit 226 and the remote access processing unit 243. As described above, the distance detection method is not particularly limited as long as the distance between the user and the remote device 104 can be detected. Furthermore, the method for determining whether the user is able to directly operate the remote device 104 is not limited to the above-described method. In step S503, if it is possible to determine whether the user is in a state where he or she can directly operate the remote device 104, a method that does not use distance may be used.

[0117] In step S504, the control unit 220 transmits a signal to release the suppression control of the remote device 104 in preparation for direct operation by the user. A signal instructing to release the suppression of direct operation (hereinafter referred to as a "release instruction") is transmitted from the remote access processing unit 226 to the remote access processing unit 243. Upon receiving the release instruction signal, the remote access processing unit 243 transmits a release instruction signal to the control unit 244. Upon receiving the release instruction signal, the control unit 244 performs control to release the suppression control.

[0118] Here, the cancellation of the suppression control will be described. For example, when a predetermined notification sound is output from the sound output unit 241, the suppression control is cancelled. The term "cancel suppression control" refers to stopping audio output. In addition, "cancel suppression control" refers to stopping the display from the display unit 242 when a predetermined notification image (display item) or the like has been displayed on the display unit 242. In addition, "cancel suppression control" refers to stopping the vibration output instruction signal to the vibration instruction unit 247 and stopping the vibration of the vibration unit 250 when a predetermined notification has been output by vibration from the vibration unit 250. When suppression control is released, control may be performed to stop the output instruction signal, or control may be performed to transmit a stop instruction signal. In addition, when direct manipulation is disabled, a signal instructing to enable direct manipulation may be transmitted to the direct manipulation unit 246.

[0119] In step S505, the control unit 220 performs processing to suppress direct operation or processing to disable direct operation so that others do not operate the remote device 104. As a result, the control unit 220 performs direct operation suppression control to prevent others from operating the remote device 104 when the user is not able to operate the remote device 104. The control unit 220 transmits a signal of instructions regarding the suppression control from the remote access processing unit 226 to the remote access processing unit 243, and thereafter transmits the signal in the same way as a release instruction signal.

[0120] In step S506, the control unit 220 determines whether or not the remote access to the remote device 104 has ended. If it is determined that the remote access to the remote device 104 has ended, the processing of this flowchart ends. If it is determined that the remote access to the remote device 104 has not ended, the process returns to step S504. However, the method for determining whether or not the remote access has ended is not particularly limited. For example, the remote access processing unit 226 may determine whether or not the remote access has ended. The remote access processing unit 226 may store status information indicating whether or not remote access is in progress in the data storage unit 225, and the control unit 220 may determine whether or not the remote access has ended based on the status information.

[0121] However, if the suppression control is already disabled at the start of the process of step S504, no particular process may be performed in step S504. Similarly, if the suppression control is already enabled at the start of the process of step S505, no particular process may be performed in step S505. The control unit 220 may determine whether to perform the processes of steps S504 and S505 based on the state of the suppression control (whether it is enabled or disabled). In this case, the information processing device 102, the remote device 104, or the like stores the state of the suppression control, and the control unit 220 or the control unit 244, or the like, reads the state of the suppression control. The control unit 220, the control unit 244, or the like may determine whether to perform the processes of steps S504 and S505 based on the read state of the suppression control. The state may be stored when the state is changed, as exemplified in steps S502, S504, and S505.

[0122] In the above, it has been described that the control unit 220 performs the determinations and processes in steps S501 to S505, but this is not limiting. For example, the control unit 244 may perform the determinations and controls in steps S501 to S505.

[0123] The suppression control described in the first embodiment is intended to prevent others from directly operating the remote device 104 while the user is remotely accessing the device. Therefore, the suppression control actually used may be any one of the suppression controls exemplified above, or a combination of multiple suppression controls. Furthermore, other suppression methods may be used as long as they can prevent others from directly operating the remote device 104. For example, a method that vibrates the device 105 Alternatively, device 105 may further restrain manipulation by display output, or may notify others by audio output that direct manipulation is not desired or is not possible.

[0124] According to the first embodiment, when remotely accessing the remote device 104, the information processing system 1, in principle, controls the remote device 104 so that it cannot be directly operated. Even in this case, if the user is able to directly operate the remote device 104, the information processing system 1 cancels the "control that prevents direct operation of the remote device 104." This allows the user to directly operate the remote device 104 during remote access, while making it difficult for others other than the user to directly operate the remote device. Therefore, when the remote device 104 is remotely accessible, it is possible to more appropriately control whether or not the remote device 104 can be directly operated.

[0125] (Embodiment 2) In the first embodiment, when the control unit 220 starts remote access, in step S502 the control unit 220 suppresses direct operation of the remote device 104. However, the control unit 220 does not have to suppress direct operation of the remote device 104 immediately after the start of remote access. Also, in the second embodiment, step S502 may be omitted from FIG. 5.

[0126] (Embodiment 3) In the first and second embodiments, in step S503, whether the user can directly operate the remote device 104 is determined based on whether the distance between the user and the remote device 104 is less than a predetermined threshold. In the third embodiment, when the remote device 104 is present in the user's field of view obtained from the imaging unit 212, the control unit 220 may determine that the remote device 104 is in a state where the user can directly operate it. When the remote device 104 is not present in the user's field of view obtained from the imaging unit 212, the control unit 220 may determine that the remote device 104 is not in a state where the user can directly operate it.

[0127] (Embodiment 4) In the first to third embodiments, the control unit 220 controls whether to suppress direct operation of the remote device 104 during remote access, depending on whether the user is able to directly operate the remote device 104. In the fourth embodiment, the control unit 220 may suppress direct operation of the remote device 104 only when it is determined that the distance between the other person and the remote device 104 is less than a predetermined threshold. In the fourth embodiment, the initial state at the start of remote access is a state in which direct operation of the remote device 104 is not suppressed.

[0128] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0129] The processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays). Programmable Gate Array (PGArray), CPLD (Complex Programmable Logic Device), etc.

[0130] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0131] (Other embodiments) The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0132] The disclosure of this embodiment includes the following configuration, method, program, and medium. (Configuration 1) The first device is an electronic device, A receiving means for receiving an operation from a user; a control means for controlling an operation of the second device in response to an operation from the user on the accepting means; and When controlling the operation of the second device in response to an operation from the user to the reception means, the control means performs a first control to suppress an operation on the second device that is different from the operation from the user to the reception means when the user is in a state where the user is unable to operate the second device. An electronic device characterized by: (Configuration 2) The state in which the user cannot operate the second device is a state in which the distance between the user and the second device is equal to or greater than a threshold. 2. The electronic device according to configuration 1. (Configuration 3) The state in which the user is unable to operate the second device is a state in which the second device is not present in the user's field of view. 2. The electronic device according to configuration 1. (Configuration 4) The control means prevents an operation on the second device that is different from an operation from the user on the reception means from being performed in the first control. 4. The electronic device according to any one of configurations 1 to 3. (Configuration 5) An electronic device that is a second device that communicates with a first device having a receiving means for receiving an operation from a user, and a control unit that performs a first control to suppress an operation on the second device, which is different from the operation on the receiving unit from the user, when the operation of the second device is controlled in response to an operation on the receiving unit from the user, in a state in which the user is unable to operate the second device. An electronic device characterized by: (Configuration 6) The state in which the user cannot operate the second device is a state in which the distance between the user and the second device is equal to or greater than a threshold. 6. The electronic device according to configuration 5. (Configuration 7) The state in which the user is unable to operate the second device is a state in which the second device is not present in the user's field of view. 6. The electronic device according to configuration 5. (Configuration 8) The control means prevents an operation on the second device that is different from an operation from the user on the reception means from being performed in the first control. 8. The electronic device according to any one of configurations 5 to 7. (Configuration 9) The first control is a process of preventing an operation on the second device from being used to control the operation of the second device, even if the operation is performed on the second device. 9. The electronic device according to any one of configurations 1 to 8. (Configuration 10) An electronic device according to any one of configurations 1 to 4, and the second device, the first control is to instruct the second device to issue a predetermined notification to prevent people other than the user from operating the second device; the second device has a notification means for issuing the predetermined notification in response to the first control; An information processing system comprising: (Configuration 11) An electronic device comprising the electronic device according to any one of configurations 5 to 8 and the first device, the first control is to issue a predetermined notification to prevent people other than the user from operating the second device; the second device has a notification means for issuing the predetermined notification in response to the first control; An information processing system comprising: (Configuration 12) The notification means is a first audio output means that issues the predetermined notification by emitting a sound. 12. The information processing system according to configuration 10 or 11. (Configuration 13) the notification means is a first display means that performs the predetermined notification by displaying a specific display item; 12. The information processing system according to configuration 10 or 11. (Configuration 14) the second device is connected to a third device; The notification means is a vibration instruction means for performing the predetermined notification by controlling the third device to vibrate. 12. The information processing system according to configuration 10 or 11. (Configuration 15) the second device is connected to a third device; The notification means is a voice output instruction means for performing the predetermined notification by controlling the third device to emit a voice. 12. The information processing system according to configuration 10 or 11. (Configuration 16) the second device is connected to a third device; The notification means is a display instruction means for performing the predetermined notification by controlling the third device to display a specific display item. 12. The information processing system according to configuration 10 or 11. (Method 1) A method for controlling an electronic device that is a first device, comprising: A reception step for receiving operations from the user; a control step of controlling an operation of the second device in response to the operation from the user accepted in the accepting step; and In the control step, when controlling the operation of the second device in response to the operation from the user accepted in the accepting step, a first control is performed to suppress an operation on the second device that is different from the operation from the user accepted in the accepting step when the user is in a state where the user is not able to operate the second device. A method for controlling an electronic device. (Method 2) A method for controlling an electronic device that is a second device communicating with a first device, comprising: an acquisition step of acquiring information on an operation from a user accepted by the first device; a control step of performing a first control to suppress an operation on the second device that is different from the operation from the user that is accepted by the first device, when the operation of the second device is controlled in accordance with the operation from the user that is accepted by the first device, in a state in which the user is not able to operate the second device; have A method for controlling an electronic device. (program) A program for causing a computer to execute each step of the control method. (medium) A computer-readable storage medium storing a program for causing a computer to execute each step of the control method. [Explanation of symbols]

[0133] 10 Local Devices 103 Operating device 104 Remote Devices 223 Control Unit

Claims

1. An electronic device that is a first device, A receiving means for receiving an operation from a user; a control unit configured to control an operation of the second device in response to an operation from the user performed on the reception unit; and When controlling the operation of the second device in response to an operation from the user to the reception means, the control means performs a first control to suppress an operation on the second device that is different from the operation from the user to the reception means when the user is in a state where the user is unable to operate the second device. An electronic device characterized by:

2. The state in which the user cannot operate the second device is a state in which the distance between the user and the second device is equal to or greater than a threshold.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

3. The state in which the user is unable to operate the second device is a state in which the second device is not present in the user's field of view.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

4. The control means, in the first control, prevents an operation on the second device that is different from an operation from the user on the accepting means.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

5. An electronic device that is a second device that communicates with a first device having a receiving means for receiving an operation from a user, and a control unit that performs a first control to suppress an operation on the second device that is different from the operation on the receiving unit from the user when the operation of the second device is controlled in response to an operation from the user on the receiving unit, in a state in which the user is unable to operate the second device. An electronic device characterized by:

6. The state in which the user cannot operate the second device is a state in which the distance between the user and the second device is equal to or greater than a threshold.

6. The electronic device according to claim 5,

7. The state in which the user is unable to operate the second device is a state in which the second device is not present in the user's field of view.

6. The electronic device according to claim 5,

8. The control means, in the first control, prevents an operation on the second device that is different from an operation from the user on the accepting means.

6. The electronic device according to claim 5,

9. The first control is a process of preventing an operation on the second device from being used to control the operation of the second device, even if the operation on the second device is performed.

9. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

10. An electronic device comprising: the electronic device according to claim 1; and the second device; the first control is to instruct the second device to issue a predetermined notification to prevent people other than the user from operating the second device; the second device has a notification means for issuing the predetermined notification in response to the first control. An information processing system comprising:

11. An electronic device comprising: the electronic device according to claim 5; and the first device; the first control is to issue a predetermined notification to prevent people other than the user from operating the second device; the second device has a notification means for issuing the predetermined notification in response to the first control. An information processing system comprising:

12. the notification means is a first audio output means that issues the predetermined notification by emitting a sound; 12. The information processing system according to claim 10 or 11.

13. the notification means is a first display means that performs the predetermined notification by displaying a specific display item; 12. The information processing system according to claim 10 or 11.

14. the second device is connected to a third device; The notification means is vibration instruction means for performing the predetermined notification by controlling the third device to vibrate.

12. The information processing system according to claim 10 or 11.

15. the second device is connected to a third device; The notification means is a voice output instruction means for performing the predetermined notification by controlling the third device to emit a voice.

12. The information processing system according to claim 10 or 11.

16. the second device is connected to a third device; The notification means is a display instruction means for performing the predetermined notification by controlling the third device to display a specific display item.

12. The information processing system according to claim 10 or 11.

17. A method for controlling an electronic device that is a first device, comprising: A reception step for receiving operations from the user; a control step of controlling an operation of the second device in response to the operation from the user accepted in the accepting step; and In the control step, when controlling the operation of the second device in response to the operation from the user accepted in the accepting step, a first control is performed to suppress an operation on the second device that is different from the operation from the user accepted in the accepting step when the user is in a state where the user is not able to operate the second device. A method for controlling an electronic device.

18. A method for controlling an electronic device that is a second device communicating with a first device, comprising: an acquisition step of acquiring information on an operation from a user accepted by the first device; a control step of performing a first control to suppress an operation on the second device that is different from the operation from the user that is accepted by the first device, when the operation of the second device is controlled in accordance with the operation from the user that is accepted by the first device, in a state in which the user is unable to operate the second device; have A method for controlling an electronic device.

19. A program for causing a computer to execute each step of the control method according to claim 17 or 18.

20. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 17 or 18.

Images