Electronic apparatus, information processing system, and method for controlling electronic apparatus

Abstract

This electronic apparatus, which is a first device, is characterized by comprising: a reception means for receiving an operation from a user; and a control means for controlling an action of a second device in response to the operation from the user received by the reception means. The electronic apparatus is also characterized in that, when the control means controls the action of the second device in response to an operation from the user received by the reception means, in a state in which the user is not in a position where the user can operate an operation member provided on the second device, the control means performs a first control for preventing an operation to the second device which is different from the operation from the user received by the reception means.

Description

Electronic device, information processing system, and method for controlling electronic device

[0001] The present invention relates to an electronic device, an information processing system, and a method for controlling an electronic device, and more particularly to an information processing system capable of controlling a remotely accessed device.

[0002] Conventionally, there are technologies related to direct operation of a remote device that is the target of remote access (operations for controlling the remote device that are detected by the remote device itself). Such technologies include a technology that displays a lock screen on the remote device to make direct operation difficult, and a technology that enables remote operation and direct operation simultaneously. A remote operation is an operation for controlling the remote device, but is an operation on another device other than the remote device.

[0003] Japanese Patent Laid-Open No. 2006-129999 discloses a technique for disabling direct manipulation of a remote device and notifying the user that the direct manipulation has been disabled by displaying a screen.

[0004] Japanese Patent Application Laid-Open No. 2002-360520

[0005] However, none of the above-mentioned techniques mentions making it easy for only the user using remote access (control of a remote device using remote operation) to directly operate the remote device.

[0006] For example, if remote access to a remote device is made directly operable, the remote access user can operate it directly while remaining in remote access mode, but it also allows a third party to operate it directly. This can be particularly inconvenient if unintended third parties operate the payment screen or online banking screen.

[0007] Therefore, an object of the present invention is to provide a technique for more appropriately controlling whether or not a remote device can be directly operated when the remote device is remotely accessible.

[0008] The electronic device of the present invention is an electronic device that is a first device and has a reception means for receiving operations from a user and a control means for controlling the operation of a second device in accordance with the operation from the user to the reception means, and is characterized in that when the control means controls the operation of the second device in accordance with the operation from the user to the reception means, when the user is not in a position where he or she can operate an operating member provided on the second device, the control means performs a first control that suppresses an operation on the second device that is different from the operation from the user to the reception means.

[0009] According to the present invention, it is possible to provide a technique for more appropriately controlling whether or not a remote device can be directly operated when the remote device is remotely accessible.

[0010] FIG. 1 is an overall diagram showing the configuration of an information processing system. FIG. 2 is a block diagram showing the configuration of the information processing system. FIG. 3 is a diagram showing a state in which an overlay display is made in a virtual space. FIG. 4 is a diagram showing a state in which the overlay display is enlarged in the virtual space. FIG. 5 is a flowchart showing control for suppressing direct manipulation. FIGS. 6A and 6B are setting screens for a control method related to direct manipulation. FIGS. 7A and 7B are screens for detailed setting of a control method related to direct manipulation.

[0011] Hereinafter, embodiments of the present invention will be described in detail by way of example with reference to the accompanying drawings. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the present invention.

[0012] First Embodiment FIG. 1 is an overall diagram showing the configuration of an information processing system 1. FIG. 2 is a block diagram showing the functional configuration of the information processing system 1. The information processing system 1 according to the first embodiment presents a mixed reality space, which is a fusion of real space and virtual space, to a user of a display device 101 who is experiencing the system. The mixed reality space is also called an MR (Mixed Reality) space. In the first embodiment, the information processing system 1 displays a composite image obtained by combining an image of a virtual space rendered by CG (Computer Graphics) with an image of a real space. In this way, the information processing system 1 presents an MR space (such as the MR space 301 shown in FIG. 3 ) to the user.

[0013] The information processing system 1 includes a local device 10, a remote device 104, and a device 105. The local device 10 is an electronic device including a display device 101, an information processing device 102, and an operation device 103.

[0014] The display device 101 is a video see-through display device, and is a head-mounted display device (HMD). However, the display device 101 is not limited to such an HMD, and may also be a handheld display (HHD). An HHD is a handheld display. In other words, the display device 101 may be a display that a user holds in their hand and peers into like binoculars to observe an image. The display device 101 may also be a display terminal such as a tablet or a smartphone.

[0015] The information processing device 102 generates a composite image by combining the image of real space captured from the display device 101 with the image of virtual space generated by the information processing device 102. The information processing device 102 outputs the composite image to the display device 101 as an MR image (mixed reality image).

[0016] The operation device 103 is capable of receiving operations from a user. The operation device 103 is, for example, an operation device for video games that can detect values ​​corresponding to the tilt direction of a stick and values ​​indicating the pressed state of a button.

[0017] The operation device 103 is not limited to an operation device that is held by the user's hand (such as an operation device for a video game), but may also be an operation device that is worn on the user's body or hand, etc. The operation device 103 may be, for example, a ring-shaped device that can be worn on the user's finger.

[0018] The remote device 104 is an electronic device, and is a terminal that can be remotely accessed (controlled in response to operations on the local device 10) from the local device 10. The remote device 104 may be a terminal such as a personal computer or a tablet.

[0019] The device 105 is a terminal held or worn by a person (hereinafter referred to as "other person") other than the user (remote access user) operating the local device 10. The device 105 may be, for example, a smartphone, a smart watch, smart glasses, or other device held or worn by the other person.

[0020] The information processing device 102 and the display device 101, the information processing device 102 and the operation device 103, and the information processing device 102 and the remote device 104 are connected to each other so that data communication is possible. Note that the data communication may be wired communication or wireless communication. The display device 101 may have the information processing device 102 built therein. Furthermore, when a camera or the like connected to the information processing device 102 is used, the camera and the information processing device 102 are connected to each other so that data communication is possible. Alternatively, the camera and the display device 101 may be connected to each other so that data communication is possible.

[0021] However, the information processing system 1 according to the first embodiment is not limited to an MR system that displays MR images. For example, the information processing system 1 may be an AR (Augmented Reality) system that presents an image of a virtual space to a user by transmitting real space. Alternatively, the information processing system 1 may be a VR (Virtual Reality) system that presents only an image of a virtual space to a user. Alternatively, the information processing system 1 may be a so-called XR (Extended Reality) system, as exemplified above.

[0022] If the information processing system 1 is a VR system, the user must be able to use an operation unit to directly operate the remote device 104 in real space. If the information processing system 1 is a VR system, direct operation may be possible, for example, by superimposing CG on the operation unit. Furthermore, the information processing system 1 does not need to be an XR system. For example, the information processing system 1 may simply be a system that allows remote access from the local device 10 to the remote device 104. In this case, the local device 10 may be a mobile terminal including a display device 101 and an information processing device 102. Furthermore, the remote device 104 is not limited to mobile terminals such as smartphones and laptops, but may also be information processing terminals such as servers and desktop personal computers. Furthermore, remote access in this embodiment refers to, for example, accessing the remote device 104 to operate it remotely based on an operation from the user to the local device 10.

[0023] (Configuration of Display Device) The display device 101 has a sound recording unit 210 , a sound output unit 211 , an imaging unit 212 , and a display unit 213 .

[0024] The recording unit 210 captures sounds around the display device 101 and the user's own voice as audio data, and outputs the data to the recording processing unit 222. Depending on the application, the recording unit 210 may be a recording device such as a directional microphone or a movable pin microphone. The recording unit 210 may also be equipped with a plurality of these recording devices.

[0025] The audio output unit 211 outputs audio in accordance with audio data output from the information processing device 102. The audio output unit 211 may be an audio output device such as a speaker, earphones, or headphones. The audio output unit 211 may also include a plurality of these audio output devices. The audio output device may be connected to the display device 101 or the information processing device 102 by wire or wirelessly. Alternatively, a device including both the recording unit 210 and the audio output unit 211 (for example, wireless earphones with a microphone function) may communicate with the display device 101 or the information processing device 102.

[0026] The imaging unit 212 captures images of the real space continuously in time series and outputs the captured images of the real space (captured images) to the information processing device 102. The imaging unit 212 may include a stereo camera. The stereo camera has two cameras so as to be able to capture images of the real space existing in the line of sight of the user of the display device 101. The two cameras capture images on the opposite side (outside) of the user as viewed from the display device 101.

[0027] Furthermore, the imaging unit 212 may include a camera that captures an image of the user's side (inside) as viewed from the display device 101 so as to be able to detect the user's expressions (body movements), such as the user's facial expressions and body language. Furthermore, the imaging unit 212 may also use a camera or the like provided externally to the display device 101 in order to detect the user's expressions. The external camera may be, for example, a camera fixed to a tripod or a wall of a building. In this case, the external camera may transmit the captured image to the information processing device 102 via the display device 101, or may transmit the captured image directly to the information processing device 102. The captured image is stored in the data storage unit 225 by the control unit 220.

[0028] In this way, the imaging unit 212 is a camera for detecting the user's expression. However, the configuration for detecting the user's expression is not limited to this, and other configurations capable of detecting the user's expression may be used instead of the imaging unit 212. For example, the imaging unit 212 may have a sensor and a detection unit instead of a camera. Multiple sensors may be arranged on the user's body, including the face, hands, and feet, and the detection unit may detect the user's expression based on values ​​acquired by the sensors.

[0029] The display unit 213 displays the MR image output from the information processing device 102. The display unit 213 may include two displays arranged to correspond to the left and right eyes of the user, respectively. In this case, the MR image for the left eye is displayed on the display corresponding to the user's left eye, and the MR image for the right eye is displayed on the display corresponding to the user's right eye.

[0030] (Configuration of information processing device) The information processing device 102 has a control unit 220, an audio output instruction unit 221, a sound recording processing unit 222, an image generation unit 223, an image synthesis unit 224, a data storage unit 225, a remote access processing unit 226, an operation processing unit 227, a vibration instruction unit 228, and a position and orientation calculation unit 229.

[0031] The control unit 220 controls the entire information processing device 102. The control unit 220 can transmit and receive signals and data to and from the image generation unit 223, data storage unit 225, remote access processing unit 226, position and orientation calculation unit 229, etc. The control unit 220 can also receive data and signals from the imaging unit 212, sound recording processing unit 222, operation processing unit 227, etc. The control unit 220 can also transmit signals and data to the audio output instruction unit 221, image synthesis unit 224, vibration instruction unit 228, etc.

[0032] The control unit 220 controls the position and orientation of a GUI (Graphical User Interface), such as a pointer, displayed on the display unit 213, based on, for example, information on the position and orientation (position and orientation) of the operation device 103 stored in the data storage unit 225. The position of the GUI is indicated by three-dimensional coordinate information according to a Cartesian coordinate system of three axes, for example, the X-axis, the Y-axis, and the Z-axis. The orientation of the GUI corresponds to the orientation of the GUI in a three-dimensional virtual space. However, the three-dimensional coordinate information is not limited to information according to a three-axis Cartesian coordinate system, and may be, for example, three-dimensional coordinate information according to a polar coordinate system. Consider the case where the GUI is a virtual ray emitted from, for example, the tip of a user's finger. When the ray indicates a certain point, the position of the GUI is, for example, the start position or end position of the ray. When the GUI is a ray, the orientation of the GUI corresponds to, for example, the direction in which the ray extends in the three-dimensional virtual space.

[0033] The control unit 220 transmits a signal of an instruction related to the direct operation suppression control to the remote access processing unit 243. The instruction related to the direct operation suppression control may be, for example, an instruction related to enabling or disabling direct operation of the remote device 104. Furthermore, the instruction related to the direct operation suppression control may be, for example, an instruction related to starting or ending the direct operation suppression control.

[0034] Here, direct operation suppression control (suppression control) will be described. In the first embodiment, when the control unit 220 is performing remote access, only the user can directly operate the remote device 104. Therefore, during remote access, the control unit 220 controls to suppress direct operation of the remote device 104 by others other than the user, or to prevent others from directly operating the remote device 104. In the first embodiment, such control by the control unit 220 is referred to as "direct operation suppression control."

[0035] The audio output instruction unit 221 instructs the audio output unit 211 to output audio. For example, the control unit 220 acquires multiple pieces of audio source data stored in the data storage unit 225 and synthesizes the multiple pieces of audio source data as necessary. As a result, the control unit 220 generates audio source data that the user can hear and passes it to the audio output instruction unit 221. In this case, the audio output instruction unit 221 instructs the audio output unit 211 to output audio using the audio source data based on the instruction of the control unit 220.

[0036] The recording processing unit 222 processes the voice data acquired from the recording unit 210. The recording processing unit 222 transmits the user's voice data acquired from the recording unit 210 to the control unit 220. The recording processing unit 222 also analyzes the user's voice data acquired from the recording unit 210. In this way, the control unit 220 may perform various controls in accordance with the voice commands issued by the user obtained from the analysis of the voice data.

[0037] The image generation unit 223 constructs a virtual space based on the virtual space data stored in the data storage unit 225. The virtual space data includes data relating to each virtual object that constitutes the virtual space, data relating to a light source that illuminates the virtual space, and information about the user's appearance.

[0038] The image generation unit 223 acquires information about the position and orientation of the imaging unit 212 calculated by the position and orientation calculation unit 229 from the data storage unit 225 via the control unit 220. The image generation unit 223 also acquires information about the position and orientation of the GUI controlled by the control unit 220 from the data storage unit 225 via the control unit 220. The image generation unit 223 generates an image of the virtual space (virtual space image) according to the position and orientation of the imaging unit 212.

[0039] In the first embodiment, the user's appearance information is expressed in a virtual space constructed by the image generation unit 223 in accordance with instructions from the control unit 220. The user's appearance information may be information that faithfully reproduces the actual user's appearance, or information that has been processed based on the actual user's appearance. The user's appearance information may be changeable, for example, by changing the skin color, including skin beautification, changing the appearance such as the color of the hair, eyelashes, eyebrows, and eyes, changing the clothing, changing the height and proportions, etc. Furthermore, the user's appearance information may be information such as a 3D avatar that is not based on the actual user's appearance.

[0040] The image synthesis unit 224 generates an MR image by synthesizing the image of the virtual space generated by the image generation unit 223 with the image of the real space captured by the imaging unit 212. At this time, the image of the virtual space generated by the image generation unit 223 may be an image representing the entire virtual space or an image representing a part of the virtual space. Furthermore, the image synthesis unit 224 may generate an MR image by performing an affine transformation on the image, or may generate an MR image by assigning the image to a parametric curved surface.

[0041] The image synthesis unit 224 outputs the generated MR image to the display unit 213. The image synthesis unit 224 is not limited to a configuration that outputs the MR image to the display unit 213. For example, the image synthesis unit 224 may transmit the MR image to an external device. Furthermore, the image synthesis unit 224 may store the MR image as a 3D video in a storage medium so that it can be later displayed on the display unit 213. For example, if the display device 101 is an optical see-through HMD, the user can see the real space even without displaying a virtual space on the display unit 213. For this reason, an image displayed on the remote device 104 may be superimposed on the external monitor and recorded data in a picture-in-picture manner. If the user's operation is not impaired even if the MR image is not displayed on the display unit 213, the image synthesis unit 224 may output the MR image to a device other than the display unit 213. Furthermore, the MR image generated by the image synthesis unit 224 may be output simultaneously to the "display unit 213" and "a display unit other than the display unit 213."

[0042] The data storage unit 225 can store various information for operating the information processing device 102. The data storage unit 225 includes at least one of storage media such as a RAM (Random Access Memory) and a hard disk drive device. In addition to the information described above as information to be stored in the data storage unit 225, the data storage unit 225 can also store information such as programs related to the operation of the information processing system 1. The data storage unit 225 can store sound source data that can be heard by the user (e.g., synthesized voice, notification sound, etc.).

[0043] The remote access processing unit 226 performs processing based on instructions from the control unit 220. For example, the remote access processing unit 226 controls the remote access processing unit 243 so that the information processing device 102 can access the remote device 104. The remote access processing unit 226 communicates with the remote access processing unit 243 of the remote device 104, which is the remote access destination, thereby realizing remote access between the information processing device 102 and the remote device 104. During remote access, the information processing device 102 generates an image to be displayed on the display unit 213 (hereinafter referred to as a "display image"). The display image may be generated by the remote access processing unit 226. At this time, the remote access processing unit 226 may receive information stored in the remote device 104 via the remote access processing unit 243.

[0044] The remote access processing unit 226 may generate a display image based on information received from the remote device 104. For example, a template is stored in advance in the data storage unit 225, and the control unit 220 reads out the template. The remote access processing unit 226 may process the template based on information received from the remote access processing unit 243 to generate a display image. The generated display image may be composited with another image by the image composition unit 224 and displayed to the user as a remote image by the display unit 213. The display image may also be a virtual object corresponding to the remote device 104 generated in a virtual space. The user may be able to control the remote device 104 by controlling the virtual object in the virtual space displayed on the display unit 213.

[0045] The remote access processing unit 226 also performs processing related to the termination of remote access based on instructions from the control unit 220. When the remote access is to be terminated, the remote access processing unit 226 transmits an instruction to terminate remote access to the remote access processing unit 243. In this way, the remote access processing unit 226 terminates the remote access from the local device 10 to the remote device 104 and also hides the remote screen being displayed on the display unit 213.

[0046] However, the timing of hiding or terminating remote access does not matter. In some cases, hiding and terminating remote access may be performed simultaneously through parallel processing. Furthermore, the instruction to terminate remote access may also serve as an instruction regarding the suppression control of the remote device 104. This is because the remote device 104 needs to release the suppression control when the remote access is terminated. Note that the remote access processing unit 226 does not need to specifically instruct the remote device 104 to release the suppression of direct operation when the remote access is terminated. For example, if the remote device 104 processes the remote device 104 to automatically release the suppression of direct operation when the remote access is terminated, the remote access processing unit 226 does not need to send a specific instruction signal regarding the termination of the suppression control.

[0047] The operation processing unit 227 receives operation information related to operations performed by the user on the operation instruction unit 231. Based on the operation information received by the operation processing unit 227, the control unit 220 instructs the image generation unit 223 to construct a virtual space that reflects the operation related to the virtual object, as necessary. In this way, the operation processing unit 227 realizes control of the virtual object corresponding to the operation performed by the user. Note that the virtual object controlled by the user does not have to be a single object, but may be, for example, a group consisting of multiple objects. Furthermore, for example, when an operation related to confirmation or cancellation is performed, the control unit 220 performs control related to confirmation or cancellation based on the user's operation.

[0048] The operation processing unit 227 changes data related to each virtual object stored in the data storage unit 225 based on operation information acquired from the operation instruction unit 231 and overwrites the data in the data storage unit 225. The operation processing unit 227 transmits data related to each virtual object and information on the position and orientation of the imaging unit 212 to the image generation unit 223. The operation processing unit 227 then controls the virtual objects by causing the image generation unit 223 to construct a virtual space that takes into account the position and orientation of the imaging unit 212. In this case, the virtual object may be an object related to the start and end of remote access, or may be a virtual object corresponding to the remote device 104 currently performing remote access. For example, the image generation unit 223 generates a virtual object of the remote device 104 based on data received from the remote access processing unit 226 via the control unit 220. However, if the data received by the control unit 220 is the virtual object itself, the image generation unit 223 does not need to generate a virtual object.

[0049] The vibration instruction unit 228 transmits a vibration instruction signal (an instruction requesting vibration) to the vibration unit 230 based on an instruction from the control unit 220. The vibration unit 230 vibrates in response to this signal, and therefore the vibration instruction unit 228 can give the user a vibration sensation.

[0050] The position and orientation calculation unit 229 calculates (acquires) the position and orientation (at least one of the position and orientation) of the image capture unit 212 in the world coordinate system. The position and orientation calculation unit 229 extracts markers assigned to the world coordinate system from an image of real space captured by the image capture unit 212. Then, the position and orientation calculation unit 229 calculates the position and orientation of the image capture unit 212 in the world coordinate system based on the positions and orientations of the extracted markers. Here, the position and orientation calculation unit 229 stores information indicating the calculated position and orientation of the image capture unit 212 (position and orientation information) in the data storage unit 225 via the control unit 220.

[0051] The position and orientation calculation unit 229 may calculate the position and orientation of the operation device 103 in the world coordinate system using the position and orientation information of the operation device 103 acquired from the position and orientation calculation unit 232. Note that the position and orientation of the operation device 103 in the world coordinate system may be calculated using, for example, an image of real space captured by the imaging unit 212. The position and orientation calculation unit 229 may store the calculated position and orientation information of the operation device 103 in the data storage unit 225 via the control unit 220. In this case, the control unit 220 may control the position of the GUI based on the position and orientation information of the imaging unit 212 stored in the data storage unit 225. Furthermore, the control unit 220 may control the position of the GUI based on either or both of the position and orientation information of the operation device 103 and the position and orientation information of the imaging unit 212.

[0052] Note that detailed description will be omitted because it is a well-known technique for generating an image in a virtual space in accordance with information on the position and orientation of the image capture unit 212. Furthermore, the information processing device 102 does not need to include the position and orientation calculation unit 229 when the control unit 220 calculates the position and orientation of the image capture unit 212 in the world coordinate system.

[0053] (Configuration of Operation Device) The operation device 103 includes a vibration unit 230 , an operation instruction unit 231 , and a position and orientation calculation unit 232 .

[0054] The vibration unit 230 vibrates to provide a vibration sensation to the user in accordance with a vibration instruction from the vibration instruction unit 228. The vibration unit 230 has a vibration element with a short response time to provide the vibration sensation. For example, a piezoelectric element can be used as the vibration element. The vibration unit 230 may also have other vibration elements such as a linear resonant actuator (LRA) or an eccentric rotating mass (ERM).

[0055] The operation instruction unit 231 outputs values ​​corresponding to the user's operation (such as a value corresponding to the tilt direction of the stick and a value indicating the pressed state of a button) as operation information to the operation processing unit 227. Note that the operation referred to here may be an operation to instruct selection, determination, or movement of a virtual object. Furthermore, the movement of a virtual object is not limited to simple translation, and may include various affine transformations such as rotation, enlargement, and reduction.

[0056] The operation instruction unit 231 may have multiple buttons or may incorporate an optical trackpad (OTP). If the operation device 103 incorporates an OTP, the user can place a finger on the OTP and rub it in any direction to align the pointer with a desired item. The user can then perform a confirmation action, such as pressing a button on the OTP, to confirm the selection of a virtual object, such as a menu item. The user can also point to another location while keeping a virtual object selected. While selecting a virtual object, the user can also move the virtual object by performing a predetermined operation, such as twisting their arm or finger.

[0057] The operation device 103 may not have the operation instruction unit 231. In this case, the control unit 220 may recognize the user's hand movement (gesture) from the image of real space captured by the imaging unit 212, and store information about the recognized gesture in the data storage unit 225.

[0058] The position and orientation calculation unit 232 calculates (acquires) the position and orientation (at least one of the position and orientation) of the operation device 103 in the world coordinate system. Then, the position and orientation calculation unit 232 outputs information on the calculated position and orientation to the operation processing unit 227. The position and orientation calculation unit 232 has, for example, a sensor (such as an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor) for calculating the position and orientation of the operation device 103. Note that the position and orientation calculation unit 232 may have multiple sensors.

[0059] The position and orientation calculation unit 232 may have an imaging unit and an imaging processing unit, and may calculate the position and orientation through processing by the imaging unit and the imaging processing unit. Specifically, the imaging processing unit may perform Simultaneous Localization and Mapping (SLAM) processing based on feature points of the captured image acquired by the imaging unit to calculate the position and orientation of the operation device 103. Furthermore, the position and orientation calculation unit 232 may calculate the position and orientation in conjunction with an optical sensor installed in real space.

[0060] Furthermore, the control unit 220 may control the position of the GUI based on the information about the position and orientation of the operation device 103 acquired by the position and orientation calculation unit 232 .

[0061] Note that when the position and orientation calculation unit 232 calculates the position and orientation, an error may occur between the calculated position and orientation of the operation device 103 and the actual position and orientation. For example, in a method of calculating the position and orientation using a combination of an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor, errors in the sensor values ​​acquired by each sensor may accumulate, resulting in a calculated value that includes an error compared to the actual position and orientation. Furthermore, in a method of calculating the position and orientation using an optical sensor installed in real space, the optical sensor may be blocked by another real object, resulting in a calculated value that includes an error compared to the actual position and orientation.

[0062] As such, there are cases where the position and orientation calculation unit 232 cannot correctly calculate the position and orientation of the operation device 103. Therefore, the position and orientation calculation unit 232 may calculate the position and orientation of the operation device 103 based on the position and orientation of the marker by extracting the marker attached to the operation device 103 from the image of real space captured by the imaging unit 212. By calculating the position and orientation of the operation device 103 based on the position and orientation of the marker in this way, the accuracy of the calculation result can be improved. In this case, the position and orientation calculation unit 232 may use all or part of the calculation result of the position and orientation acquired from the position and orientation calculation unit 229.

[0063] The method of calculating the position and orientation by the position and orientation calculation unit 232 is not limited to a method using markers, and may be a method of calculating the position and orientation by SLAM processing. Furthermore, an orientation sensor unit (not shown) of the display device 101 may detect the orientation. The orientation sensor unit has an inertial measurement unit (IMU) and outputs information about the orientation of the display device 101 (orientation information) to the information processing device 102.

[0064] Furthermore, when the operation device 103 does not appear in the image of real space captured by the imaging unit 212, the position and orientation calculation unit 229 may calculate the position and orientation of the operation device 103 based on the value acquired from the position and orientation calculation unit 232. At this time, the position and orientation calculation unit 229 may calculate the position and orientation of the operation device 103 based on the value acquired from the position and orientation calculation unit 232 and other information. For example, when the operation device 103 does not appear in the image of real space, if the operation device 103 is equipped with an acceleration sensor or the like, the position and orientation calculation unit 229 may calculate the position and orientation of the operation device 103 based on the detection result of the acceleration sensor or the like.

[0065] However, the position and orientation of the operation device 103 does not have to be calculated based on various sensors. For example, the position and orientation calculation unit 229 adds the amount of movement of the operation device 103 from a past time point, calculated from the acceleration, to position information of the operation device 103 based on an image in which the operation device 103 appears among images in real space at a past time point. In this way, the position and orientation calculation unit 229 may calculate the current position and orientation of the operation device 103. Note that if the operation device 103 is not appearing in an image in real space, the position and orientation calculation unit 229 may detect only the orientation of the operation device 103.

[0066] As described above, the position and orientation calculation unit 232 may output information (values ​​output from the sensor) or images for calculating the position and orientation of the operation device 103 to the information processing device 102. In this case, the position and orientation calculation unit 229 may calculate the position and orientation of the operation device 103 based on the information or images acquired from the position and orientation calculation unit 232. In this case, the operation device 103 does not need to have the position and orientation calculation unit 232.

[0067] The position and orientation calculation unit 232 may calculate the values ​​of either the position component or the orientation component, rather than calculating the values ​​of both.

[0068] (Configuration of Remote Device) The remote device 104 includes an audio output unit 241 , a display unit 242 , a remote access processing unit 243 , a control unit 244 , a data storage unit 245 , a direct operation unit 246 , and a vibration instruction unit 247 .

[0069] The audio output unit 241 outputs audio based on instructions from the control unit 244 .

[0070] The display unit 242 displays information based on instructions from the control unit 244 .

[0071] The remote access processing unit 243 performs "processing related to the start and end of remote access" and "processing related to suppression control" based on information received from the information processing device 102. The information that the remote access processing unit 243 receives from the information processing device 102 is, for example, a notification of the start and end of remote access. However, the notification of the start and end of remote access may not be received from the information processing device 102 but may be received from another device (such as the device 105). The processing related to the start and end of remote access itself is a well-known technique, so will not be described here.

[0072] When remote access starts, the remote access processing unit 243 acquires screen information (information about the image displayed on the display unit 242) of the remote device 104 via the control unit 244 and transmits the screen information to the information processing apparatus 102. When remote access ends, the remote access processing unit 243 stops acquiring screen information of the remote device 104 from the control unit 244. The screen information of the remote device 104 may be image data such as bitmap and JPEG, or may be image data containing only difference information from a template screen or data in another format.

[0073] The control unit 244 controls the entire remote device 104. The control unit 244 can transmit and receive signals and data to and from the remote access processing unit 243 and the like. The control unit 244 can receive signals from the direct operation unit 246 and the like. The control unit 244 can transmit signals and data to the audio output unit 241, the display unit 242, and the like.

[0074] When direct operation on the remote device 104 is enabled, the control unit 244 performs control based on an operation instruction signal from the direct operation unit 246. In this case, the control unit 244 instructs the display unit 242 to perform display output, or instructs the audio output unit 241 to perform audio output.

[0075] The control unit 244 may perform control to issue a predetermined notification to prevent others from operating the remote device 104 in response to a request from the remote access processing unit 243 based on an instruction from the control unit 220 .

[0076] The control unit 244 may transmit a "signal instructing to give a predetermined notification by voice" to the audio output unit 241. The control unit 244 may transmit a "signal instructing to give a predetermined notification by generating a specific display item related to the predetermined notification and displaying the specific display item" to the display unit 242.

[0077] When the remote device 104 is connected to the device 105, the control unit 244 may send a "signal instructing the vibration instruction unit 247 to make a predetermined notification by vibrating the vibration unit 250."

[0078] If the remote device 104 has an audio instruction unit and the device 105 has an audio output unit, the control unit 244 may cause the audio output unit of the device 105 to make a predetermined audio notification via the audio instruction unit of the remote device 104.

[0079] If the remote device 104 has a display instruction unit and the device 105 has a display unit, the control unit 244 may cause the display unit of the device 105 to display a specific display item related to a specified notification via the display instruction unit of the remote device 104.

[0080] However, the control for issuing a predetermined notification is not limited to these, and any control for preventing others from operating the remote device 104 may be used.

[0081] The control unit 244 may perform processing to disable an operation input to the direct operation unit 246 in response to a request from the remote access processing unit 243 based on an instruction from the control unit 220. The control unit 244 may also perform processing to enable an operation input to the direct operation unit 246 in response to a request from the remote access processing unit 243 based on an instruction from the control unit 220.

[0082] Here, enabling and disabling of operations will be described. "Enabling an operation" means that when a user performs an operation on the direct operation unit 246, the control unit 244 controls the remote device 104 to a state in which processing (control) corresponding to the operation is performed. "Disabling an operation" means that even when a user performs an operation on the direct operation unit 246, the control unit 244 controls the remote device 104 to a state in which processing (control) corresponding to the operation is not performed.

[0083] The control unit 244 may transmit a signal of a direct operation control instruction (control instruction) to the direct operation unit 246. The control instruction is an instruction to the direct operation unit 246 to perform processing to enable or disable direct operation. The direct operation unit 246 performs processing to enable or disable direct operation based on the signal from the control unit 244. In particular, when disabling direct operation, operation information is no longer transmitted from the direct operation unit 246, so that it is expected that the amount of communication within the remote device 104 will be reduced.

[0084] However, if processing is performed to disable operations on the direct operation unit 246, operation information will no longer be transmitted from the direct operation unit 246, and therefore, when an attempt is made to enable an operation, components other than the direct operation unit 246 will be unable to perform processing in response to the direct operation. Therefore, the direct operation unit 246 itself needs to have the function of enabling and disabling direct operations.

[0085] The signal that the control unit 244 receives from the remote access processing unit 243 may be a signal of abstract instructions regarding suppression control, or a signal of specific instructions regarding suppression control. If the signal regarding suppression control is a signal of abstract instructions, the control unit 244 performs specific suppression control at its own discretion. In this case, the control unit 244 may perform suppression control in accordance with default settings regarding direct operation stored in the data storage unit 245. Specific instructions regarding suppression control and abstract instructions regarding suppression control will be described in detail below.

[0086] The data storage unit 245 can store various information for operating the remote device 104. The data storage unit 245 includes at least one of storage media such as a RAM (Random Access Memory) and a hard disk drive device. The data storage unit 245 can store settings related to direct operations that can be read by the control unit 244. However, the remote device 104 does not necessarily have to have the data storage unit 245.

[0087] The direct operation unit 246 is an operation member used when the user operates the remote device 104. For example, if the remote device 104 is a personal computer, the direct operation unit 246 corresponds to input devices such as a mouse and a keyboard. Even if the remote device 104 is a device other than a personal computer, the direct operation unit 246 is not particularly limited as long as it is an input device that can directly operate the remote device 104.

[0088] The vibration instruction unit 247 transmits a vibration instruction signal to the vibration unit 250 of the device 105 based on a signal from the control unit 244. Communication between the vibration instruction unit 247 and the device 105 may be established in advance. When transmitting the vibration instruction signal, the vibration instruction unit 247 may dynamically establish communication and then transmit the vibration instruction signal.

[0089] Typically, the vibration instruction unit 247 notifies the control unit 244 in advance that the device 105 is present, and based on that information, the control unit 244 transmits a vibration instruction signal to the vibration unit 250 via the vibration instruction unit 247. Therefore, it is assumed that communication between the control unit 244 and the vibration unit 250 is first established or communication possibility is confirmed. However, if communication establishment can be performed in a short time that does not affect usability, communication may be established dynamically when transmitting a vibration instruction signal as described above. However, as long as a vibration instruction signal can be transmitted from the control unit 244 to the vibration unit 250, the method and timing of establishing communication are not limited.

[0090] (Device Configuration) The device 105 has a vibration section 250 .

[0091] The vibration unit 250 vibrates based on a signal from the vibration instruction unit 247 of the remote device 104 .

[0092] Furthermore, the device 105 may have a display unit (not shown). When the device 105 has a display unit, the remote device 104 may have a display instruction unit (not shown). The display unit may display an image to be shown to the user (display output to the user) based on a signal from the display instruction unit of the remote device 104.

[0093] Furthermore, the device 105 may have an audio output unit (not shown). When the device 105 has an audio output unit, the remote device 104 may have an audio instruction unit (not shown). The audio output unit may output audio to be heard by the user (audio output to the user) based on a signal from the audio output instruction unit of the remote device 104.

[0094] The device 105 operates based on an instruction from the remote device 104, but may also operate based on an instruction signal from a terminal other than the remote device 104. For example, the information processing device 102 may send an instruction signal to the device 105, or another terminal (not shown) may send an instruction signal to the device 105. When a terminal equipped with a sensor such as a camera determines that direct operation should be suppressed, the terminal may send an instruction signal to the device 105, causing the device 105 to operate.

[0095] (Instructions Related to Direct Operation Inhibition Control) The following describes direct operation inhibition control (inhibition control). An instruction related to inhibition control is, for example, an instruction to output audio from the audio output unit 241 of the remote device 104. An instruction related to inhibition control may also be an instruction to display a specific display item related to a predetermined notification on the display unit 242. An instruction related to inhibition control may also be an instruction to vibrate the vibration unit 250 of the device 105 via the remote device 104. An instruction related to inhibition control may also be an instruction to display output on a display unit (not shown) of the device 105, or an instruction to output audio from the audio output unit.

[0096] In the first embodiment, abstract instructions regarding inhibitory control are instructions such as whether to perform inhibitory control. Specific instructions regarding inhibitory control include instructions regarding individual control of the method of notification to others. Specific instructions regarding inhibitory control include instructions for setting whether to enable or disable direct operation of the remote device 104 by others. In other words, even when inhibitory control is performed, others may actually be able to operate the remote device 104 simply by issuing a predetermined notification, or others may actually be unable to operate the remote device 104 after issuing a predetermined notification. Furthermore, others may actually be unable to operate the remote device 104 without issuing a predetermined notification.

[0097] Now, consider a case where the information processing device 102 and the remote device 104 communicate with each other regarding instructions related to power reduction. When the signal transmitted from the control unit 220 is a specific instruction related to power reduction, the remote access processing unit 226 receives the signal transmitted from the control unit 220 and transmits it to the remote access processing unit 243. Based on the received signal, the remote access processing unit 243 transmits an instruction signal related to power reduction to the control unit 244.

[0098] If the instruction transmitted from the information processing device 102 to the remote device 104 is an abstract instruction regarding suppression control, the remote device 104 determines the specific suppression control (e.g., the method of notifying others). For example, if the signal transmitted from the control unit 220 is an abstract instruction regarding suppression control, the remote access processing unit 243 may determine the specific control regarding suppression control. In this case, the remote access processing unit 243 may transmit a signal containing the specific instruction regarding suppression control to the control unit 244 based on the determination result. Alternatively, the signal transmitted from the remote access processing unit 243 to the control unit 244 may include only the abstract instruction regarding suppression control, and the control unit 244 may determine the specific suppression control.

[0099] (MR Space) Here, an example of an MR image generated by the image synthesis unit 224 is shown. Fig. 3 is a diagram showing an MR image displayed on the display unit 213, which is an image when superimposed (overlaid) on a remote device. Fig. 4 is a diagram showing an MR image displayed on the display unit 213, which is an image when enlarged and displayed on the screen of the remote device. The MR image generated by the image synthesis unit 224 includes an MR space 301 and a virtual object 202.

[0100] The MR space 301 is generated by the image generation unit 223 based on an image captured by the imaging unit 212. The image generation unit 223 generates the virtual object 202 based on information acquired from the remote device 104 (control unit 244). The remote device 104 is placed in the MR space 301, and the virtual object 302 is configured as a virtual object corresponding to the screen of the remote device 104.

[0101] In the state shown in FIG. 3 , a virtual object 302 is arranged in the MR space 301 so as to be superimposed on the screen portion of the remote device 104. This allows the user to control the remote device 104 through the MR space 301 in the same way as in the real space. In this case, the user can check the screen of the remote device 104 via the display unit 213. However, in the remote device 104 in the real space, the display unit 242 may be set to hidden, or a specific display item related to a predetermined notification may be displayed on the display unit 242. In other words, the display unit 242 in the real space may be displayed in a way that makes it difficult for others to directly operate the remote device 104.

[0102] 4 , a virtual object 302 is displayed in the MR space 301 and is enlarged compared to the screen portion of the remote device 104. The screen portion of the remote device 104 is blacked out. This allows the user, when attempting to control the remote device 104 through the MR space 301, to check the screen of the remote device 104 by remote access without directly operating the remote device 104.

[0103] However, the display method of the virtual object 302 is not limited to this. For example, the image synthesis unit 224 may place the virtual object 302 in the MR space 301 so as to be superimposed on the remote device 104. In this case, the image synthesis unit 224 may display the virtual object 302 at a position adjusted so as to face the user. Furthermore, the image synthesis unit 224 may fix the display direction of the virtual object 302 depending on the content of the MR space 301. Furthermore, the virtual object 302 does not need to be generated by the image generation unit 223. For example, the display unit 213 may display an image displayed on the screen of the remote device 104 acquired from the imaging unit 212, instead of the virtual object 302.

[0104] (Control for Suppressing Direct Operation of Remote Device) FIG. 5 is a flowchart showing suppression control for the remote device 104. In the first embodiment, the information processing apparatus 102 can be configured to "suppress direct operation during remote access by displaying a setting screen such as that shown in FIG. 6A on the display unit 213." Here, a case where the setting for suppressing direct operation during remote access is effective is considered. In the following, control will be described for this case in which direct operation of the remote device 104 by a user is suppressed while direct operation of the remote device 104 by a person other than the user during remote access is suppressed.

[0105] In step S501, the control unit 220 determines whether or not the user has instructed the remote device 104 to start remote access. If it is determined that the user has instructed the remote device 104 to start remote access, the process proceeds to step S502. If it is determined that the user has not instructed the remote device 104 to start remote access, the process of step S501 is repeated.

[0106] For example, the user may instruct the start of remote access by selecting a virtual object corresponding to the remote device 104 displayed on the display unit 213. The control unit 220 acquires information acquired by the operation processing unit 227 (information acquired based on an operation by the user on the operation instruction unit 231).

[0107] When the user issues an instruction to start remote access, the control unit 220 transmits a remote access start instruction signal to the remote access processing unit 226, and the remote access processing unit 226 communicates with the remote access processing unit 243. This starts remote access. The remote access processing unit 243 requests screen information of the remote device 104 from the control unit 244 and receives the screen information from the control unit 244. The remote access processing unit 243 then begins transmitting the screen information of the remote device 104 to the remote access processing unit 226.

[0108] However, the method of instructing the start of remote access is not particularly limited. For example, when the user's line of sight is detected using the imaging unit 212, the user may instruct the start of remote access by gazing at a virtual object corresponding to the remote device 104 for a predetermined period of time or more. In this case, the control unit 220 controls the information processing device 102 based on the user's line of sight information detected by the imaging unit 212, without via the operation processing unit 227. In the first embodiment, an example is shown in which the start of remote access is instructed using a virtual object corresponding to the remote device 104, but an actual image of the remote device 104 (an image captured of the remote device 104) may also be used.

[0109] In step S502, the control unit 220 transmits a signal from the remote access processing unit 226 to the remote access processing unit 243 as an instruction regarding restriction control.

[0110] When the remote access processing unit 243 receives an instruction signal regarding suppression control, it transmits the instruction signal regarding suppression control to the control unit 244. When the control unit 244 receives the instruction signal regarding suppression control, it transmits a control instruction signal to the direct operation unit 246 to disable direct operation. Alternatively, the control unit 244 transmits a signal to the audio output unit 241 and the display unit 242 of the remote device 104 to provide a predetermined notification based on a request from the remote access processing unit 243. By issuing an instruction regarding suppression control to the remote device 104, a notification is provided to the display unit 242, etc., that direct operation is not possible. While the control within the remote device 104 has been described above, the control unit 244 may also transmit an instruction signal regarding suppression control to the device 105. For example, the control unit 244 may suppress direct operation by vibrating the vibration unit 250 via the vibration instruction unit 247.

[0111] Here, the setting of the control method for direct manipulation will be described. FIGS. 6A and 6B show setting screens for the control method for direct manipulation. The user may be able to set the restriction and prohibition of direct manipulation by displaying setting screens such as those shown in FIGS. 6A and 6B on the display unit 213 as the settings for the above-mentioned instructions for suppression control and control instructions. Based on the user's operation (input) on the operation instruction unit 231, the control unit 220 may transmit a setting instruction signal from the remote access processing unit 226 to the remote access processing unit 243. Information entered by the user regarding the method for suppressing direct manipulation is transmitted to the control unit 244, and the control unit 244 controls the remote device 104 based on the received information. While an example has been shown in which the user sets the setting screens shown in FIGS. 6A and 6B from the display unit 213, a similar setting screen may also be displayed on the remote device 104. In this case, the control unit 244 performs control according to the settings of the remote device 104.

[0112] 7A and 7B show screens for detailed settings of the control method for direct manipulation. When suppressing direct manipulation is selected in the settings for the control method for suppressing direct manipulation described above, a settings screen such as that shown in FIG. 7A is displayed on the display unit 213. This may allow the user to set detailed control of suppressing direct manipulation. For example, when the user selects "enable control for suppressing direct manipulation," each notification method, such as audio notification, vibration notification, and visual notification, may be individually enabled or disabled.

[0113] Furthermore, the detailed settings screen may allow the user to select whether to enable direct operation of the remote device 104. In the screen of FIG. 7A , audio notification and display notification are selected as notification methods. Therefore, to prevent others from operating the remote device 104, a predetermined audio notification is output from the audio output unit 241 and a predetermined notification is provided via a specific display item displayed on the display unit 242. Also, in the screen of FIG. 7A , direct operation is enabled. Therefore, the remote device 104 provides audio and visual notifications to prevent others from operating the remote device 104, but allows others to operate the remote device 104. However, detailed control does not have to be set individually; for example, multiple instructions may be set in a single package. This may reduce communication traffic.

[0114] As shown in Fig. 7A , the notification method and enabling / disabling direct manipulation may be selectable as separate items, or the user may be able to select detailed settings for suppressing direct manipulation from each item, as shown in Fig. 7B . In Fig. 7B , the detailed settings for suppressing direct manipulation include audio notification, display screen notification, and disabling direct manipulation. That is, the screen shown in Fig. 7B is configured to provide audio and visual notifications to prevent others from operating the remote device 104, and also to prevent the user from performing direct manipulation.

[0115] In step S503, the control unit 220 determines whether or not the user is in a state where he or she can directly operate the remote device 104. If it is determined that the user is in a state where he or she can directly operate the remote device 104, the process proceeds to step S504. If it is determined that the user is not in a state where he or she can directly operate the remote device 104, the process proceeds to step S505.

[0116] For example, the control unit 220 determines that the user is in a state where the user can directly operate the remote device 104 when the distance between the user and the remote device 104 is less than a threshold. Furthermore, the control unit 220 determines that the user is not in a state where the user can directly operate the remote device 104 when the distance between the user and the remote device 104 is equal to or greater than a threshold. One method for detecting the distance between the user and the remote device 104 is to detect the distance between the user and the remote device 104 based on an image of the remote device 104 obtained from the imaging unit 212. Alternatively, an imaging unit (not shown) provided in the remote device 104 may acquire images of the user and the display device 101, etc., and the control unit 220 may detect the distance between the remote device 104 and the user. Furthermore, the control unit 220 may detect the distance between the user and the remote device 104 based on the communication strength between the devices, such as the communication strength between the remote access processing unit 226 and the remote access processing unit 243. As described above, the distance detection method is not particularly limited as long as the distance between the user and the remote device 104 can be detected. Furthermore, the method for determining whether the user is able to directly operate the remote device 104 is not limited to the above-described method. In step S503, if it is possible to determine whether the user is in a state where he or she can directly operate the remote device 104, a method that does not use distance may be used.

[0117] In step S504, the control unit 220 transmits a signal to release the suppression control of the remote device 104 in preparation for direct operation by the user. A signal instructing to release the suppression of direct operation (hereinafter referred to as a "release instruction") is transmitted from the remote access processing unit 226 to the remote access processing unit 243. Upon receiving the release instruction signal, the remote access processing unit 243 transmits a release instruction signal to the control unit 244. Upon receiving the release instruction signal, the control unit 244 performs control to release the suppression control.

[0118] Here, the cancellation of suppression control will be described. "Canceling suppression control" means, for example, stopping the audio output from the audio output unit 241 when a predetermined notification audio is being output from the audio output unit 241. Furthermore, "canceling suppression control" means stopping the display from the display unit 242 when an image (display item) of a predetermined notification is being displayed from the display unit 242. Furthermore, "canceling suppression control" means, when a predetermined notification is being output by vibration from the vibration unit 250, stopping the vibration output instruction signal to the vibration instruction unit 247 and stopping the vibration of the vibration unit 250. When canceling suppression control, control may be performed to stop the output instruction signal, or control may be performed to transmit a stop instruction signal. Furthermore, when direct manipulation is disabled, a signal instructing the direct manipulation unit 246 to enable direct manipulation may be transmitted.

[0119] In step S505, the control unit 220 performs processing to suppress direct operation or processing to disable direct operation so that others do not operate the remote device 104. As a result, the control unit 220 performs direct operation suppression control to prevent others from operating the remote device 104 when the user is not able to operate the remote device 104. The control unit 220 transmits an instruction signal related to the suppression control from the remote access processing unit 226 to the remote access processing unit 243, and thereafter transmits the signal in the same manner as a release instruction signal.

[0120] In step S506, the control unit 220 determines whether or not remote access to the remote device 104 has ended. If it is determined that remote access to the remote device 104 has ended, the processing of this flowchart ends. If it is determined that remote access to the remote device 104 has not ended, the process returns to step S504. However, the method for determining whether or not remote access has ended is not particularly limited. For example, the remote access processing unit 226 may determine whether or not remote access has ended. The remote access processing unit 226 may store status information indicating whether remote access is in progress in the data storage unit 225, and the control unit 220 may determine whether or not remote access has ended based on the status information.

[0121] However, if the suppression control is already disabled at the start of the process of step S504, no particular process may be performed in step S504. Similarly, if the suppression control is already enabled at the start of the process of step S505, no particular process may be performed in step S505. The control unit 220 may determine whether to perform the processes of steps S504 and S505 based on the state of the suppression control (whether it is enabled or disabled). In this case, the information processing device 102 or the remote device 104 may store the state of the suppression control, and the control unit 220 or the control unit 244 may read the state of the suppression control. The control unit 220 or the control unit 244 may determine whether to perform the processes of steps S504 and S505 based on the read state of the suppression control. The timing for storing the state may be the timing when the state is changed, as exemplified in steps S502, S504, and S505.

[0122] In the above description, the control unit 220 performs the determinations and processes in steps S501 to S505, but this is not limiting. For example, the control unit 244 may perform the determinations and controls in steps S501 to S505.

[0123] Furthermore, the inhibitory control described in embodiment 1 is intended to prevent others from directly operating the device while the user is remotely accessing it. Therefore, the inhibitory control actually used may be any of the inhibitory controls exemplified above, or a combination of multiple inhibitory controls. Furthermore, other inhibitory methods may be used as long as they can prevent others from directly operating the remote device 104. For example, the device 105 may vibrate to alert others and inhibit direct operation. Alternatively, the device 105 may further deter operation by display output or may inform others by audio output that direct operation is not desired or is not possible.

[0124] According to the first embodiment, when remotely accessing the remote device 104, the information processing system 1, in principle, controls the remote device 104 so that it cannot be directly operated. Even in this case, if the user is able to directly operate the remote device 104, the information processing system 1 cancels the "control to prevent direct operation of the remote device 104." This allows the user to directly operate the remote device 104 during remote access, while making it difficult for others other than the user to directly operate it. Therefore, when the remote device 104 is remotely accessible, it is possible to more appropriately control whether or not the remote device 104 can be directly operated.

[0125] (Embodiment 2) In embodiment 1, when the control unit 220 starts remote access, in step S502 the control unit 220 suppresses direct operation of the remote device 104. However, the control unit 220 does not have to suppress direct operation of the remote device 104 immediately after the start of remote access. Also, in embodiment 2, step S502 may be omitted from FIG. 5.

[0126] (Embodiment 3) In Embodiments 1 and 2, in step S503, whether or not the user can directly operate the remote device 104 is determined based on whether or not the distance between the user and the remote device 104 is less than a predetermined threshold. In Embodiment 3, if the remote device 104 is present in the user's field of view obtained from the imaging unit 212, the control unit 220 may determine that the remote device 104 is in a state where the user can directly operate it. If the remote device 104 is not present in the user's field of view obtained from the imaging unit 212, the control unit 220 may determine that the remote device 104 is not in a state where the user can directly operate it.

[0127] (Embodiment 4) In embodiments 1 to 3, the control unit 220 controls whether to suppress direct operation of the remote device 104 during remote access, depending on whether the user is able to directly operate the remote device 104. In embodiment 4, the control unit 220 may suppress direct operation of the remote device 104 only when it is determined that the distance between the other person and the remote device 104 is less than a predetermined threshold. In embodiment 4, the initial state at the start of remote access is a state in which direct operation of the remote device 104 is not suppressed.

[0128] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0129] The processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, a central processing unit (CPU), a micro processing unit (MPU), and a digital signal processor (DSP). Dedicated processors include, for example, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), and a programmable logic device (PLD). Examples of programmable logic devices include a field programmable gate array (FPGA) and a complex programmable logic device (CPLD).

[0130] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0131] The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit that realizes one or more functions.

[0132] The disclosure of the present embodiments includes the following configurations, methods, programs, and media. (Configuration 1) An electronic device that is a first device, comprising: a receiving unit that receives an operation from a user; and a control unit that controls an operation of a second device in response to the user's operation on the receiving unit, wherein, when controlling the operation of the second device in response to the user's operation on the receiving unit, the control unit performs a first control to suppress an operation on the second device that is different from the user's operation on the receiving unit when the user is not in a position where he or she can operate an operation member provided on the second device. (Configuration 2) The electronic device described in Configuration 1, characterized in that the state where the user is not in a position where he or she can operate the operation member refers to a state where the distance between the user and the second device is equal to or greater than a threshold. (Configuration 3) The electronic device described in Configuration 1, characterized in that the state where the user is not in a position where he or she can operate the operation member refers to a state where the second device is not in the user's field of view. (Configuration 4) The electronic device described in any of Configurations 1 to 3, characterized in that, in the first control, the control unit prevents an operation on the second device that is different from the user's operation on the receiving unit. (Configuration 5) An electronic device that is a second device communicating with a first device having a receiving means for receiving an operation from a user, the electronic device comprising: second receiving means for receiving an operation on an operating member provided on the second device; and control means for performing first control to suppress an operation on the second device that is different from the operation on the receiving means from the user when controlling an operation of the second device in accordance with an operation from the user on the receiving means, when the user is not in a position where he or she can operate the operating member. (Configuration 6) The electronic device according to Configuration 5, wherein the state where the user is not in a position where he or she can operate the operating member is a state where the distance between the user and the second device is equal to or greater than a threshold.(Configuration 7) The electronic device according to Configuration 5, wherein the state in which the user is not in a position where the operating member can be operated by the user refers to a state in which the second device is not present in the user's field of vision. (Configuration 8) The electronic device according to any one of Configurations 5 to 7, wherein the control means, in the first control, prevents an operation on the second device that is different from an operation from the user on the accepting means from being performed. (Configuration 9) The electronic device according to any one of Configurations 1 to 8, wherein the first control is processing to prevent an operation on the second device from being used to control the operation of the second device, even if an operation is performed on the second device. (Configuration 10) An information processing system comprising: the electronic device according to any one of Configurations 1 to 4; and the second device; wherein the first control instructs the second device to issue a predetermined notification to prevent people other than the user from operating the second device; and the second device has notification means for issuing the predetermined notification in response to the first control. (Configuration 11) An information processing system comprising the electronic device according to any one of configurations 5 to 8 and the first device, wherein the first control is to instruct a predetermined notification to prevent people other than the user from operating the second device, and the second device has notification means for issuing the predetermined notification in response to the first control. (Configuration 12) The information processing system according to configuration 10 or 11, wherein the notification means is first audio output means for issuing the predetermined notification by emitting a sound. (Configuration 13) The information processing system according to configuration 10 or 11, wherein the notification means is first display means for issuing the predetermined notification by displaying a specific display item. (Configuration 14) The information processing system according to configuration 10 or 11, wherein the second device is connected to a third device, and the notification means is vibration instruction means for issuing the predetermined notification by controlling the third device to vibrate.(Configuration 15) The information processing system according to Configuration 10 or 11, wherein the second device is connected to a third device, and the notification means is audio output instruction means that performs the predetermined notification by controlling the third device to emit a sound. (Configuration 16) The information processing system according to Configuration 10 or 11, wherein the second device is connected to a third device, and the notification means is display instruction means that performs the predetermined notification by controlling the third device to display a specific display item. (Method 1) A control method for an electronic device that is a first device, comprising: a receiving step of receiving an operation from a user; and a control step of controlling an operation of a second device in accordance with the operation from the user received in the receiving step, wherein, when the operation of the second device is controlled in accordance with the operation from the user received in the receiving step, the control step performs first control to suppress an operation on the second device that is different from the operation from the user received in the receiving step when the user is not in a position where he or she can operate an operation member provided on the second device. (Method 2) A control method for an electronic device that is a second device that communicates with a first device, comprising: an acquisition step of acquiring information on an operation from a user accepted by the first device; and a control step of performing first control to suppress an operation on the second device that is different from the operation from the user accepted by the first device, when controlling the operation of the second device in accordance with the operation from the user accepted by the first device, when the user is not in a position where he or she can operate an operation member provided on the second device. (Program) A program for causing a computer to execute each step of the control method. (Medium) A computer-readable storage medium storing a program for causing a computer to execute each step of the control method.

[0133] The present invention is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to apprise the public of the scope of the present invention.

[0134] This application claims priority based on Japanese Patent Application No. 2024-104044, filed on June 27, 2024, the entire contents of which are incorporated herein by reference.

[0135] 10 local device 103 operation device 104 remote device 223 control unit

Claims

1. An electronic device that is a first device, comprising: a receiving means for receiving operations from a user; and a control means for controlling the operation of a second device in response to the operation from the user on the receiving means, wherein when the control means controls the operation of the second device in response to the operation from the user on the receiving means, when the user is not in a position where he or she can operate an operating member provided on the second device, the electronic device performs a first control that suppresses operations on the second device that are different from operations on the receiving means from the user.

2. The electronic device according to claim 1, characterized in that the state in which the user is not in a position where the operating member can be operated is a state in which the distance between the user and the second device is equal to or greater than a threshold value.

3. The electronic device according to claim 1, wherein the state in which the user is not in a position where the operating member can be operated is a state in which the second device is not present in the user's field of vision.

4. The electronic device according to claim 1, characterized in that the control means, in the first control, prevents operations on the second device that are different from operations from the user on the reception means.

5. An electronic device that is a second device that communicates with a first device having a receiving means for receiving operations from a user, characterized in that it has a second receiving means for receiving operations on an operating member provided on the second device, and a control means that performs first control to suppress operations on the second device that are different from operations on the receiving means from the user when controlling the operation of the second device in accordance with operations from the user on the receiving means, when the user is not in a position where they can operate the operating member.

6. The electronic device according to claim 5, wherein the state in which the user is not in a position where the operating member can be operated is a state in which the distance between the user and the second device is equal to or greater than a threshold value.

7. The electronic device according to claim 5, wherein the state in which the user is not in a position where the operating member can be operated is a state in which the second device is not present in the user's field of vision.

8. The electronic device according to claim 5, wherein the control means, in the first control, prevents operations on the second device that are different from operations from the user on the reception means.

9. An electronic device as described in any one of claims 1 to 8, characterized in that the first control is a process that prevents an operation on the second device from being used to control the operation of the second device, even if an operation is performed on the second device.

10. An information processing system comprising the electronic device of claim 1 and the second device, wherein the first control instructs the second device to issue a predetermined notification to prevent people other than the user from operating the second device, and the second device has notification means for issuing the predetermined notification in response to the first control.

11. An information processing system comprising the electronic device of claim 4 and the first device, wherein the first control is to instruct a predetermined notification to prevent people other than the user from operating the second device, and the second device has notification means for issuing the predetermined notification in response to the first control.

12. The information processing system according to claim 10 or 11, wherein the notification means is a first audio output means that issues the predetermined notification by emitting a sound.

13. The information processing system according to claim 10 or 11, wherein the notification means is a first display means that performs the predetermined notification by displaying a specific display item.

14. The information processing system according to claim 10 or 11, characterized in that the second device is connected to a third device, and the notification means is a vibration instruction means that performs the specified notification by controlling the third device to vibrate.

15. An information processing system as described in claim 10 or 11, characterized in that the second device is connected to a third device, and the notification means is a voice output instruction means that performs the specified notification by controlling the third device to emit a sound.

16. The information processing system according to claim 10 or 11, characterized in that the second device is connected to a third device, and the notification means is a display instruction means that performs the specified notification by controlling the third device to display a specific display item.

17. A control method for an electronic device that is a first device, comprising: a reception step of receiving an operation from a user; and a control step of controlling the operation of a second device in accordance with the operation from the user received in the reception step, wherein in the control step, when the operation of the second device is controlled in accordance with the operation from the user received in the reception step, if the user is not in a position where he or she can operate an operating member provided on the second device, a first control is performed to suppress an operation on the second device that is different from the operation from the user received in the reception step.

18. A control method for an electronic device that is a second device that communicates with a first device, comprising: a second receiving step of receiving an operation on an operating member provided on the second device; an acquisition step of acquiring information on the operation from the user received by the first device; and a control step of performing a first control to suppress an operation on the second device that is different from the operation from the user received by the first device when controlling the operation of the second device in accordance with the operation from the user received by the first device, when the user is not in a position where he or she can operate the operating member.

19. A program for causing a computer to execute each step of the control method according to claim 17 or 18.

20. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 17 or 18.

Images