Electronic device, method for controlling electronic device, and storage medium

Abstract

To provide a technology for enabling a user to easily confirm a real space at a suitable timing.SOLUTION: According to an aspect of the present disclosure, there is provided an electronic apparatus including a control unit configured to control a head-mounted display device to provide a first three dimensional space to a user, and a detection unit configured to detect a predetermined situation in which the user confirms a specific real object used by the user, wherein, in a case where the predetermined situation is detected, the control unit performs control to switch the three dimensional space provided to the user by the head-mounted display device from the first three dimensional space to a second three dimensional space having higher visibility of a real space than the first three dimensional space.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an electronic device, a control method for an electronic device, and a program. [Background technology]

[0002] There is known technology that provides users with a virtual reality space (virtual space), which is referred to as VR (Virtual Reality) technology. There is also known technology that provides users with a mixed reality space that seamlessly combines real space and virtual reality space in real time, which is referred to as MR (Mixed Reality) technology. Users can experience virtual reality space (VR space) or mixed reality space (MR space) using, for example, an HMD (Head Mounted Display).

[0003] Users experiencing a VR space wearing an HMD cannot easily check the real space, and must remove the HMD from their head or switch the 3D space they are experiencing from VR to MR in order to check the real space. Users experiencing an MR space also cannot easily check the real space, depending on the MR space, and must remove the HMD from their head in order to check the real space.

[0004] Patent document 1 discloses a technology that, when a user's action of attempting to perform an information input operation on an information processing device existing in real space is detected, displays an image of the information processing device detected from real space on an HMD. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-9647 Summary of the Invention [Problem to be solved by the invention]

[0006] However, a user may search for an information processing device or move closer to the information processing device before attempting to perform an information input operation. In such cases, with the technology disclosed in Patent Document 1, the user must remove the HMD from their head in order to search for the information processing device or move closer to the information processing device.

[0007] The present invention aims to provide a technology that enables a user to easily check a real space at a suitable timing. [Means for solving the problem]

[0008] The electronic device of the present invention has a control means for controlling a head-mounted display device to provide a first three-dimensional space to a user, and a detection means for detecting a predetermined situation in which the user confirms a specific real object used by the user, and is characterized in that when the predetermined situation is detected, the control means controls the three-dimensional space provided by the head-mounted display device to be switched from the first three-dimensional space to a second three-dimensional space in which the visibility of the real space is higher than that of the first three-dimensional space. [Effects of the Invention]

[0009] This allows the user to easily check the real space at an appropriate time. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of an information processing system. [Figure 2] FIG. 1 is a block diagram of an information processing system. [Figure 3] FIG. 2 is a diagram illustrating a software configuration of the information processing device. [Figure 4] FIG. 1 is a schematic diagram of a real space. [Figure 5] FIG. 2 is a schematic diagram of a display image. [Figure 6]FIG. 2 is a schematic diagram of a display image according to the first embodiment. [Figure 7] 4 is a flowchart showing an operation example according to the first embodiment. [Figure 8] FIG. 10 is a schematic diagram of a display image according to a second embodiment. [Figure 9] 10 is a flowchart showing an operation example according to the second embodiment. [Figure 10] FIG. 10 is a schematic diagram of a display image according to a third embodiment. [Figure 11] 10 is a flowchart showing an operation example according to the third embodiment. [Figure 12] FIG. 10 is a schematic diagram of a display image according to a fourth embodiment. [Figure 13] 10 is a flowchart showing an example of operation according to the fourth embodiment. [Figure 14] 10 is a flowchart showing an example of operation according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] (System configuration) An information processing system according to this embodiment will be described with reference to Fig. 1. The information processing system in Fig. 1 includes an HMD (Head Mounted Display) 100, an information processing device 110, and a controller 120.

[0013] The HMD 100 is an example of a head-mounted display device that can be attached to or detached from the user's head. The HMD 100 can provide the user with a three-dimensional space such as a virtual reality space (VR space) or an augmented reality space (MR space). For example, the HMD 100 displays (presents to the user) a composite image, as an image of the MR space, that combines a captured image of the area in front of the HMD 100 (user) in real space with content such as CG in a form that corresponds to the posture of the HMD 100. The HMD 100 can also display various screens.

[0014] The information processing device 110 is an example of an electronic device to which the present invention is applied, and controls the HMD 100. For example, the information processing device 110 is a personal computer (PC), a smartphone, a tablet terminal, or the like. The information processing device 110 is connected to the HMD 100 via a wired or wireless connection. There is no particular limitation on the cable used for the wired connection, and a USB cable, for example, is used. There is also no particular limitation on the wireless connection method, and for example, Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark) is used. The information processing device 110 transmits an image to be displayed on the HMD 100 (display image) to the HMD 100. For example, the HMD 100 generates a composite image by combining a captured image and CG, and transmits the composite image to the HMD 100. Note that each component of the information processing device 110 may be provided in the HMD 100. In other words, the electronic device to which the present invention is applied may be the HMD 100.

[0015] The controller 120 performs various controls of the HMD 100. When a user operates the controller 120, the HMD 100 is controlled in accordance with the user's operation. The controller 120 can be worn and supported by a user's finger, as shown in FIG. The controller 120 is a ring-shaped device. The controller 120 has physical buttons. The user can change the pointed position in the image (three-dimensional space or screen) displayed on the HMD 100 by moving the controller 120. The HMD 100 may display a dot indicating the pointed position, or a virtual ray connecting the pointed position and the controller 120 with a straight line (line segment) or dotted line. For example, the user can confirm or select an item on a menu screen by pressing a button on the controller 120 while the pointed position is over the item. The controller 120 communicates wirelessly with the information processing device 110 via Bluetooth (registered trademark).

[0016] Note that the controller 120 is not limited to a ring-shaped device, and may be, for example, another device that is pointed at by a finger, a hand, or an arm. For example, the controller 120 may be a handheld device that is held in the hand. The controller 120 may be attached to an object held in the hand (for example, an object simulating a tool), and position information and orientation information of the attachment position may be acquired from a sensor to detect the movement of the controller 120. Multiple controllers may be used.

[0017] Furthermore, the operating members of the controller 120 are not limited to physical buttons, but may be, for example, a trackpad, a touch panel, a wheel, a trackball, etc. User operations are not limited to pressing buttons, but may be, for example, slide operations, flick operations, touch operations, etc.

[0018] Furthermore, the communication of the controller 120 is not limited to wireless communication using Bluetooth (registered trademark). The controller 120 may perform wireless communication using another communication method, or may perform wired communication. The controller 120 may also communicate with the HMD 100.

[0019] (Internal structure of HMD) The internal configuration of the HMD 100 will be described with reference to Fig. 2. The HMD 100 has an HMD control unit 201, an imaging unit 202, an image display unit 203, a posture sensor unit 204, a non-volatile memory 205, a working memory 206, a line-of-sight imaging unit 207, and a power control unit 208.

[0020] The HMD control unit 201 is a CPU that controls each unit of the HMD 100. When the HMD control unit 201 acquires a display image (for example, the above-mentioned composite image) from the information processing device 110, the HMD control unit 201 displays the display image on the image display unit 203. Note that instead of the HMD control unit 201 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing.

[0021] The imaging unit 202 is a camera (image capturing device) that captures images of real space to be used for compositing with images of virtual space (e.g., CG) and for generating position and orientation information (position and orientation information) of the HMD 100. The imaging unit 202 has an imaging unit for the left eye and an imaging unit for the right eye. The imaging unit for the left eye captures moving images of real space corresponding to the left eye of the user wearing the HMD 100 and outputs images of each frame of the moving images (captured images). The imaging unit for the right eye captures moving images of real space corresponding to the right eye of the user wearing the HMD 100 and outputs images of each frame of the moving images (captured images). In other words, the imaging unit 202 captures stereo images having parallax corresponding to the parallax between the left eye and right eye of the user wearing the HMD 100. The imaging unit 202 can also acquire information about the distance from the imaging unit 202 to a subject (distance information) by known distance measurement using a stereo camera. It is preferable that the optical axis direction of the left eye imaging unit (the direction of the optical axis passing through the center of the imaging range) approximately coincides with the line of sight of the left eye, and the optical axis direction of the right eye imaging unit approximately coincides with the line of sight of the right eye, especially when providing an MR space to the user.

[0022] Each of the left-eye imaging unit and the right-eye imaging unit has an optical system and an imaging device. Light from the outside world is irradiated onto the imaging device via an optical system, and the imaging device outputs a captured image corresponding to the irradiated light. The imaging unit 202 outputs the two captured images, one on the left and one on the right, to the HMD control unit 201, or outputs them to the information processing device 110 via the HMD control unit 201.

[0023] The image display unit 203 displays a display image (for example, the above-described composite image). The image display unit 203 has a display panel such as a liquid crystal panel or an organic EL panel. The image display unit 203 has an image display unit for a left eye and an image display unit for a right eye. The image display unit for the left eye is disposed in front of the user's left eye when the user is wearing the HMD 100. The image display unit 203 can display an image of an MR space using MR (Mixed Reality) technology, or an image of a VR space using VR (Virtual Reality) technology without using captured images. Note that the image display unit 203 may be a device using a semi-transparent half mirror. In this case, for example, the image display unit 203 can display CG as if it were superimposed on a real space visible through a half mirror using AR (Augmented Reality) technology. AR technology can be considered a type of MR technology. There are no particular limitations on the method by which the image display unit 203 presents the display image to the user. For example, the image display unit 203 may use a laser or the like to directly project the display image onto the user's retina.

[0024] The orientation sensor unit 204 acquires position and orientation information of the HMD 100. The orientation sensor unit 204 may also acquire position and orientation information of a user (a user wearing the HMD 100) that corresponds to the position and orientation of the HMD 100. The orientation sensor unit 204 has an inertial measurement unit (IMU) that includes an acceleration sensor, an angular acceleration sensor, a geomagnetic sensor, and the like. The orientation sensor unit 204 acquires the position and orientation information of the HMD 100, and the HMD control unit 201 outputs the position and orientation information to the information processing device 110. The position and orientation information is acquired using, for example, at least one of a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, an acceleration sensor, and an angular velocity sensor.

[0025] The HMD control unit 201 estimates the feature points and posture of the user's hand and fingers from the stereo images (two captured images, one on each side) acquired by the imaging unit 202. For example, the feature points estimated include the positions of the finger joints, the positions of the fingertips, a representative position on the back of the hand (palm) (e.g., the center of the back of the hand), and a representative position on the arm (e.g., the positions of both ends of the arm). The coordinates of the feature points are acquired as feature point information, and the posture of the hand and fingers can be estimated from the information on multiple feature points. The method for estimating the feature points and posture of the hand and fingers is not particularly limited. For example, the feature points and posture can be estimated using various publicly known techniques for object recognition and pose estimation. A trained model may be generated in advance by machine learning using a convolutional neural network, and the feature points and posture may be estimated using the trained model. The depth coordinates of the feature points can be estimated, for example, by performing stereo matching using the stereo images (two captured images, one on each side) acquired by the imaging unit 202. Based on the stereo matching results, the distance from the imaging unit 202 to the feature points can be calculated using the principle of triangulation. The estimated feature points and posture information are output from the HMD control unit 201 to the information processing device 110.

[0026] The nonvolatile memory 205 is an electrically erasable and recordable nonvolatile memory, and stores programs executed by the HMD control unit 201 and the like.

[0027] The work memory 206 is used as a buffer memory for temporarily storing image data captured by the image capturing unit 202, an image display memory for the image display unit 203, a work area for the HMD control unit 201, and the like.

[0028] The gaze imaging unit 207 acquires images for detecting the gaze of the user. The gaze imaging unit 207 has a left-eye gaze imaging unit and a right-eye gaze imaging unit. The left-eye gaze imaging unit is a camera that captures an image of the left eye of the user wearing the HMD 100, and is attached inside the HMD 100. The right-eye gaze imaging unit is a camera that captures an image of the right eye of the user wearing the HMD 100, and is attached inside the HMD 100. The captured images of the left and right eyes captured by the gaze imaging unit 207 are output to a control unit 211 of the information processing device 110 via the HMD control unit 201. The control unit 211 detects the gaze of the user wearing the HMD 100 from the images captured by the gaze imaging unit 207, and identifies the user's gaze position in the three-dimensional space (VR space or MR space) or on the screen displayed by the image display unit 203.

[0029] The power supply control unit 208 is composed of a power supply detection circuit, a DC-DC converter, a switch circuit for switching between power-supply blocks, a charging circuit, etc., and detects whether a power supply is attached, the type of power supply, and the remaining battery level, and charges the battery. The power supply control unit 208 also controls the DC-DC converter based on the detection results and instructions from the HMD control unit 201, and supplies the necessary voltage for the necessary period to the components of the HMD 100. The power supply is composed of a secondary battery such as a Li battery, an AC adapter, etc.

[0030] Note that power may be supplied from the information processing device 110 to the HMD 100. This allows the weight of the HMD 100 to be reduced. Furthermore, multiple power sources may be used for the HMD 100, such as a power source on the information processing device 110 side and a power source on the HMD 100 side. This allows the HMD 100 to continue operating with power supplied from another power source when power supply from one power source is stopped.

[0031] (Internal structure of the controller) The internal configuration of the controller 120 will be described with reference to Fig. 2. The controller 120 has a controller control unit 221, an operation unit 222, a communication unit 223, a controller attitude sensor unit 224, an output unit 225, and a power control unit 226.

[0032] The controller control unit 221 is a CPU that controls each unit of the controller 120. Instead of the controller control unit 221 controlling the entire device, the entire device may be controlled by having a plurality of hardware devices share the processing.

[0033] The operation unit 222 includes operation members such as buttons. The operation unit 222 detects a user operation such as pressing a button, and transmits the detection result to the information processing device 110 via the communication unit 223. The operation unit 222 may have one operation member or multiple operation members.

[0034] The communication unit 223 performs wireless communication with the information processing device 110 using Bluetooth (registered trademark). When multiple controllers are used, each controller may communicate with the information processing device 110.

[0035] The controller attitude sensor unit 224 has an inertial measurement unit (IMU) that is configured with an acceleration sensor, an angular acceleration sensor, a geomagnetic sensor, etc. The inertial measurement unit detects changes in the position and attitude of the controller 120. Information about the detected changes in position and attitude is transmitted from the communication unit 223 to the information processing device 110 via the controller control unit 221.

[0036] The output unit 225 is configured by a light source such as an LED, a speaker, a vibration element, and the like.

[0037] The power supply control unit 226 is composed of a power supply detection circuit, a DC-DC converter, a charging circuit, etc., and detects whether a power supply is attached, the type of power supply, and the remaining battery capacity, and charges the battery. The power supply control unit 226 also controls the DC-DC converter based on the detection results and instructions from the controller control unit 221, and supplies the necessary voltage for the necessary period to the components of the controller 120. The power supply is composed of primary batteries such as alkaline batteries or lithium batteries, secondary batteries such as Li batteries, etc.

[0038] (Internal configuration of information processing device) 2, the internal configuration of the information processing device 110 will be described. The information processing device 110 has a control unit 211, a nonvolatile memory 212, a working memory 213, a communication unit 214, and a recording medium 215.

[0039] The control unit 211 is a CPU that controls each unit of the information processing device 110 in accordance with input signals and a program described later. Instead of the control unit 211 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing. The control unit 211 receives, from the HMD 100, an image (captured image in real space) acquired by the imaging unit 202 and position and orientation information acquired by the orientation sensor unit 204. The control unit 211 performs image processing on the captured image in real space to cancel aberrations in the optical system of the imaging unit 202 and the optical system of the image display unit 203. The control unit 211 then combines the captured image in real space with any CG to generate a composite image, which is an image in MR space. The control unit 211 can also generate images of other three-dimensional spaces. For example, the control unit 211 can acquire data of a VR space from the recording medium 215, the nonvolatile memory 212, or the like, and generate an image of the VR space based on the data. The control unit 211 transmits the generated image to the HMD control unit 201 in the HMD 100 as a display image.

[0040] Furthermore, the control unit 211 detects the controller 120 from the captured image, and recognizes the position and orientation of the controller 120 using information obtained via the communication unit 214 (information on changes in the position and orientation of the controller 120). The control unit 211 then performs control according to the recognition result. The control unit 211 may use the information obtained via the communication unit 214 to generate a display image including a point or ray indicating the indicated position.

[0041] The control unit 211 controls the position, orientation, and size of the CG in the display image based on the position and orientation information of the HMD 100. Consider a case where a CG virtual object is placed near a specific real object (a specific object existing in the real space) in the MR space represented by the composite image. In this case, the control unit 211 renders an image of the virtual object at a larger size as the distance between the specific real object and the imaging unit 202 is closer. By controlling the position, orientation, and size of the CG in this manner, the control unit 211 can generate a composite image in which a CG virtual object that is not placed in the real space appears to be placed in the real space. Furthermore, the control unit 211 can generate an image of the VR space that allows the user to experience as if they were in the VR space.

[0042] The control unit 211 also receives information estimated by the HMD control unit 201 (information on feature points and posture of the user's hands and fingers) and temporarily stores it in the working memory 213. The control unit 211 can accept gesture operations, for example, based on time-series information on feature points and posture. The control unit 211 performs, for example, image processing in response to the gesture operations. The control unit 211 may connect to a smartphone, a PC, or the like via the communication unit 214, receive information on the screen of the smartphone or PC, and display the screen on the HMD 100.

[0043] The feature points and posture of the user's hand and fingers may be estimated by the information processing device 110. For example, the control unit 211 of the information processing device 110 may estimate the feature points and posture of the user's hand and fingers based on the captured image captured by the imaging unit 202.

[0044] The nonvolatile memory 212 is an electrically erasable and recordable nonvolatile memory, and stores information such as programs to be described later executed by the control unit 211, CG, etc. The control unit 211 can switch the CG (CG used to generate images in the MR space or VR space) read from the nonvolatile memory 212.

[0045] The working memory 213 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202, time-series information on feature points and posture estimated by the HMD control unit 201, etc. The working memory 213 is also used as an image display memory for the image display unit 203, a working area for the control unit 211, etc.

[0046] Note that processing (control) other than the above-described processing (control) may be further performed. For example, the HMD control unit 201, the control unit 211 of the information processing device 110, etc. may perform various image processing on the captured image acquired by the imaging unit 202 and the display image displayed on the image display unit 203.

[0047] (Software configuration of information processing device) The software configuration of the information processing device 110 will be described with reference to FIG. 3. The platform 302 can be configured on an OS 301 such as Microsoft Windows (registered trademark), Linux (registered trademark), or Google Android (registered trademark). The platform 302 provides an API for an application 303 running on the platform 302 to use various hardware. The platform 302 also provides an API for the application 303 to use position and orientation information of the HMD 100 estimated based on various sensors and captured images, information on feature points and orientations of the hands and fingers, and information on the distance to the subject. Various applications 303 can be installed in the information processing device 110 and run on the OS 301 or platform 302.

[0048] (Real space and displayed image) 4, a real space 400 around a user wearing an HMD 100 will be described. The real space 400 includes a user 401, the HMD 100 worn by the user 401, and a desk 402. The user 401 is wearing the HMD 100 and is looking in the direction of an arrow 403.

[0049] With reference to FIGS. 5(A) and 5(B), a display image displayed on the HMD 100 of FIG. 4 will be described. FIG. 5(A) shows an image 510 in a VR space. In the image 510, a virtual object 511 and a virtual object 512 exist. The virtual object 512 exists at the position of the desk 402. The image 510 is composed only of CG, and when the image 510 is displayed, the user cannot easily confirm the real space. FIG. 5(B) shows an image 520 in an MR space. In the image 520, a virtual object 521 and the desk 402, which is a real object, exist. In the image 520, the image of the virtual object 521 is superimposed on the image of the real space.

[0050] (First embodiment) The operation according to the first embodiment will be described.

[0051] When an image of a VR space is displayed on the HMD 100, the remaining battery power of the information processing device 110 or its external device (for example, the HMD 100 or the controller 120) falls below a threshold. When the remaining battery power of the information processing device 110 or its external device becomes low, the user needs to check the real objects such as the device with the low remaining battery power and its battery. Therefore, in the first embodiment, in such a situation, the display image displayed on the HMD 100 is switched from an image in the VR space to an image in the MR space.

[0052] In the first embodiment, the control unit 211 sets one of a plurality of display modes including a VR mode and an MR mode. The VR mode is a display mode in which an image in a VR space such as image 510 in FIG. 5A is displayed on the HMD 100, and the MR mode is a display mode in which an image in an MR space such as image 520 in FIG. 5B is displayed on the HMD 100. The control unit 211 switches the set display mode from the VR mode to the MR mode, thereby switching the displayed image from an image in the VR space to an image in the MR space.

[0053] With reference to FIG. 6(A), the operation when the remaining battery charge of the controller 120 in the VR mode falls below a threshold will be described.

[0054] The controller control unit 221 can detect, using the power supply control unit 226, that the remaining battery power of the controller 120 has fallen below a threshold. When the controller control unit 221 detects that the remaining battery power of the controller 120 has fallen below a threshold, it notifies the control unit 211 of the information processing device 110 via the communication unit 223 that the remaining battery power of the controller 120 is low. As a result, the control unit 211 detects that the remaining battery power of the controller 120 has fallen below a threshold.

[0055] Then, in response to the notification from the controller control unit 221, the control unit 211 switches the set display mode from VR mode to MR mode and displays the image of the MR space shown in Fig. 6(A). The image in Fig. 6(A) includes a screen 601. The screen 601 includes text indicating that the remaining battery charge of the controller 120 is low and text urging the user to charge the controller 120 or replace the battery.

[0056] By switching the set display mode to the MR mode, the user can view the real space while wearing the HMD 100, and can easily charge the controller 120 or replace the battery.

[0057] When the controller control unit 221 detects that the state in which the remaining battery charge of the controller 120 is below the threshold has been resolved, the controller control unit 221 notifies the control unit 211 of the resolution of the state, using the power supply control unit 226. Then, in response to the notification from the controller control unit 221, the control unit 211 changes the set display mode back from the MR mode to the VR mode.

[0058] The following describes the operation when a power cable is disconnected from the information processing device 110 or its external device (for example, the HMD 100 or the controller 120) while an image of a VR space is displayed on the HMD 100. When a power cable is disconnected from the information processing device 110 or its external device, the user needs to check the device whose power cable has been disconnected and the real object such as the power cable. Therefore, in the first embodiment, in such a situation, the displayed image is switched from an image of the VR space to an image of the MR space.

[0059] 6(B), a description will be given of the operation when the power cable is disconnected from the HMD 100 in the VR mode. Note that even if the power cable is disconnected from the HMD 100, the HMD 100 can continue to operate on the battery.

[0060] The HMD control unit 201 uses the power supply control unit 208 to determine whether the power cable is connected to the HMD 100. When the HMD control unit 201 detects that the power cable has been unplugged from the HMD 100, it notifies the control unit 211 of the information processing device 110 that the power cable has been unplugged from the HMD 100. As a result, the control unit 211 detects that the power cable has been unplugged from the HMD 100.

[0061] Then, in response to the notification from the HMD control unit 201, the control unit 211 switches the set display mode from VR mode to MR mode and displays the image of the MR space shown in Fig. 6(B). The image in Fig. 6(B) includes a screen 602. The screen 602 includes text indicating that the power cable has been unplugged from the HMD 100 and text urging the user to plug the power cable into the HMD 100.

[0062] By switching the set display mode to the MR mode, the user can view the real space while wearing the HMD 100 and can easily insert the power cable into the HMD 100.

[0063] When the HMD control unit 201 detects that the situation in which the power cable is unplugged from the HMD 100 has been resolved, the HMD control unit 201 notifies the control unit 211 of the resolution of the situation, using the power control unit 208. Then, in response to the notification from the HMD control unit 201, the control unit 211 returns the set display mode from the MR mode to the VR mode.

[0064] When switching from VR mode to MR mode, for example, the platform 302 performs one of the following three processes on the running VR application (application in VR mode). Hide the entire image in the VR application (switch the entire image to the real world image). Hide part of the image in the VR application (switch part of it to an image in the real world). Increase the transparency of the images in the VR application so that the images in the real world are transparent to the images in the VR application.

[0065] The display mode may be switched by the platform 302 notifying the application 303, or by the platform 302 changing the display image.

[0066] Although the example has been described in which the set display mode is switched from VR mode to MR mode when the remaining battery power is low or when the power cable is unplugged, the present invention is not limited to this. For example, the set display mode may be switched from VR mode to MR mode when a malfunction occurs in the information processing device 110 or its external device (for example, the HMD 100 or the controller 120).

[0067] Although the example of switching the display image from an image in VR space to an image in MR space has been described, the present invention is not limited to this. It is sufficient that the visibility of the real space in the image after switching is higher than the visibility of the real space in the image before switching. For example, an image in MR space with low visibility of the real space may be switched to an image in MR space with high visibility of the real space.

[0068] The flow of operations according to the first embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of operations according to the first embodiment. The operations shown in Fig. 7 are realized by the control unit 211 of the information processing device 110 expanding a program recorded in the nonvolatile memory 212 into the work memory 213 and executing the program. For example, when the HMD 100, the information processing device 110, and the controller 120 are started and connected to each other, the control unit 211 executes the operations shown in Fig. 7. The operation in FIG. 7 is performed repeatedly, for example, at a predetermined cycle.

[0069] In S701, the control unit 211 determines whether the remaining battery charge of the HMD 100, the information processing device 110, or the controller 120 is equal to or less than a threshold value (predetermined value). If the control unit 211 determines that the remaining battery charge is equal to or less than the threshold value (that there is a device whose remaining battery charge is equal to or less than the threshold value), the control unit 211 proceeds to S702; otherwise, the operation in FIG. 7 ends. Note that the control unit 211 may also determine whether the remaining battery charge of another external device (for example, a smartphone, earphones, a tracker, a PC, etc.) connected to the information processing device 110 is equal to or less than a threshold value.

[0070] In S702, the control unit 211 determines whether or not the VR mode is set. If the control unit 211 determines that the VR mode is set, the process proceeds to S703, and if not, the operation in FIG. 7 ends.

[0071] In S703, the control unit 211 switches the set display mode to the MR mode (sets the MR mode).

[0072] In S704, the control unit 211 determines whether the remaining battery charge of the HMD 100, the information processing device 110, or the controller 120 is equal to or less than a threshold value (predetermined value). The control unit 211 repeats the determination in S704 until it determines that the remaining battery charge is not equal to or less than the threshold value (that is, there is no device whose remaining battery charge is equal to or less than the threshold value), and if so, proceeds to S705.

[0073] In S705, the control unit 211 switches the set display mode to the VR mode (sets the VR mode).

[0074] (Second embodiment) The operation according to the second embodiment will be described.

[0075] An operation will be described when a connection screen is displayed on the HMD 100 while an image of a VR space is being displayed on the HMD 100. The connection screen is a screen that prompts a user to perform an operation on an external device (e.g., a controller 120, a smartphone, earphones, a tracker, a PC, etc.) to initiate connection between the information processing device 110 and the external device. When such a connection screen is displayed, the user needs to confirm the external device, which is a real object. Therefore, in the second embodiment, in such a situation, the display image displayed on the HMD 100 is switched from an image of the VR space to an image of the MR space. Note that the display image may be switched from an image of the VR space to an image of the MR space after the connection screen is displayed, or the connection screen may be displayed after the display image is switched from an image of the VR space to an image of the MR space. The switching of the display image and the display of the connection screen may be performed simultaneously.

[0076] FIG. 8(A) shows an example of an image displayed on the HMD 100 in VR mode. The image in FIG. 8(A) includes a connection preparation screen 801. The connection preparation screen 801 is a screen for selecting an external device (connection target) to connect to the information processing device 110, and is displayed by, for example, the application 303. When the user selects one of the multiple candidates (multiple connection target candidates) displayed on the connection preparation screen 801, the control unit 211 switches the set display mode from VR mode to MR mode and displays the MR mode image shown in FIG. 8(B). The image in FIG. 8(B) includes a connection screen 802. The connection screen 802 includes text prompting the user to perform an operation on the connection target.

[0077] By switching the configured display mode to MR mode, the user can This makes it easy to search for an external device and operate the external device.

[0078] When the connection between the information processing device 110 and the external device is completed and the connection screen 802 is hidden, the control unit 211 returns the set display mode from the MR mode to the VR mode. Note that the display mode may be returned to the VR mode after the connection screen 802 is hidden, or the connection screen 802 may be hidden after the display mode is returned to the VR mode. The resetting of the VR mode and the hiding of the connection screen 802 may be performed simultaneously.

[0079] The connection screen may be a pairing screen that prompts the user to perform a pairing operation as a user operation on the external device.

[0080] FIG. 8(C) shows an example of an image displayed on the HMD 100 in VR mode. The image in FIG. 8(C) includes a pairing preparation screen 803. The pairing preparation screen 803 is a screen for selecting an external device (pairing target) to connect to the information processing device 110 via Bluetooth (registered trademark), and is displayed by, for example, the application 303. When the user selects one of the multiple candidates (multiple pairing target candidates) displayed on the pairing preparation screen 803, the control unit 211 switches the set display mode from VR mode to MR mode and displays the MR mode image shown in FIG. 8(D). The image in FIG. 8(D) includes a pairing screen 804. The pairing screen 804 includes text prompting the user to perform an operation on the pairing target.

[0081] By switching the set display mode to the MR mode, the user can easily perform tasks such as searching for an external device to be paired and operating the external device.

[0082] When the connection between the information processing device 110 and the external device is completed and the pairing screen 804 is hidden, the control unit 211 returns the set display mode from the MR mode to the VR mode. Note that the display mode may be returned to the VR mode after the pairing screen 804 is hidden, or the pairing screen 804 may be hidden after the display mode is returned to the VR mode. Resetting the VR mode and hiding the pairing screen 804 may be performed simultaneously.

[0083] In addition, the set display mode may be switched from VR mode to MR mode when displaying other screens (specified screens) on which the user can check specific real-world objects used by the user, not limited to the connection screen (including the pairing screen).

[0084] The flow of operations according to the second embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of operations according to the second embodiment. The operations shown in Fig. 9 are realized by the control unit 211 of the information processing device 110 expanding a program recorded in the nonvolatile memory 212 into the work memory 213 and executing the program. For example, when the HMD 100 and the information processing device 110 are started and connected to each other, the control unit 211 starts the operations shown in Fig. 9. The operations shown in Fig. 9 are performed repeatedly, for example, at a predetermined cycle.

[0085] In S901, the control unit 211 determines whether or not a connection screen is being displayed on the HMD 100. If the control unit 211 determines that a connection screen is being displayed, the process proceeds to S902, and if not, the operation in FIG. 9 ends.

[0086] In S902, the control unit 211 determines whether or not the VR mode is set. If the control unit 211 determines that the VR mode is set, the process proceeds to S903, and if not, the operation in FIG. 9 ends.

[0087] In S903, the control unit 211 switches the set display mode to the MR mode (sets the MR mode).

[0088] In S904, the control unit 211 determines whether or not the connection screen is displayed on the HMD 100. The control unit 211 repeats the determination in S904 until it determines that the connection screen is not displayed, and if so, proceeds to S905.

[0089] In S905, the control unit 211 switches the set display mode to the VR mode (sets the VR mode).

[0090] (Third embodiment) The operation according to the third embodiment will be described.

[0091] The following describes an operation when a password entry screen that prompts a user to enter a password is displayed on the HMD 100 while an image of a VR space is displayed on the HMD 100. When the password entry screen is displayed, the user needs to check real objects such as a note on which the password is written or a person nearby who may be peeking. Therefore, in the third embodiment, in such a situation, the display image displayed on the HMD 100 is switched from an image of the VR space to an image of the MR space. Note that the display image may be switched from an image of the VR space to an image of the MR space after the password entry screen is displayed, or the password entry screen may be displayed after the display image is switched from an image of the VR space to an image of the MR space. The switching of the display image and the display of the password entry screen may be performed simultaneously.

[0092] With reference to Fig. 10, an operation when a password entry screen is displayed in VR mode will be described. When a user instructs the control unit 211 to display the password entry screen, the control unit 211 switches the set display mode from VR mode to MR mode and displays an image of the MR space shown in Fig. 10. The image in Fig. 10 includes a password entry screen 1001. The password entry screen 1001 includes a text box that displays the entered password in obscured characters. Note that the method for detecting the display of the password entry screen is not particularly limited. For example, the platform 302 may detect obscured characters on the displayed screen via the OS 301, thereby detecting the screen as a password entry screen.

[0093] By switching the set display mode to MR mode, the user can view the real space while wearing the HMD100, easily performing tasks such as checking a note containing a password or making sure that there are no people around.

[0094] When the password entry is completed and the password entry screen 1001 is hidden, the control unit 211 returns the set display mode from the MR mode to the VR mode. Note that the display mode may be returned to the VR mode after the password entry screen 1001 is hidden, or the password entry screen 1001 may be hidden after the display mode is returned to the VR mode. The resetting of the VR mode and the hiding of the password entry screen 1001 may be performed simultaneously.

[0095] The flow of operations according to the third embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of operations according to the third embodiment. The operations shown in Fig. 11 are realized by the control unit 211 of the information processing device 110 expanding a program recorded in the nonvolatile memory 212 into the work memory 213 and executing the program. For example, when the HMD 100, the information processing device 110, and the controller 120 are started and connected to each other, the control unit 211 starts the operations shown in Fig. 11. The operations shown in Fig. 11 are performed repeatedly, for example, at a predetermined cycle.

[0096] In S1101, the control unit 211 determines whether or not a password entry screen is displayed on the HMD 100. If the control unit 211 determines that a password entry screen is displayed, the process proceeds to S1102; otherwise, the operation of FIG. 11 ends.

[0097] In S1102, the control unit 211 determines whether or not the VR mode is set. If the control unit 211 determines that the VR mode is set, the process proceeds to S1103, and if not, the operation of FIG. 11 ends.

[0098] In S1103, the control unit 211 switches the set display mode to the MR mode (sets the MR mode).

[0099] In S1104, the control unit 211 determines whether or not a password entry screen is displayed on the HMD 100. The control unit 211 repeats the determination in S1104 until it determines that the password entry screen is not displayed, and if so, proceeds to S1105.

[0100] In S1105, the control unit 211 switches the set display mode to the VR mode (sets the VR mode).

[0101] (Fourth embodiment) An operation according to the fourth embodiment will be described. In the fourth embodiment, an external device of the information processing device 110 detects a predetermined situation in which the user checks a specific real object used by the user, and outputs a predetermined signal. The external device is, for example, the HMD 100, the controller 120, a smartphone, earphones, a tracker, a PC, or the like. Then, the control unit 211 of the information processing device 110 detects the predetermined situation by detecting reception of the predetermined signal.

[0102] The operation when displaying a two-step authentication screen, which is an example of a password input screen, in VR mode will be described with reference to Fig. 12. Screen 1201 in Fig. 12 is a two-step authentication screen for inputting a one-time password, and is displayed by, for example, application 303. When the two-step authentication screen 1201 is displayed, a one-time password (one-time passcode; security code) is sent to the user's smartphone using SMS (short message service) or the like, and the user checks and inputs the passcode.

[0103] The smartphone can detect the reception of the one-time password. When the smartphone detects the reception of the one-time password, it transmits a predetermined signal (for example, a control signal instructing the setting of MR mode) to the control unit 211 of the information processing device 110. As a result, the control unit 211 detects the reception of the one-time password by the smartphone. The detection of the reception of the one-time password may be interpreted as the detection of the display of the two-step authentication screen 1201.

[0104] Then, the control unit 211 switches the set display mode from the VR mode to the MR mode in response to a predetermined signal from the smartphone, and displays the image of the MR space shown in FIG.

[0105] By switching the set display mode to MR mode, the user can view the real world while wearing the HMD 100 and easily check the one-time password on their smartphone. Note that the one-time password is not transmitted from the smartphone, so the security level is not reduced.

[0106] When the input of the one-time password is completed and the two-step authentication screen 1201 is to be hidden, the control unit 211 changes the set display mode back from the MR mode to the VR mode.

[0107] The flow of operation of the smartphone according to the fourth embodiment will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of operation of the smartphone according to the fourth embodiment. The operation of Fig. 13 is performed repeatedly, for example, at a predetermined cycle. In S1301, the smartphone determines whether or not a one-time password has been received. If it is determined that a one-time password has been received, the smartphone proceeds to processing in S1302; otherwise, the smartphone ends the operation of Fig. 13. In S1302, the smartphone transmits a predetermined signal to the information processing device 110.

[0108] The flow of operations of the information processing device 110 according to the fourth embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of operations of the information processing device 110 according to the fourth embodiment. The operations of Fig. 14 are realized by the control unit 211 of the information processing device 110 expanding a program recorded in the nonvolatile memory 212 into the work memory 213 and executing the program. For example, when the HMD 100, the information processing device 110, and the controller 120 start up and are connected to each other, the control unit 211 starts the operations of Fig. 11. The operations of Fig. 11 are performed repeatedly, for example, at a predetermined cycle.

[0109] In S1401, the control unit 211 determines whether or not a predetermined signal has been received from the smartphone. If the control unit 211 determines that the predetermined signal has been received, the process proceeds to S1402; otherwise, the operation in FIG. 14 ends.

[0110] In S1402, the control unit 211 determines whether or not the VR mode is set. If the control unit 211 determines that the VR mode is set, the process proceeds to S1403, and if not, the operation of FIG. 14 ends.

[0111] In S1403, the control unit 211 switches the set display mode to the MR mode (sets the MR mode).

[0112] In S1404, the control unit 211 determines whether the screen (two-step authentication screen) that was displayed on the HMD 100 in S1401 is still being displayed. The control unit 211 repeats the determination in S1404 until it determines that the screen display is not still being displayed, and if so, proceeds to S1405.

[0113] In S1405, the control unit 211 switches the set display mode to the VR mode (sets the VR mode).

[0114] According to each of the above-described embodiments, a predetermined situation in which the user checks a specific real object used by the user is detected. Then, when the predetermined situation is detected, the 3D space provided to the user by the head-mounted display device is switched so as to enhance the visibility of the real space. This allows the user to easily check the real space at an appropriate timing.

[0115] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0116] The above processors are processors in a broad sense, and include general-purpose processors and dedicated processors. This includes processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0117] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0118] (Other embodiments) The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0119] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a control means for controlling the head-mounted display device to provide a first three-dimensional space to a user; detection means for detecting a predetermined situation in which the user recognizes a specific real object used by the user; and When the predetermined situation is detected, the control means controls the head-mounted display device to switch the three-dimensional space provided to the user from the first three-dimensional space to a second three-dimensional space in which the visibility of the real space is higher than that of the first three-dimensional space. An electronic device characterized by: (Configuration 2) the first three-dimensional space is a virtual reality space; The second three-dimensional space is a mixed reality space. 2. The electronic device according to configuration 1. (Configuration 3) The predetermined situation includes a situation where the remaining battery power of the electronic device or the remaining battery power of a first external device connected to the electronic device is below a threshold. 3. The electronic device according to configuration 1 or 2. (Configuration 4) The predetermined situation includes a situation in which a power cable is disconnected from the electronic device or a first external device connected to the electronic device. 4. The electronic device according to any one of configurations 1 to 3. (Configuration 5) The predetermined situation includes a situation in which a malfunction occurs in the electronic device or a first external device connected to the electronic device. 5. The electronic device according to any one of configurations 1 to 4. (Configuration 6) The first external device is the head-mounted display device or a controller for the head-mounted display device. controller 6. The electronic device according to any one of configurations 3 to 5. (Configuration 7) The predetermined situation includes a situation in which the head-mounted display device presents to the user a predetermined screen in which the user can see the specific real object. 7. The electronic device according to any one of configurations 1 to 6. (Configuration 8) The predetermined screen includes a connection screen that prompts a user to perform an operation on a second external device to start a connection between the electronic device and the second external device. 8. The electronic device according to configuration 7. (Configuration 9) 9. The electronic device according to configuration 7 or 8, wherein the predetermined screen includes a password entry screen that prompts the user to enter a password. (Configuration 10) a third external device is connected to the electronic device and detects the predetermined situation and outputs a predetermined signal; The detection means is capable of detecting the predetermined situation by detecting reception of the predetermined signal by the electronic device. 10. The electronic device according to any one of configurations 1 to 9. (Configuration 11) The third external device detects the reception of the one-time password and outputs the predetermined signal. 11. The electronic device according to configuration 10. (Configuration 12) When the predetermined situation is resolved, the control means controls the head-mounted display device to return the three-dimensional space provided to the user from the second three-dimensional space to the first three-dimensional space. 12. The electronic device according to any one of configurations 1 to 11. (method) controlling the head-mounted display device to present a first three-dimensional space to the user; detecting a predetermined situation in which the user observes a particular real object used by the user; When the predetermined situation is detected, controlling the head-mounted display device to switch the three-dimensional space provided to the user from the first three-dimensional space to a second three-dimensional space in which the visibility of real space is higher than that of the first three-dimensional space; 1. A method for controlling an electronic device, comprising: (program) 13. A program for causing a computer to function as each means of the electronic device according to any one of configurations 1 to 12. [Explanation of symbols]

[0120] 110: Information processing device 211: Control unit

Claims

1. a control means for controlling the head-mounted display device to provide a first three-dimensional space to a user; detection means for detecting a predetermined situation in which the user recognizes a specific real object used by the user; and When the predetermined situation is detected, the control means controls the head-mounted display device to switch the three-dimensional space provided to the user from the first three-dimensional space to a second three-dimensional space in which the visibility of the real space is higher than that of the first three-dimensional space. An electronic device characterized by:

2. the first three-dimensional space is a virtual reality space; The second three-dimensional space is a mixed reality space.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

3. The predetermined situation includes a situation where the remaining battery power of the electronic device or the remaining battery power of a first external device connected to the electronic device is below a threshold.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

4. The predetermined situation includes a situation in which a power cable is disconnected from the electronic device or a first external device connected to the electronic device.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

5. The predetermined situation includes a situation in which a malfunction occurs in the electronic device or a first external device connected to the electronic device.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

6. The first external device is the head-mounted display device or a controller for the head-mounted display device.

4. The electronic device according to claim 3.

7. The predetermined situation includes a situation in which the head-mounted display device presents to the user a predetermined screen in which the user can see the specific real object.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

8. The predetermined screen includes a connection screen that prompts a user to perform an operation on a second external device to start a connection between the electronic device and the second external device.

8. The electronic device according to claim 7,

9. 8. The electronic device according to claim 7, wherein the predetermined screen includes a password entry screen that prompts the user to enter a password.

10. a third external device is connected to the electronic device and detects the predetermined situation and outputs a predetermined signal; The detection means is capable of detecting the predetermined situation by detecting reception of the predetermined signal by the electronic device.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

11. The third external device detects the reception of the one-time password and outputs the predetermined signal. do 11. The electronic device according to claim 10.

12. When the predetermined situation is resolved, the control means controls the head-mounted display device to return the three-dimensional space provided to the user from the second three-dimensional space to the first three-dimensional space.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

13. controlling the head-mounted display device to provide a first three-dimensional space to a user; detecting a predetermined situation in which the user observes a particular real object used by the user; When the predetermined situation is detected, controlling the head-mounted display device to switch the three-dimensional space provided to the user from the first three-dimensional space to a second three-dimensional space in which the visibility of real space is higher than that of the first three-dimensional space; 1. A method for controlling an electronic device, comprising:

14. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 12.

Images