Information processing apparatus, information processing system, information processing method, and program

Abstract

To provide a system that can improve operability by minimizing the time and effort for play area settings.SOLUTION: An information processing apparatus has: acquisition means that acquires a picked-up image; and control means that performs control to selectively display, on a display, a composite image of the picked-up image acquired by the acquisition means and an image of a virtual object, and an image of a virtual space, and determines whether to set a boundary that divides a plurality of regions from each other in a real space, on the basis of whether the image displayed on the display is the composite image or the image of the virtual space.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] VR (Virtual Reality) technology is known as a technology that allows users to experience virtual spaces. MR (Mixed Reality) technology is also known as a technology that seamlessly fuses real and virtual spaces in real time. Head-mounted devices, such as HMDs (Head Mounted Displays), are used to experience these technologies. However, wearing an HMD makes it difficult to see the surroundings, which increases the risk of bumping into obstacles.

[0003] For example, Patent Document 1 discloses a method of setting the boundaries of an area (play area) in which the user can move safely in advance, and warning the user when the user approaches the boundary line of that area, thereby reducing the risk of the user colliding with an obstacle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Table 2017-535901 Summary of the Invention [Problem to be solved by the invention]

[0005] The timing for setting an area where one can move safely (play area) is often when the HMD is started up. When using VR technology, the surrounding real space cannot be seen, so setting a play area allows for safer use. However, when using MR technology, which allows the surrounding real space to be seen, the user can visually identify obstacles, reducing the risk of colliding with them, so the need to set a play area is reduced. Even when the need to set a play area is low, requiring a configuration that always requires setting a play area is cumbersome for the user. Therefore, an object of the present invention is to provide a system that can improve operability by minimizing the effort required for setting a play area. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device characterized by having an acquisition means for acquiring an image of a captured image, and a control means for selectively controlling a display unit to display a composite image of the captured image acquired by the acquisition means and an image of a virtual object, and an image of a virtual space, and for determining whether to set boundaries separating multiple areas in real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space. [Effects of the Invention]

[0007] According to the present invention, a system can be provided that can improve operability by minimizing the effort required to set up a play area. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing the internal configuration of an HMD and the like according to the first embodiment. [Figure 3] FIG. 1 is a diagram showing a real space in which a user wearing an HMD exists in the first embodiment. [Figure 4] 10A to 10C are diagrams showing an example of a series of screen displays in the first embodiment. [Figure 5] 10 is a flowchart illustrating a processing procedure for displaying a play area setting screen in accordance with an application in the first embodiment. [Figure 6] FIG. 10 is a diagram showing an example of a screen display in the second embodiment. [Figure 7] 10 is a flowchart illustrating a processing procedure for displaying a setting screen for a play area according to the proportion of a CG (virtual object) in an application started in MR mode in the second embodiment. [Figure 8] FIG. 11 is a diagram showing an example of a screen display in the third embodiment. [Figure 9] 11 is a flowchart illustrating the processing steps for starting a VR application without setting a play area when there is little movement in a scene according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] (Embodiment 1) An information processing system 1 according to the first embodiment will be described with reference to Fig. 1. The information processing system 1 includes an HMD 100, a PC (personal computer) 110, and a controller 120.

[0011] The HMD 100 is a head-mounted display device (electronic device) that can be worn on the head of a user. The HMD 100 displays a composite image that combines a captured image of the area in front of the user captured by the HMD 100 with content such as CG in a form that corresponds to the posture of the HMD 100.

[0012] The PC 110 controls the HMD 100. The PC 110 is connected to the HMD 100 via a wired connection such as a USB cable, or wirelessly via Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). The PC 110 generates a composite image by combining a captured image with CG, and transmits the composite image to the HMD 100. Note that although a PC is described here as an example of an information processing device, the information processing device is not limited to this. For example, the information processing device may be a smartphone or a tablet terminal, and each component of the PC 110 may be included in the HMD 100.

[0013] The controller 120 performs various controls of the HMD 100. When the PC 110 is in a specific control mode, a user operation on the controller 120 controls the HMD 100 in accordance with the user operation. As shown in FIG. 1, the controller 120 may be in the shape of a ring that can be worn on the user's finger or a handheld type that can be held in the hand. The controller 120 also has physical buttons for performing confirmation operations and selection operations on the display. The controller 120 communicates wirelessly with the PC 110 via Bluetooth. The controller is not limited to a system that communicates with the PC 110, but may also be a system that communicates with the HMD 100.

[0014] By moving the controller 120, the user can change the pointed position on the display in accordance with the movement of the controller. The pointed position may be represented by a point, or by a virtual ray connecting the pointed position point and the controller with a straight line (line segment) or dotted line. By pressing a physical button, a menu confirmation operation or selection operation can be performed. Note that while the controller 120 has been described as being ring-shaped or handheld, the shape is not limited to this as long as it can be supported by a finger, hand, or arm. Furthermore, although the buttons have been described as being physical buttons, they may be operable in the form of a trackpad, touch panel, wheel, or trackball, and may be operated by a slide operation, flick operation, or touch operation in addition to pressing a button.

[0015] Note that the controller may be worn on at least one of a finger, a hand, or an arm.

[0016] Note that the controller may be attached to an object held by hand, and used to acquire position information and attitude information of the attached position from a sensor. Examples of such an object include an object imitating a tool.

[0017] <Internal Structure of HMD> Referring to FIG. 2, the internal structure of the HMD 100 will be described. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an image display unit 203, an attitude sensor unit 204, a non-volatile memory 205, a working memory 206, and a gaze imaging unit 207.

[0018] The HMD control unit 201 is a CPU that controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by synthesizing an imaging image captured by the imaging unit 202 of the space in front of the user and a CG) from the PC 110, the HMD control unit 201 displays the composite image on the image display unit 203. Note that instead of the HMD control unit 201 controlling the entire device, the entire device may be controlled by a plurality of hardware sharing the processing.

[0019] The imaging unit 202 includes two cameras (imaging devices). The two cameras are used to capture images used for combining with images of the virtual space and generating position and orientation information, and include an imaging unit for the left eye and an imaging unit for the right eye. The imaging unit for the left eye captures a moving image of the real space corresponding to the left eye of the person wearing the HMD 101, and the left eye imaging unit outputs an image (captured image) of each frame of the moving image. The imaging unit for the right eye captures a moving image of the real space corresponding to the right eye of the person wearing the HMD 101, and the right eye imaging unit outputs an image (captured image) of each frame of the moving image. In other words, the imaging unit 202 acquires captured images as stereo images with parallax that approximately matches the positions of the left and right eyes of the person wearing the HMD 101. Furthermore, distance information can be acquired from distance measurement using the stereo cameras, indicating the distance from the two cameras to the subject. Note that in an HMD for an MR system, it is preferable to arrange the imaging unit so that the central optical axis of the imaging range of the imaging unit approximately matches the line of sight of the person wearing the HMD.

[0020] The left-eye imaging unit and the right-eye imaging unit each have an optical system and an imaging device. Light entering from the outside enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the incoming light as a captured image. The images of the subject (the area in front of the user) captured by the two cameras are output to the PC 110 and the control unit 201. Note that the imaging unit 202 may capture and output video instead of captured images.

[0021] The image display unit 203 displays a composite image. The image display unit 203 includes a liquid crystal panel, an organic EL panel, or the like. When the user wears the HMD 100, the image display unit 203 is disposed in front of each of the user's eyes. Note that a device using a semi-transparent half mirror may also be used for the image display unit 203. In this case, for example, the image display unit 203 may use a technology generally called AR (Augmented Reality) to display an image so that CG appears to be directly superimposed on the real space visible through the half mirror. Furthermore, the image display unit 203 may use a technology generally called VR (Virtual Reality) to display an image of a completely virtual space without using captured images.

[0022] The attitude sensor unit 204 acquires attitude (and position) information of the HMD 100. The attitude sensor unit 204 may acquire attitude information of the user (the user wearing the HMD 100) that corresponds to the attitude (and position) of the HMD 100. The attitude sensor unit 204 has an inertial measurement unit (IMU) configured with an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The attitude sensor unit 204 is used when acquiring information on the user's attitude (attitude information), and the HMD control unit 201 outputs the information on the user's attitude (attitude information) to the PC 110. The attitude information may be acquired from one or more of a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, an acceleration sensor, and an angular velocity sensor.

[0023] The HMD control unit 201 estimates the position or posture of each joint point of the user's hand and fingers from the two camera images acquired by the imaging unit 202. The joint points include characteristic points of parts of the body, such as the finger joints and fingertips, the back of the hand (palm), and the arm. Each joint point indicates a coordinate position, and the posture can be estimated from information on multiple joint points. Methods for estimating the position or posture of the hand and each joint point can include known object recognition and pose estimation techniques based on machine learning using, for example, a convolutional neural network. Depth position information of each joint point of the hand can be obtained by calculating the distance from the imaging unit 202 to each joint point, for example, by triangulation using stereo matching using the two camera images acquired by the imaging unit 202. The estimated coordinate information of each joint point of the hand is output from the control unit 201 to the PC 110.

[0024] The nonvolatile memory 205 is an electrically erasable and recordable nonvolatile memory, and stores programs to be executed by the control unit 211, which will be described later.

[0025] The working memory 206 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202, an image display memory for the image display unit 203, a working area for the control unit 201, and the like.

[0026] The gaze imaging unit 207 is a camera that acquires an image for detecting the user's gaze, and is attached inside the HMD to capture an image of the user's eyes when the user wears the HMD 100. The image of the subject (user's eyes) captured by the camera is output to the control unit 211 of the PC 110 via the HMD control unit 201. The control unit 211 detects the gaze of the user wearing the HMD 100 from the image captured by the gaze imaging unit 207, and identifies the location on the image display unit 203 that the user is gazing at.

[0027] <Internal configuration of the controller> Referring to FIG. 2, the internal configuration of the controller 120 will be described. The controller 120 includes a controller control unit 221, an operation unit 222, a communication unit 223, and a controller attitude sensor unit 224.

[0028] The controller control unit 221 is a CPU that controls each component of the controller 120. Note that instead of the controller control unit 221 controlling the entire device, a plurality of hardware components may share the processing to control the entire device.

[0029] The operation unit ******** includes buttons. The operation unit 222 detects whether a button has been operated and transmits detection information to the PC 110 via the communication unit 223. Note that the operation unit 222 may have multiple input formats.

[0030] The communication unit 223 performs wireless communication with the PC 110 via Bluetooth. If there are multiple controllers, each performs wireless communication with the PC 110 via Bluetooth.

[0031] The controller attitude sensor unit 224 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The inertial measurement unit detects changes in the position or attitude of the controller 120. The detected position and attitude change information is communicated from the communication unit ******** to the PC 110 via the controller control unit 221.. )]]

[0032] The output unit 225 is composed of an LED light source, a speaker, a vibration element, etc.

[0033] <Internal Configuration of the PC> Referring to FIG. 2, the internal configuration of the PC 110 will be described. The PC 110 includes a control unit 211, a non - volatile memory 212, a working memory 213, a communication unit 214, and a recording medium 215.

[0034] It should be noted that there seems to be an incomplete tag "********" in the translation of item . Please check and correct it if necessary.The control unit 211 is a CPU that controls each unit of the PC 110 in accordance with input signals and a program described below. Note that instead of the control unit 211 controlling the entire device, the entire device may be controlled by a plurality of hardware units sharing the processing. The control unit 211 receives, from the HMD 100, an image (captured image) acquired by the imaging unit 202 and orientation information acquired by the orientation sensor unit 204. The control unit 211 performs image processing on the captured image to cancel aberrations in the optical system of the imaging unit 202 and the optical system of the image display unit 203. The control unit 211 then composites the captured image with any CG to generate a composite image. The control unit 211 transmits the composite image to the HMD control unit 201 in the HMD 100.

[0035] The control unit 211 also finds the number of controllers included in the captured image. It also executes processing to recognize the attached position of each controller using information obtained via the communication unit 214. Then, based on the recognition result, it performs control to change the operation content for each controller in response to the input information.

[0036] The control unit 211 controls the position, orientation, and size of the CG in the composite image based on information (distance information and orientation information) acquired by the HMD 100. For example, when placing a virtual object represented by CG near a specific object existing in real space in the space represented by the composite image, the control unit 211 makes the virtual object (CG) larger the closer the distance between the specific object and the imaging unit 202. By controlling the position, orientation, and size of the CG in this way, the control unit 211 can generate a composite image in which a CG object that is not located in real space appears to be located in real space.

[0037] The control unit 211 also receives information estimated by the control unit 201 of the HMD 100. The received information is temporarily stored in the working memory 213.

[0038] Furthermore, in the control unit 211, the communication unit 214 receives information about a change in the position or posture of the controller 120 from the communication unit 223 of the controller 120. The control unit 211 superimposes on the combined image a designated position according to the information about the change in the position or posture of the controller 120. Note that the control unit 211 may also superimpose on the combined image a designated position according to the information about the change in the position and posture of the controller 120.

[0039] The nonvolatile memory 212 is an electrically erasable and recordable nonvolatile memory, and stores information such as programs to be described later and CG executed by the control unit 211. The control unit 211 can switch the CG read from the nonvolatile memory 212 (i.e., the CG used to generate a composite image).

[0040] The working memory 213 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202 and time-series information on the estimated coordinate positions of each joint point of the hand, as an image display memory for the image display unit 203, a working area for the control unit 211, etc.

[0041] The wrist joints may also be estimated in the PC 110. In this case, after the captured image is output from the imaging unit 202 to the PC 110, the control unit 211 of the PC 110 estimates the position or posture of each joint point of the hand, processes the image using this information, and outputs it to the HMD 100. Note that the control unit 211 may also estimate the position and posture of each joint point of the hand, processes the image using this information, and outputs it to the HMD 100.

[0042] In addition to the above components of the MR system, for example, the control unit of either the HMD 100 or the PC 110 performs various image processing on the captured image acquired by the imaging unit 202 and the display image displayed on the image display unit 203. However, this is not the main focus of the present invention, so a description thereof will be omitted.

[0043] <Play area settings> The user sets up a play area where they can work safely while wearing the HMD.

[0044] The play area can be freely drawn by the user (for example, following the shape of the room), or it can be a fixed area based on the initial user's position (for example, a 2m radius). The user can also use a controller or hand tracking to set the play area. When using a controller, the play area is set based on the detection results of the controller's position and orientation. The play area can also be set using ray operation using the controller's orientation, or the play area can be set according to the detection results of the controller's position using the controller's position and orientation.

[0045] If it is determined that the position of the HMD estimated by the orientation sensor unit 204 approaches the boundary of the play area or has gone beyond the play area, the user is notified.

[0046] If it is determined that the space has a play area already set, the play area setting can be omitted by using the already set play area information.

[0047] <Processing to display the play area setting screen depending on the home screen setting status and whether the application is VR or MR> An example of a real space in which a user operates an HMD will be described with reference to Fig. 3. A real space 300 shows a user 301, an HMD 100 worn by the user 301, and a desk 302, and the user 301 is looking in the direction of an arrow 303 through the HMD 100.

[0048] 4A to 4C, a screen displayed on the HMD 100 shown in FIG. 3 will be described. FIG. 4A is an example of a virtual space screen, i.e., a VR screen. A screen 410 displays a virtual object 411 and a virtual object 412. The virtual object 412 is displayed so as to be superimposed on a desk 302. The screen 410 does not display a real space, and the entire screen is composed of CG. FIG. 4B is an example of a mixed reality space screen, i.e., an MR screen. A screen 420 displays a virtual object 421 and the desk 302, which is a real object. The screen 420 displays the virtual object 421 superimposed on the real space. The HMD 100 of FIG. 3 displays screens such as those shown in FIGS. 4A and 4B.

[0049] The process of displaying the play area setting screen according to the HMD settings and the application being launched will be described with reference to Figures 4 and 5. In Figure 5, when the user turns on the HMD, the process proceeds according to the following flow.

[0050] In step S501, the control unit 211 determines whether the home screen is VR. If the control unit 211 determines that the home screen is VR, that is, that the home screen is an image of a virtual space, the process proceeds to step S502, and if the control unit 211 determines that the home screen is not VR (is MR), the process proceeds to step S505.

[0051] The home screen refers to a screen on which the user can configure the HMD and select and launch various installed applications. For example, a home screen with an image of a virtual space as shown in Figure 4(c) or a home screen with an image of a mixed reality space as shown in Figure 4(d) is displayed, and the user can select a desired application from one or more applications.

[0052] 4(c) shows the home screen of the virtual space, displaying a menu screen 433. The menu screen 433 displays a VR / MR switching button 434 and icons for launching various applications. It also displays a play area boundary 435 that was set before the VR screen was displayed. The play area boundary 435 may be displayed at all times, or may be displayed only when the user approaches the boundary.

[0053] 4(d) is a home screen of the mixed reality space displaying a menu screen 433. The menu screen 433 displays a VR / MR switching button 434 and icons for launching various applications.

[0054] Here, the user can switch the home screen between VR mode and MR mode using the VR / MR switching button 434. That is, a composite image or a virtual space image can be selectively set as the home screen. Here, the VR mode is a mode in which an image of a virtual space is displayed on the image display unit, and a real space is not displayed. As the image of the virtual space, an image corresponding to the position and orientation of the HMD based on the captured image acquired by the imaging unit 202 may be displayed, or an image not corresponding to the position and orientation of the HMD may be displayed. Note that, in the VR mode, if a dangerous situation is determined, the captured image or a composite image in which CG is superimposed on the captured image may be displayed. Also, the MR mode is a mode in which a composite image in which CG is superimposed on an image corresponding to the position and orientation of the HMD based on the captured image acquired by the imaging unit 202 is displayed on the image display unit. Note that, in the MR mode, if a dangerous situation is determined, the captured image may be displayed without the superimposition of CG. The processing when the VR / MR switching button 434 is operated will be described later in step S506.

[0055] In step S502, the control unit 211 determines whether the user has instructed the user to skip play area setting. If the control unit 211 determines that the user has instructed the user to skip play area setting, the process proceeds to step S505. If the control unit 211 determines that the user has not instructed the user to skip play area setting, the process proceeds to step S503. Here, in the screen 450 shown in FIG. 4(e), a button for allowing the user to select whether or not to set a play area is provided in the pop-up screen 453. In this screen, an operation not to set a play area corresponds to an instruction to skip play area setting, and an operation to set a play area corresponds to an instruction not to skip play area setting. Also, in the screen 460 shown in FIG. 4(f), a button for allowing the user to select whether or not to display a VR mode home screen is provided in the pop-up screen 463. In this screen, an operation not to display the VR mode home screen corresponds to an instruction to skip play area setting, and an operation to display the VR mode home screen corresponds to an instruction not to skip play area setting, i.e., an instruction to perform play area setting. Note that, on the screen shown in FIG. 4(f), instead of the button that allows the user to select whether to display the VR mode home screen, a button for selecting whether to display the MR mode home screen may be provided. On this screen, the operation of displaying the MR mode home screen corresponds to an instruction to omit play area setting, and the operation of not displaying the MR mode home screen corresponds to an instruction to not omit play area setting, i.e., an instruction to perform play area setting. Note that, on the screen shown in FIG. 4(f), instead of the button that allows the user to select whether to display the VR mode home screen, a button for selecting whether to display the VR mode home screen or the MR mode home screen may be provided. On this screen, the operation of displaying the MR mode home screen corresponds to an instruction to omit play area setting, and the operation of displaying the VR mode home screen corresponds to an instruction to not omit play area setting, i.e., an instruction to perform play area setting.

[0056] If it is determined that the user has instructed to skip play area settings, the home screen can be quickly displayed by switching from the VR screen set on the home screen to the MR screen.

[0057] In step S503, the control unit 211 displays a play area setting screen, and the user sets the play area by following the instructions on the screen. FIG. 4(g) shows an example of the screen used when setting the play area. When setting the play area, the user moves the controller to draw a line along the plane of real space that will become the boundary of the play area, thereby setting the area that will become the play area. Screen 470 depicts a scene in which the boundary of the play area is being set. Here, ray 476 is a virtual object that appears to extend from the controller, changes according to the movement of the controller, and indicates the position the controller is pointing to. A dotted line 475 that will become the boundary of the play area is indicated by a ray being drawn. FIG. 4(h) shows an example of the screen displayed when the play area setting is complete. Screen 480 in FIG. 4(h) displays the set play area boundary 435, and a pop-up screen 483 notifying the user that the play area setting is complete. While the play area is being set in Figures 4(g) and 4(h), an image captured by the imaging unit 202 is displayed, and the dotted line 475 that marks the boundary of the play area, the play area boundary 435, rays 476, etc. are displayed as CG.

[0058] Here, the play area is defined as an area where a user operates an HMD, an area where the user is allowed to be present, or an area where the user can safely move in real space. For example, if the play area is an allowed area where the user is allowed to be present, the area outside the play area is an unallowable area where the user is not allowed to be present. Furthermore, if the user leaves the play area, i.e., if the user approaches or crosses the boundary of the play area, a warning message is displayed to the user. As shown in FIG. 4(i), the warning message is superimposed on the displayed image. FIG. 4(i) shows a VR screen in a scene where the user has left the play area. In the screen 490 in FIG. 4(i), a virtual object 412 is displayed superimposed on the desk 302. However, because the user 301 has left the play area set by the user 301, the virtual object 412 becomes semi-transparent, revealing the desk 302, with a warning message 493 superimposed on it.

[0059] The warning information may be presented to the user by sound or vibration, by emitting sound from the HMD or the controller, by vibrating the HMD or the controller, or by presenting the same warning information to the HMD and the controller, or by presenting different warning information to the HMD and the controller.

[0060] Although the play area is set in step S503, an obstacle or an area where safety is not guaranteed may be specified instead of the play area, and the area other than the specified area may be regarded as the play area.

[0061] Instead of making the virtual object 412 semi-transparent, a captured image captured by the imaging unit 202 may be displayed.

[0062] In step S504, the control unit 211 displays a VR home screen. For example, a home screen of an image of a virtual space as shown in FIG.

[0063] In step S505, the control unit 211 displays a home screen of MR, for example, a home screen of an image of a mixed reality space as shown in FIG.

[0064] In step S506, the control unit 211 determines whether the user has issued an instruction to switch the VR / MR setting on the home screen. If the control unit 211 determines that an instruction to switch the VR / MR setting on the home screen has been issued, the process proceeds to step S501. If the control unit 211 determines that an instruction to switch the VR / MR setting on the home screen has not been issued, the process proceeds to step S507. For example, when the VR / MR switching button 434 is operated on the home screen shown in FIG. 4(c), the control unit 211 determines that an instruction to switch the VR / MR setting on the home screen has been issued. In the example shown in FIG. 4(c), the operation of the VR / MR switching button 434 is an operation to switch from the VR home screen to the MR home screen. Furthermore, when the VR / MR switching button 434 is operated on the home screen shown in FIG. 4(d), the control unit 211 determines that an instruction to switch the VR / MR setting on the home screen has been issued. In the example shown in FIG. 4(d), the operation of the VR / MR switching button 434 is an operation to switch from the MR home screen to the VR home screen.

[0065] In step S507, the control unit 211 determines whether or not the user has instructed the launch of an application. If the control unit 211 determines that the user has instructed the launch of an application, the process proceeds to S508, and if the control unit 211 determines that the user has not instructed the launch of an application, the process proceeds to step S506. For example, on the home screen shown in FIG. 4(c), an operation to select and launch an application is performed by operating the buttons on the controller. If such an operation has not been performed, the control unit 211 determines that the user has not instructed the launch of an application.

[0066] In step S508, the control unit 211 determines whether the application instructed to be launched is to be displayed on the home screen. The control unit 211 determines whether the application is to be displayed on the home screen based on the type of application and the settings of the HMD. If the control unit 211 determines that the application instructed to be launched is to be displayed on the home screen, the process proceeds to step S511; if the control unit 211 determines that the application instructed to be launched is not to be displayed on the home screen, the process proceeds to step S509. Note that when an application is displayed on the home screen, multiple applications can be launched and used simultaneously. Furthermore, when an application is displayed on the home screen, the VR / MR mode is maintained based on the settings of the home screen.

[0067] In step S509, the control unit 211 determines whether the application instructed to be started is a VR application. If it is determined to be a VR application, the process proceeds to step S510. If it is determined not to be a VR application, that is, if it is determined to be an MR application, the process proceeds to step S511.

[0068] In step S510, the control unit 211 displays a play area setting screen, and the user sets the play area according to the screen. Note that if the play area has been set to display a VR screen on the home screen, the processing of step S510 may not be performed.

[0069] In step S511, the control unit 211 starts the application instructed to be started.

[0070] It is also possible that when the launched application is an MR application, the image of the real world can be temporarily turned off. Furthermore, the image displayed on the HMD may contain a large proportion of CG, making it difficult to see the image of the real world. If the user is unable or finds it difficult to see the image of the real world while the MR application is running, the user may be prompted to set the play area at that time.

[0071] In addition, when switching between VR mode and MR mode on the home screen and in common with an application, the play area setting screen may be displayed if the HMD is in VR mode when it is started, or when it is switched from MR mode to VR mode.

[0072] (Embodiment 2) In the second embodiment, a system will be described that sets a play area when the ratio of the image of a virtual object to the composite image displayed is equal to or greater than a predetermined value, even in the MR mode.

[0073] With reference to FIGS. 6 and 7, a process for displaying a play area setting screen according to the proportion of CG (virtual object) in an application started in MR mode will be described.

[0074] 6 is an MR mode screen, and virtual objects 601, 602, and 603 are superimposed on each other. Here, virtual objects 602 and 603 are semi-transparent, and the desk 302 in the real space can be seen behind the virtual object 603. Although the screen 600 shown in FIG. 6 is an MR mode screen, the virtual objects occupy more than half of the screen, making it difficult to see the real space. In other words, the display ratio of the image of the HMD is high, making it difficult to see the image of the real space.

[0075] The flow in Figure 7 will be explained below assuming a scene in which the home screen in MR mode shown in Figure 4(d) is displayed. Also, in this scene, it is assumed that the play area has not been set.

[0076] In step S701, the control unit 211 determines whether or not the user has instructed the start of an application. If the control unit 211 determines that the user has instructed the start of an application, the process proceeds to step S702, and if the control unit 211 determines that the user has not instructed the start of an application, the process repeats step S701.

[0077] In step S702, the control unit 211 determines whether the proportion of the image of the virtual object in the composite image to be displayed is equal to or greater than a predetermined value. If the control unit 211 determines that the proportion of the image of the virtual object in the composite image to be displayed is equal to or greater than the predetermined value, the process proceeds to step S703. If the control unit 211 determines that the proportion of the image of the virtual object in the composite image to be displayed is less than the predetermined value, the process proceeds to step S704.

[0078] In the application instructed to start in step S701, it is assumed that a screen like the one shown in Fig. 6 is displayed in MR. In such a case, the image displayed on the HMD is largely CG, making it difficult to see the image in real space. Therefore, even in MR mode, a play area is set before starting the application.

[0079] In step S703, the control unit 211 displays a play area setting screen, and the user sets the play area according to the screen.

[0080] In step S704, the control unit 211 starts the application instructed to be started.

[0081] It should be noted that if the proportion of the image of the virtual object in the displayed composite image is less than a predetermined value, it may be determined not to set a play area.

[0082] Furthermore, in screen 600 shown in FIG. 6 , virtual objects 602 and 603 are semi-transparent, and desk 302 is visible. Thus, even if the proportion of the images of virtual objects in the displayed composite image is equal to or greater than a predetermined value, the play area may be set only when the proportion of the images of virtual objects with transparency less than a predetermined value is equal to or greater than a predetermined value. That is, even if the proportion of the images of virtual objects in the displayed composite image is equal to or greater than a predetermined value, the play area may be set only when the proportion of the images of virtual objects excluding images of virtual objects with transparency equal to or greater than a predetermined value is equal to or greater than a predetermined value. In other words, the play area is determined based on the proportion of the composite image occupied by virtual objects with low transparency, excluding virtual objects with high transparency. Here, transparency represents how much a virtual object is transparent to objects in the background; the more transparent the virtual object, the higher the transparency value. For example, virtual object 601 does not transmit objects in the background, so its transparency is 0. A higher transparency value makes it easier to see objects further behind.

[0083] 7, the process in step S702 is as follows. In step S702, control unit 211 determines whether the proportion of images of virtual objects whose transparency is less than a predetermined value in the composite image to be displayed is equal to or greater than a predetermined value. If control unit 211 determines that the proportion of images of virtual objects whose transparency is less than a predetermined value in the composite image to be displayed is equal to or greater than the predetermined value, the process proceeds to step S703. If control unit 211 determines that the proportion of images of virtual objects whose transparency is less than a predetermined value in the composite image to be displayed is less than the predetermined value, the process proceeds to step S704. Note that the composite image to be displayed may be compared with the captured image before CG is composited, and based on the matching area, it may be determined whether the proportion of images of virtual objects in the composite image to be displayed is less than a predetermined value.

[0084] In addition to making a judgment on the screen displayed at startup, the processing in step S702 may be repeated even after the application is launched, and the user may be prompted to set the play area when the proportion of the virtual object image in the displayed composite image reaches a predetermined value or more.

[0085] (Embodiment 3) In the third embodiment, a system is described in which a VR application is launched without setting a play area when there are few scenes involving movement in the VR application. If it is expected that the user will not move in the virtual space, the application may be launched without setting a play area. For example, in an application such as that shown in FIG. 8, a large screen 801 is displayed on a screen 800 in the virtual space, but the screen does not change even when the user moves in the real space. In such a case, it is expected that the user will operate the HMD while remaining stationary, such as while sitting in a chair. Setting a play area in such a case would be troublesome for the user.

[0086] If movement is not possible in the virtual space, the VR application may be launched without setting a boundary. If the screen 801 appears to be fixed in the same position even when the HMD is moved, the screen 801 cannot be seen when moved, so it is assumed to be stationary, and the VR application may be launched without setting a boundary.

[0087] 9, a process for starting a VR application without setting a play area will be described when the VR application does not involve much movement. Note that explanations of parts that overlap with the above content will be omitted.

[0088] In step S509, the control unit 211 determines whether the application instructed to be started is a VR application. If it is determined to be a VR application, the process proceeds to step S909. If it is determined not to be a VR application, that is, if it is determined to be an MR application, the process proceeds to step S511.

[0089] In step S909, the control unit 211 determines whether the VR application instructed to be started is expected to be operated in a stationary state. If it is determined that it is expected to be operated in a stationary state, the process proceeds to step S511, and if it is determined that it is not expected to be operated in a stationary state, that is, if it is determined that it is expected to be operated while moving, the process proceeds to step S510.

[0090] In addition, when the VR application is launched without setting a play area, the user may be prompted to set a play area when it is determined that the user has moved a predetermined distance or more.

[0091] As described above, according to the embodiment, it is possible to provide a system that can improve operability by minimizing the effort required for setting up a play area while maintaining safety.

[0092] Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0093] (Other embodiments) The present invention can also be realized by executing the following process: software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0094] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0095] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, an apparatus may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0096] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0097] [Configuration 1] an acquisition means for acquiring a captured image; a control means for selectively displaying on a display unit a composite image of the captured image and an image of a virtual object acquired by the acquisition means and an image of a virtual space, and for determining whether or not to set boundaries separating a plurality of areas in a real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space. 1. An information processing device comprising:

[0098] [Configuration 2] the control means determines not to set the boundary when the image to be displayed on the display unit is the composite image, and determines to set the boundary when the image to be displayed on the display unit is the image of the virtual space. 2. The information processing device according to configuration 1,

[0099] [Configuration 3] When the image to be displayed on the display unit is the composite image, the control means determines not to set the boundary if a ratio of the image of the virtual object to the composite image is less than a predetermined value, or determines to set the boundary if a ratio of the image of the virtual object to the composite image is equal to or greater than a predetermined value. 3. The information processing device according to configuration 1 or 2.

[0100] [Configuration 4] the control means determines to set the boundary when the image to be displayed on the display unit is an image of the virtual space and the image of the virtual space is an image based on the captured image, and determines not to set the boundary when the image of the virtual space is an image not based on the captured image. 4. The information processing device according to any one of configurations 1 to 3.

[0101] [Configuration 5] When the power supply of the information processing device is turned on, the control means controls the display unit to display a home screen on which a predetermined application from one or more applications can be selected, and when the home screen is an image of the virtual space, determines to set the boundary, and when the home screen is the composite image, determines not to set the boundary. 5. The information processing device according to any one of configurations 1 to 4.

[0102] [Configuration 6] the control means determines to set the boundary when an image of the application displayed on the display unit is an image of the virtual space, and determines not to set the boundary when an image of the application displayed on the display unit is the composite image. 6. The information processing device according to configuration 5.

[0103] [Configuration 7] When the boundary has been set before the home screen is displayed or before a first application is launched, the control means sets the set boundary as the boundary for the second application when the second application is launched. 7. The information processing device according to configuration 5 or 6.

[0104] [Configuration 8] When the image of the virtual space is to be displayed on the display unit, the control means controls to set the boundary before displaying the image of the virtual space. 8. The information processing device according to any one of configurations 1 to 7.

[0105] [Configuration 9] the control means determines to set the boundary when an operation to display an image of the virtual space is received from a user. 9. The information processing device according to any one of configurations 1 to 8.

[0106] [Configuration 10] the control means determines not to set the boundary when an operation to display the composite image is received from a user. 10. The information processing device according to any one of configurations 1 to 9.

[0107] [Configuration 11] The plurality of regions are a first region and a second region different from the first region. 11. The information processing device according to any one of configurations 1 to 10.

[0108] [Configuration 12] the first area is an area in which the user can safely move in the real space; 12. The information processing device according to configuration 11.

[0109] [Configuration 13] The first area is a space for a user to perform operations. 12. The information processing device according to configuration 11.

[0110] [Configuration 14] The first area is an allowed area in which the user is allowed to exist, and the second area is an unallowed area in which the user is not allowed to exist. 12. The information processing device according to configuration 11.

[0111] [Configuration 15] further comprising a second acquisition means for acquiring a detection result of the position or orientation of the controller; the control means sets the boundary based on the detection result of the position or orientation of the controller acquired by the second acquisition means. 15. The information processing device according to any one of configurations 1 to 14.

[0112] [Configuration 16] The image capturing device further includes a detecting unit for detecting a user's hand from the captured image acquired by the acquiring unit, the control means sets the boundary based on the position of the user's hand detected by the detection means. 16. The information processing device according to any one of configurations 1 to 15.

[0113] [Configuration 17] the control means sets the boundary of the specific range based on the captured image acquired by the acquisition means. 17. The information processing device according to any one of configurations 1 to 16.

[0114] [Configuration 18] the control means determines to set the boundary when the image to be displayed on the display unit is the composite image and a ratio of images of virtual objects having transparency less than a predetermined value in the composite image is equal to or greater than a predetermined value. 18. The information processing device according to any one of configurations 1 to 17.

[0115] [Configuration 19] When the control means determines that the user has left the boundary in a state in which the boundary has been set, the control means displays the captured image acquired by the acquisition means on the display unit. 19. The information processing device according to any one of configurations 1 to 18.

[0116] [Configuration 20] When it is determined that the user has left the boundary in a state in which the boundary has been set, the control means controls to present warning information to the user. 20. The information processing device according to any one of configurations 1 to 19.

[0117] [Configuration 21] a third acquisition means for acquiring a position or a posture of the user from the captured image acquired by the acquisition means; The control means controls to determine whether the user has left the boundary based on the position or posture of the user acquired by the third acquisition means. 21. The information processing device according to any one of configurations 1 to 20.

[0118] [Configuration 22] The method further comprises a storage means for storing the boundary set by the control means or each area divided by the boundary, the control means determines not to set the boundary when it is determined that the image to be displayed on the display unit is an image of the virtual space and that the user is present within the boundary stored in the storage means; 22. The information processing device according to any one of configurations 1 to 21.

[0119] [Configuration 23] the information processing device is a head-mounted display, Further comprising the display unit, 23. The information processing device according to any one of configurations 1 to 22.

[0120] [Control method] an acquisition step of acquiring a captured image; a control step of controlling the display unit to selectively display a composite image of the captured image and the image of the virtual object acquired by the acquisition means and an image of the virtual space, and determining whether or not to set boundaries separating a plurality of areas in the real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space. 2. A method for controlling an information processing apparatus comprising:

[0121] [program] A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 23.

[0122] [system] an acquisition device that acquires a captured image; a control device that controls a display unit to selectively display a composite image of the captured image and an image of a virtual object acquired by the acquisition means and an image of a virtual space, and determines whether to set boundaries that separate a plurality of areas in real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space. An information processing system comprising:

Claims

1. An information processing device, A means for acquiring captured images of real space, A display control means controls the display unit to display an image of a mixed reality space, which is an image obtained by combining the captured image obtained by the acquisition means and an image of a virtual object, or an image of a virtual space that does not include the real space, as the home screen. When the image of the virtual space is displayed on the display unit as the home screen, the system includes control means for notifying the user when the user crosses the boundary of the real space. After the information processing device is started, even if the boundary has not been set, the image of the mixed reality space is displayed on the display unit as the home screen. When the image of the virtual space is displayed on the display unit as the home screen, the boundary is set. When a predetermined application is selected from the home screen, the screen of the selected application is displayed. An information processing device characterized by the following:

2. When the user gives a first instruction to launch a first type of application that displays an image of the virtual space, the display control means displays a first screen on the display unit before launching the first type of application, allowing the user to choose whether or not to set a boundary in the real space. The information processing apparatus according to feature 1.

3. When the user issues a first instruction to launch a first type of application on which an image of the virtual space is displayed, the display control means, after the boundary has been set, displays an image related to the first type of application on the display unit. The information processing apparatus according to feature 1.

4. When the user issues a first instruction to launch a first type of application on which an image of the virtual space is displayed, the display control means displays an image related to the first type of application on the display unit with the boundary set. The information processing apparatus according to feature 1.

5. If the control means has set the boundary before starting the predetermined application, when starting the predetermined application, it sets the set boundary as the boundary in the predetermined application. The information processing apparatus according to feature 1.

6. The aforementioned real space is divided by the boundary into a first region and a second region different from the first region. The information processing apparatus according to feature 1.

7. The first area is an allowable area where the user is permitted to exist, and the second area is an unallowable area where the user is not permitted to exist. The information processing apparatus according to feature 6.

8. If the control means determines that the image displayed on the display unit is an image of the virtual space and that the user is located within a first area demarcated by the already established boundary, it controls the display unit not to display the screen for setting the boundary. The information processing apparatus according to feature 6.

9. The system further includes a second acquisition means for acquiring the detection result of the controller's position or orientation. The control means sets the boundary based on the detection result of the position or orientation of the controller obtained by the second acquisition means. The information processing apparatus according to feature 1.

10. The system further includes a detection means for detecting the user's hand from the captured image acquired by the acquisition means, The control means sets the boundary based on the position of the user's hand detected by the detection means. The information processing apparatus according to feature 1.

11. The control means sets the boundary within a specific range based on the captured image acquired by the acquisition means. The information processing apparatus according to feature 1.

12. When the display control means determines that the user has crossed the boundary while the boundary has been set, it displays the captured image acquired by the acquisition means on the display unit. The information processing apparatus according to feature 1.

13. The system further includes a third acquisition means for acquiring the user's position or orientation, The control means controls the system to determine whether the user has crossed the boundary based on the user's position or orientation acquired by the third acquisition means. The information processing apparatus according to feature 1.

14. The information processing device is a head-mounted display having the display unit. The information processing apparatus according to feature 1.

15. The image in the virtual space is an image that does not include the captured image. The information processing apparatus according to feature 1.

16. A method for controlling an information processing device, The acquisition step involves obtaining an image captured from the real space, A display control step controls the display unit to display an image of a mixed reality space, which is an image obtained by combining the captured image obtained in the acquisition step with an image of a virtual object, or an image of a virtual space that does not include the real space, as the home screen. When the image of the virtual space is displayed on the display unit as the home screen, the control step includes controlling the system to notify the user when the user crosses the boundary in the real space, After the information processing device is started, even if the boundary has not been set, the image of the mixed reality space is displayed on the display unit as the home screen. When the image of the virtual space is displayed on the display unit as the home screen, the boundary is set. When a predetermined application is selected from the home screen, the screen of the selected application is displayed. A control method for an information processing device characterized by the following features.

17. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.

18. An information processing system, An acquisition device that acquires captured images of real space, A display control device controls the display device to display an image of a mixed reality space, which is an image obtained by combining the captured image acquired by the acquisition device with an image of a virtual object, or an image of a virtual space that does not include the real space, as the home screen. When the image of the virtual space is displayed on the display device as the home screen, the device includes a control device that controls the device to notify the user when the user crosses the boundary in the real space, After the information processing system is started, even if the boundary has not been set, the image of the mixed reality space is displayed on the display device as the home screen. When the image of the virtual space is displayed on the display device as the home screen, the boundary is set. When a predetermined application is selected from the home screen, the screen of the selected application is displayed. An information processing system characterized by the following:

19. An acquisition means for acquiring captured images, The system includes a control means that controls the display unit to display an image of a mixed reality space, which is an image obtained by combining the captured image obtained by the acquisition means and an image of a virtual object, and when the user gives a first instruction to launch an application of a first type while the image of the mixed reality space is displayed on the display unit, the control means controls the display unit to display a first screen that allows the user to choose whether or not to set a boundary in the real space. An information processing device characterized by the following:

20. The control means controls the placement and display of a second screen for selecting the first type of application in the mixed reality space, and when the first instruction is given by the user via the second screen, the control means causes the first screen to be displayed on the display unit. The information processing apparatus according to feature 19.

Images