Information processing apparatus, information processing system, information processing method and program

Abstract

To provide a system which can improve operability by minimizing efforts to set up a play area.SOLUTION: An information processing apparatus includes: acquisition means for acquiring a picked-up image; and control means for executing control in such a manner that an image of a composite real space as an image obtained by synthesizing the picked-up image acquired by the acquisition means and an image of a virtual object and an image of a virtual space are selectively displayed on a display unit and, based on whether the image displayed on the display unit is the image of the composite real space or the image of the virtual space, executing control whether a screen for setting boundaries in a real space is displayed on the display unit or not.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] VR (Virtual Reality) technology is known as a technology that allows users to experience virtual space. In addition, so-called MR (Mixed Reality) technology is known as a technology that seamlessly fuses real space and virtual space in real time. For example, a head-mounted device such as an HMD (Head Mounted Display) is used as a device that allows users to experience such technology. However, wearing an HMD makes it difficult to check the surroundings, and there is a risk of bumping into obstacles.

[0003] For example, Patent Document 1 discloses a method of setting boundaries of an area (play area) in which a user can move around safely in advance, and warning the user when the user approaches the boundary line of the area, thereby reducing the risk of the user colliding with an obstacle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special table 2017-535901 Summary of the Invention [Problem to be solved by the invention]

[0005] The timing for setting an area (play area) where safe movement can be performed is often when the HMD is started. When using VR technology, the surrounding real space cannot be seen, so setting a play area allows safer use. However, when using MR technology where the surrounding real space can be seen, the user can visually recognize obstacles and the risk of colliding with the obstacles is reduced, so the need to set a play area is reduced. Even when the need to set a play area is low, it is troublesome for the user to have a configuration that requires setting a play area. Therefore, the present invention aims to provide a system that can improve operability by minimizing the effort required for setting a play area. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device characterized by having an acquisition means for acquiring an captured image, and a control means for selectively controlling a composite image of the captured image acquired by the acquisition means and an image of a virtual object and an image of a virtual space to be displayed on a display unit, and determining whether or not to set a boundary separating multiple areas in real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space. Effect of the Invention

[0007] According to the present invention, a system can be provided that can improve operability by minimizing the effort required for setting up a play area. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing the internal configuration of an HMD and the like in the first embodiment. [Diagram 3] 1 is a diagram showing a real space in which a user wearing an HMD exists in embodiment 1. FIG. [Figure 4] 1A to 1C are diagrams showing an example of a series of screen displays in the first embodiment. [Diagram 5] 11 is a flowchart for explaining the procedure of a process for displaying a setting screen for a play area depending on an application in the first embodiment. [Figure 6] FIG. 11 is a diagram showing an example of a screen display in the second embodiment. [Figure 7] 13 is a flowchart for explaining the procedure of a process for displaying a setting screen for a play area according to the proportion of a CG (virtual object) in an application started in MR mode in the second embodiment. [Figure 8] FIG. 13 is a diagram showing an example of a screen display in the third embodiment. [Figure 9] 13 is a flowchart illustrating the processing steps for starting a VR application without setting a play area when there is little movement in a scene in embodiment 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] (Embodiment 1) An information processing system 1 according to the first embodiment will be described with reference to Fig. 1. The information processing system 1 includes an HMD 100, a PC (personal computer) 110, and a controller 120.

[0011] The HMD 100 is a head-mounted display device (electronic device) that can be worn on the head of a user. The HMD 100 displays a composite image that combines a captured image of the area in front of the user captured by the HMD 100 with content such as CG in a form that corresponds to the posture of the HMD 100.

[0012] The PC 110 controls the HMD 100. The PC 110 is connected to the HMD 100 by a wired connection such as a USB cable, or wirelessly by a connection such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). The PC 110 generates a composite image by combining a captured image with CG, and transmits the composite image to the HMD 100. Note that, although a PC will be described here as an example of an information processing device, the information processing device is not limited to this. For example, the information processing device may be a smartphone or a tablet terminal, and each component of the PC 110 may be included in the HMD 100.

[0013] The controller 120 performs various controls of the HMD 100. If the PC 110 is in a specific control mode, when a user operates the controller 120, the HMD 100 is controlled in response to the user's operation. As shown in FIG. 1, the controller 120 may be in the shape of a finger ring that can be worn and supported by the user, or in the shape of a handheld type that can be held in the hand. The controller 120 also has physical buttons for performing a decision operation or a selection operation on the display. The controller 120 performs wireless communication with the PC 110 via Bluetooth. The controller is not limited to a system that communicates with the PC 110, and may be a system that communicates with the HMD 100.

[0014] By moving the controller 120, the user can change the indicated position on the display in accordance with the movement of the controller. The indicated position may be expressed as a point, or may be expressed as a virtual ray connecting the point of the indicated position and the controller with a straight line (segment) or dotted line. By pressing a physical button, a menu decision operation or selection operation can be performed. Although the shape of the controller 120 has been described as being ring-shaped or handheld, the shape is not limited to this as long as it can be supported by a finger, hand, or arm. Furthermore, although the buttons have been described as being physical buttons, they may be operated in the same manner as a trackpad, touch panel, wheel, or trackball, and may be operated by sliding, flicking, or touching in addition to pressing a button.

[0015] Note that the controller may be worn on at least one of a finger, a hand, or an arm.

[0016] Note that the controller may be attached to an object held by hand, and used to acquire position information and attitude information of the attached position from a sensor. Examples of such an object include an object imitating a tool.

[0017] <Internal Configuration of HMD> With reference to FIG. 2, the internal configuration of the HMD 100 will be described. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an image display unit 203, an attitude sensor unit 204, a non-volatile memory 205, a working memory 206, and an imaging unit 207 for the line of sight.

[0018] The HMD control unit 201 is a CPU that controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by synthesizing an imaging image captured by the imaging unit 202 of the space in front of the user and a CG) from the PC 110, the HMD control unit 201 displays the composite image on the image display unit 203. Note that instead of the HMD control unit 201 controlling the entire device, a plurality of hardware may share the processing to control the entire device.

[0019] The imaging unit 202 includes two cameras (imaging devices). The two cameras are for capturing captured images used for compositing with an image of a virtual space and generating position and orientation information, and have an imaging unit for a left eye and an imaging unit for a right eye. The imaging unit for the left eye captures a moving image of the real space corresponding to the left eye of the person wearing the HMD 101, and an image (captured image) of each frame in the moving image is output from the imaging unit for the left eye. The imaging unit for the right eye captures a moving image of the real space corresponding to the right eye of the person wearing the HMD 101, and an image (captured image) of each frame in the moving image is output from the imaging unit for the right eye. That is, the imaging unit 202 acquires captured images as stereo images having parallax that approximately matches the positions of the left eye and right eye of the person wearing the HMD 101. In addition, information on the distance from the two cameras to the subject can be acquired as distance information by distance measurement using the stereo cameras. Note that in an HMD for an MR system, it is preferable to arrange the imaging unit so that the central optical axis of the imaging range of the imaging unit approximately matches the line of sight direction of the person wearing the HMD.

[0020] Each of the left eye imaging section and the right eye imaging section has an optical system and an imaging device. Light entering from the outside world enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the entering light as a captured image. Images of a subject (a range in front of the user) captured by the two cameras are output to the PC 110 and the control unit 201. Note that the imaging unit 202 may capture and output a video instead of a captured image.

[0021] The image display unit 203 displays a composite image. The image display unit 203 has a liquid crystal panel or an organic EL panel. When the user wears the HMD 100, the image display unit 203 is disposed in front of each eye of the user. Note that a device using a semi-transmissive half mirror can also be used for the image display unit 203. In this case, for example, the image display unit 203 may display an image by using a technology generally called AR (Augmented Reality) so that the CG is directly superimposed on the real space seen through the half mirror. The image display unit 203 may also display an image of a complete virtual space without using a captured image by using a technology generally called VR (Virtual Reality).

[0022] The attitude sensor unit 204 acquires attitude (and position) information of the HMD 100. The attitude sensor unit 204 may acquire attitude information of a user (a user wearing the HMD 100) corresponding to the attitude (and position) of the HMD 100. The attitude sensor unit 204 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The attitude sensor unit 204 is used when acquiring information on the user's attitude (attitude information), and the HMD control unit 201 outputs the information on the user's attitude (attitude information) to the PC 110. The attitude information may be acquired from one or more of a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, an acceleration sensor, and an angular velocity sensor.

[0023] The HMD control unit 201 estimates the position or posture of each joint point of the user's hand and fingers from the two camera images obtained by the imaging unit 202. The joint points include points that are characteristic of parts such as the joints and fingertips of the fingers, the back of the hand (palm), and the arms. Each joint point indicates a coordinate position, and the posture can be estimated from information on a plurality of joint points. A known object recognition or pose estimation method of machine learning using a convolutional neural network can be used as a method for estimating the position or posture of the hand and each joint point of the hand. In addition, the position information of each joint point of the hand in the depth direction can be obtained by calculating the distance from the imaging unit 202 to each joint point by triangulation using stereo matching using the two camera images obtained by the imaging unit 202. The estimated coordinate information of each joint point of the hand is output from the control unit 201 to the PC 110.

[0024] The non-volatile memory 205 is an electrically erasable and recordable non-volatile memory, and stores programs executed by the control unit 211, which will be described later.

[0025] The working memory 206 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202, an image display memory for the image display unit 203, a working area for the control unit 201, and the like.

[0026] The gaze imaging unit 207 is a camera that acquires an image for detecting the gaze of the user, and is attached inside the HMD to capture an image of the user's eyes when the user wears the HMD 100. The image of the subject (user's eyes) captured by the camera is output to the control unit 211 of the PC 110 via the HMD control unit 201. The control unit 211 detects the gaze of the user wearing the HMD 100 from the image captured by the gaze imaging unit 207, and identifies the location on the image display unit 203 that the user is gazing at.

[0027] <Internal configuration of the controller> Referring to FIG. 2, the internal configuration of the controller 120 will be described. The controller 120 includes a controller control unit 221, an operation unit 222, a communication unit 223, and a controller attitude sensor unit 224.

[0028] The controller control unit 221 is a CPU that controls each component of the controller 120. Note that instead of the controller control unit 221 controlling the entire device, a plurality of hardware components may share the processing to control the entire device.

[0029] The operation unit 222 includes buttons. The operation unit 222 detects whether a button has been operated and transmits detection information to the PC 110 via the communication unit 223. Note that the operation unit 222 may have multiple input formats.

[0030] The communication unit 223 performs wireless communication with the PC 110 via Bluetooth. If there are multiple controllers, each performs wireless communication with the PC 110 via Bluetooth.

[0031] The controller attitude sensor unit 224 includes an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The inertial measurement unit detects changes in the position or attitude of the controller 120. The detected position and attitude change information is communicated from the communication unit 223 to the PC 110 via the controller control unit 221.

[0032] The output unit 225 is composed of an LED light source, a speaker, a vibration element, etc.

[0033] <Internal Configuration of the PC> Referring to FIG. 2, the internal configuration of the PC 110 will be described. The PC 110 includes a control unit 211, a non-volatile memory 212, a working memory 213, a communication unit 214, and a recording medium 215.

[0034] The control unit 211 is a CPU that controls each unit of the PC 110 according to input signals and a program described later. Instead of the control unit 211 controlling the entire device, the entire device may be controlled by having a plurality of hardware share the processing. The control unit 211 receives an image (captured image) acquired by the imaging unit 202 and orientation information acquired by the orientation sensor unit 204 from the HMD 100. The control unit 211 performs image processing on the captured image to cancel aberrations in the optical system of the imaging unit 202 and the optical system of the image display unit 203. Then, the control unit 211 composites the captured image with any CG to generate a composite image. The control unit 211 transmits the composite image to the HMD control unit 201 in the HMD 100.

[0035] The control unit 211 also obtains the number of controllers included in the captured image. It also executes a process for recognizing the attachment position of each controller using information obtained via the communication unit 214. Then, depending on the recognition result, it performs control to change the operation content for the input information of each controller for each controller.

[0036] The control unit 211 controls the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the HMD 100. For example, when placing a virtual object represented by the CG near a specific object existing in real space in the space represented by the composite image, the control unit 211 makes the virtual object (CG) larger as the distance between the specific object and the imaging unit 202 is closer. By controlling the position, orientation, and size of the CG in this manner, the control unit 211 can generate a composite image in which a CG object that is not placed in real space appears as if it were placed in real space.

[0037] In addition, the control unit 211 receives information estimated by the control unit 201 of the HMD 100. The received information is temporarily stored in the working memory 213.

[0038] Furthermore, in the control unit 211, the communication unit 214 receives information on a change in the position or attitude of the controller 120 from the communication unit 223 of the controller 120. The control unit 211 superimposes on the combined image an indication position according to the information on the change in the position or attitude of the controller 120. Note that the control unit 211 may also superimpose on the combined image an indication position according to the information on the change in the position and attitude of the controller 120.

[0039] The non-volatile memory 212 is an electrically erasable and recordable non-volatile memory, and stores information such as programs to be described later and CG executed by the control unit 211. The control unit 211 can switch the CG read from the non-volatile memory 212 (i.e., the CG used to generate a composite image).

[0040] The working memory 213 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202 and time-series information on the estimated coordinate positions of each joint point of the hand, as an image display memory for the image display unit 203, a working area for the control unit 211, etc.

[0041] The wrist joints may be estimated in the PC 110. In this case, after the captured image is output from the imaging unit 202 to the PC 110, the control unit 211 of the PC 110 estimates the position or posture of each joint point of the hand, processes the image using the information, and outputs the image to the HMD 100. Note that the control unit 211 may estimate the position and posture of each joint point of the hand, processes the image using the information, and outputs the image to the HMD 100.

[0042] In addition to the above, the MR system also includes other components, such as a control unit of either the HMD 100 or the PC 110, which performs various image processing on the captured image acquired by the imaging unit 202 and the display image displayed on the image display unit 203. However, this is not the focus of the present invention, so a description thereof will be omitted.

[0043] <Play area settings> The user sets up a play area where they can work safely while wearing the HMD.

[0044] The play area may be freely drawn by the user (for example, to fit the shape of the room), or it may be a fixed area based on the initial user's position (for example, 2m in circumference). The user may set the play area using a controller or hand tracking. When using a controller, the play area is set based on the detection results of the controller's position and orientation. The play area may be set by ray operation using the controller's orientation, or the play area may be set according to the detection results of the controller's position using the controller's position and orientation.

[0045] When it is determined that the position of the HMD estimated by the orientation sensor unit 204 approaches the boundary of the play area or goes beyond the play area, the user is notified.

[0046] When it is determined that the space has already been set as a play area, the play area setting can be omitted by using the already set play area information.

[0047] <Processing to display the play area setting screen depending on the home screen setting status and whether the application is VR or MR> An example of a real space in which a user operates an HMD will be described with reference to Fig. 3. A user 301, an HMD 100 worn by the user 301, and a desk 302 are shown in real space 300, and the user 301 is looking in the direction of an arrow 303 through the HMD 100.

[0048] Also, with reference to FIG. 4, a screen displayed on the HMD 100 shown in FIG. 3 will be described. FIG. 4(a) is an example of a screen of a virtual space, that is, a VR screen. A virtual object 411 and a virtual object 412 are displayed on a screen 410. The virtual object 412 is displayed so as to be superimposed on a desk 302. The screen 410 does not show a real space, and the entire screen is composed of CG. FIG. 4(b) is an example of a screen of a mixed reality space, that is, a MR screen. A virtual object 421 and the desk 302, which is a real object, are displayed on a screen 420. The virtual object 421 is displayed so as to be superimposed on the real space on the screen 420. The screen shown in FIG. 4(a) and FIG. 4(b) is displayed on the HMD 100 of FIG. 3.

[0049] A process for displaying a play area setting screen according to the HMD settings and the application to be launched will be described with reference to Figures 4 and 5. In Figure 5, when the user turns on the power of the HMD, the process proceeds according to the following flow.

[0050] In step S501, the control unit 211 determines whether the home screen is VR or not. If the control unit 211 determines that the home screen is VR, that is, that the home screen is an image of a virtual space, the process proceeds to step S502, and if the control unit 211 determines that the home screen is not VR (is MR), the process proceeds to step S505.

[0051] The home screen refers to a screen on which the user can set up the HMD, select various installed applications, and start them. For example, a home screen showing an image of a virtual space as shown in Fig. 4(c) or a home screen showing an image of a mixed reality space as shown in Fig. 4(d) is displayed, and the user can select a desired application from one or more applications.

[0052] 4(c) is a home screen of the virtual space on which a menu screen 433 is displayed. A VR / MR switching button 434 and icons for launching various applications are displayed on the menu screen 433. Also displayed is a boundary 435 of the play area that was set before the VR screen was displayed. Note that the boundary 435 of the play area may be displayed at all times, or may be displayed when the user approaches the boundary.

[0053] 4(d) shows the home screen of the mixed reality space displaying a menu screen 433. The menu screen 433 displays a VR / MR switching button 434 and icons for launching various applications.

[0054] Here, the user can switch the home screen between the VR mode and the MR mode by using the VR / MR switching button 434. That is, the composite image and the image of the virtual space can be selectively set as the home screen. Here, the VR mode is a mode in which the image of the virtual space is displayed on the image display unit, and the real space is not displayed. As the image of the virtual space, an image corresponding to the position and posture of the HMD may be displayed based on the captured image acquired by the imaging unit 202, or an image not corresponding to the position and posture of the HMD may be displayed. In addition, in the VR mode, when it is determined that the situation is dangerous, the captured image or a composite image in which CG is superimposed on the captured image may be displayed. In addition, the MR mode is a mode in which a composite image in which CG is superimposed on an image corresponding to the position and posture of the HMD based on the captured image acquired by the imaging unit 202 is displayed on the image display unit. In addition, in the MR mode, when it is determined that the situation is dangerous, the captured image may be displayed without superimposing CG. The process when the VR / MR switching button 434 is operated will be described in step S506 described later.

[0055] In step S502, the control unit 211 determines whether or not the user has instructed to omit the play area setting. If the control unit 211 determines that the user has instructed to omit the play area setting, the process proceeds to step S505. If the control unit 211 determines that the user has not instructed to omit the play area setting, the process proceeds to step S503. Here, in the screen 450 shown in FIG. 4(e), a button for the user to select whether or not to set a play area is provided in the pop-up screen 453. In this screen, an operation to not set a play area corresponds to an instruction to omit the play area setting, and an operation to set a play area corresponds to an instruction not to omit the play area setting. In addition, in the screen 460 shown in FIG. 4(f), a button for the user to select whether or not to display a home screen of the VR mode is provided in the pop-up screen 463. In this screen, an operation to not display a home screen of the VR mode corresponds to an instruction to omit the play area setting, and an operation to display a home screen of the VR mode corresponds to an instruction not to omit the play area setting, that is, an instruction to perform the play area setting. In addition, instead of the button for selecting whether to display the VR mode home screen to the user on the screen shown in FIG. 4(f), a button for selecting whether to display the MR mode home screen may be provided. In this screen, an operation for displaying the MR mode home screen corresponds to an instruction to omit the play area setting, and an operation for not displaying the MR mode home screen corresponds to an instruction not to omit the play area setting, that is, an instruction to perform the play area setting. In addition, instead of the button for selecting whether to display the VR mode home screen to the user on the screen shown in FIG. 4(f), a button for selecting whether to display the VR mode home screen or the MR mode home screen may be provided. In this screen, an operation for displaying the MR mode home screen corresponds to an instruction to omit the play area setting, and an operation for displaying the VR mode home screen corresponds to an instruction not to omit the play area setting, that is, an instruction to perform the play area setting.

[0056] If it is determined that the user has instructed to skip play area setting, the home screen can be quickly displayed by switching from the VR screen set as the home screen to the MR screen.

[0057] In step S503, the control unit 211 displays a play area setting screen, and the user sets the play area according to the screen. FIG. 4(g) shows an example of a screen when setting the play area. In setting the play area, the controller is moved so as to draw a line that will be the boundary of the play area along a plane in real space, and an area that will be the play area is set. A screen 470 is assumed to be a scene in the middle of setting the boundary of the play area. Here, a ray 476 is a virtual object that is displayed as if extending from the controller, and changes according to the movement of the controller, and represents the position that the controller is pointing to. A dotted line 475 that will be the boundary of the play area is indicated as if it is drawn with a ray. FIG. 4(h) shows an example of a screen when the setting of the play area is completed. A screen 480 in FIG. 4(h) displays the boundary 435 of the set play area, and a pop-up screen 483 notifying that the setting of the play area is completed is displayed. While the play area is being set in Figures 4(g) and 4(h), an image captured by the imaging unit 202 is displayed, and the dotted line 475 that marks the boundary of the play area, the play area boundary 435, rays 476, etc. are displayed as CG.

[0058] Here, the play area is set as an area for the user to operate the HMD, an area in which the user is permitted to be present, or an area in which the user can safely move in real space. For example, if the play area is an permitted area in which the user is permitted to be present, the area outside the play area is an unallowable area in which the user is not permitted to be present. In addition, if the user goes outside the play area, that is, if the user approaches or crosses the boundary of the play area, warning information is presented to the user. As shown in FIG. 4(i), the warning information is superimposed on the image to be displayed. FIG. 4(i) shows a VR screen in a scene in which the user has gone outside the play area. In the screen 490 of FIG. 4(i), a virtual object 412 is displayed so as to be superimposed on the desk 302. However, since the user has gone outside the play area set by the user 301, the virtual object 412 becomes semi-transparent, the desk 302 becomes visible, and a warning display 493 is displayed superimposed thereon.

[0059] The method of presenting the warning information to the user may be by sound or vibration, by outputting a sound from the HMD or the controller, by vibrating the HMD or the controller, or by presenting the same warning information to the HMD and the controller, or by presenting different warning information to the HMD and the controller.

[0060] Although the play area is set in step S503, an obstacle or an area where safety is not guaranteed may be specified instead of the play area, and the area other than the specified area may be regarded as the play area.

[0061] Instead of making the virtual object 412 semi-transparent, a captured image captured by the imaging unit 202 may be displayed.

[0062] In step S504, the control unit 211 displays a VR home screen. For example, the control unit 211 displays a home screen of a virtual space image as shown in FIG.

[0063] In step S505, the control unit 211 displays a home screen of MR. For example, a home screen of an image of a mixed reality space as shown in FIG.

[0064] In step S506, the control unit 211 determines whether or not the user has issued a VR / MR setting switching instruction for the home screen. If the control unit 211 determines that a VR / MR setting switching instruction for the home screen has been issued, the process proceeds to step S501. If the control unit 211 determines that a switching instruction has not been issued, the process proceeds to step S507. For example, when the VR / MR switching button 434 is operated on the home screen shown in FIG. 4(c), the control unit 211 determines that a VR / MR setting switching instruction for the home screen has been issued. In the example shown in FIG. 4(c), the operation of the VR / MR switching button 434 is an operation to switch from the VR home screen to the MR home screen. Also, when the VR / MR switching button 434 is operated on the home screen shown in FIG. 4(d), the control unit 211 determines that a VR / MR setting switching instruction for the home screen has been issued. In the example shown in FIG. 4(d), the operation of the VR / MR switching button 434 is an operation to switch from the MR home screen to the VR home screen.

[0065] In step S507, the control unit 211 determines whether or not the user has instructed to start an application. If the control unit 211 determines that the user has instructed to start an application, the process proceeds to S508, and if the control unit 211 determines that the user has not instructed to start an application, the process proceeds to step S506. For example, on the home screen shown in FIG. 4(c), an operation to select and start an application is performed by operating the buttons on the controller. If such an operation has not been performed, the control unit 211 determines that the user has not instructed to start an application.

[0066] In step S508, the control unit 211 determines whether the application instructed to be started is to be displayed on the home screen. The control unit 211 determines whether the application is to be displayed on the home screen based on the type of application and the settings of the HMD. If the control unit 211 determines that the application instructed to be started is to be displayed on the home screen, the process proceeds to step S511, and if the control unit 211 determines that the application instructed to be started is not to be displayed on the home screen, the process proceeds to step S509. Note that when an application is displayed on the home screen, multiple applications can be simultaneously started and multiple applications can be simultaneously used. Also, when an application is displayed on the home screen, the VR / MR mode is maintained based on the settings of the home screen.

[0067] In step S509, the control unit 211 determines whether the application instructed to be started is a VR application. If it is determined that the application is a VR application, the process proceeds to step S510. If it is determined that the application is not a VR application, that is, if it is determined that the application is an MR application, the process proceeds to step S511.

[0068] In step S510, the control unit 211 displays a play area setting screen, and the user sets the play area according to the screen. Note that if the play area has been set in order to display a VR screen on the home screen, the process of step S510 may not be performed.

[0069] In step S511, the control unit 211 starts the application instructed to be started.

[0070] In addition, when the launched application is an MR application, it is possible to temporarily turn off the image of the real space. In addition, it is possible that the image displayed on the HMD has a high proportion of CG, making it difficult to confirm the image of the real space. In this way, if the user is unable to confirm the image of the real space while the MR application is launched, or it becomes difficult to confirm it, the user may be prompted to set the play area at that time.

[0071] In addition, when switching between VR mode and MR mode on the home screen and in a common application, the play area setting screen may be displayed if the HMD is in VR mode when started up, or when switching from MR mode to VR mode.

[0072] (Embodiment 2) In the second embodiment, a system will be described that sets a play area when the ratio of the image of a virtual object to the total of a displayed composite image is equal to or greater than a predetermined value even in the MR mode.

[0073] A process of displaying a setting screen for a play area according to the proportion of a CG (virtual object) in an application started in MR mode will be described with reference to FIGS.

[0074] 6 is a screen in MR mode, and virtual objects 601, 602, and 603 are superimposed on each other. Here, virtual objects 602 and 603 are semi-transparent, and a desk 302 in real space can be seen behind virtual object 603. Although screen 600 shown in Fig. 6 is a screen in MR mode, virtual objects occupy more than half of the screen, making it difficult to see the real space. In other words, the display ratio of CG in the image displayed by the HMD is high, making it difficult to confirm the image of the real space.

[0075] The flow in Fig. 7 will be explained below assuming a scene in which the home screen in MR mode shown in Fig. 4(d) is displayed. Also, in this scene, it is assumed that the play area has not been set.

[0076] In step S701, the control unit 211 determines whether or not the user has instructed the start of an application. If the control unit 211 determines that the user has instructed the start of an application, the process proceeds to S702, and if the control unit 211 determines that the user has not instructed the start of an application, the process repeats the process of step S701.

[0077] In step S702, the control unit 211 determines whether the ratio of the image of the virtual object to the composite image to be displayed is equal to or greater than a predetermined value. If the control unit 211 determines that the ratio of the image of the virtual object to the composite image to be displayed is equal to or greater than the predetermined value, the process proceeds to S703, and if the control unit 211 determines that the ratio of the image of the virtual object to the composite image to be displayed is less than the predetermined value, the process proceeds to step S704.

[0078] In the application instructed to start in step S701, it is assumed that a screen like that shown in Fig. 6 is displayed in MR. In such a case, since the image displayed on the HMD has a high proportion of CG, making it difficult to confirm the image in real space, the play area is set before starting the application, even in MR mode.

[0079] In step S703, the control unit 211 displays a play area setting screen, and the user sets the play area according to the screen.

[0080] In step S704, the control unit 211 starts the application instructed to be started.

[0081] It should be noted that if the ratio of the image of the virtual object to the total composite image displayed is less than a predetermined value, it may be determined not to set a play area.

[0082] Also, in the screen 600 shown in FIG. 6, the virtual objects 602 and 603 are semi-transparent, and the desk 302 can be seen. In this way, even if the ratio of the images of the virtual objects to the displayed composite image is equal to or greater than a predetermined value, the play area may be set only when the ratio of the images of the virtual objects whose transparency is less than a predetermined value is equal to or greater than a predetermined value. That is, even if the ratio of the images of the virtual objects to the displayed composite image is equal to or greater than a predetermined value, the play area may be set only when the ratio of the images of the virtual objects excluding the images of the virtual objects whose transparency is equal to or greater than a predetermined value is equal to or greater than a predetermined value. In other words, the play area is determined based on the ratio of the images of the virtual objects with low transparency excluding the virtual objects with high transparency. Here, the transparency indicates how much the virtual object is transparent to the objects in the back, and the more the virtual object is transparent, the higher the transparency value. For example, the virtual object 601 does not transmit the objects in the back, so the transparency is 0, and if the transparency value is high, the objects in the back can be easily seen.

[0083] When applied to FIG. 7, the process in step S702 is as follows. In step S702, the control unit 211 determines whether the ratio of the images of virtual objects whose transparency is less than a predetermined value to the composite image to be displayed is equal to or greater than a predetermined value. If the control unit 211 determines that the ratio of the images of virtual objects whose transparency is less than a predetermined value to the composite image to be displayed is equal to or greater than a predetermined value, the process proceeds to S703. If the control unit 211 determines that the ratio of the images of virtual objects whose transparency is less than a predetermined value to the composite image to be displayed is less than a predetermined value, the process proceeds to step S704. Note that the composite image to be displayed may be compared with the captured image before compositing with CG, and based on the matching area, it may be determined whether the ratio of the images of virtual objects to the composite image to be displayed is less than a predetermined value.

[0084] In addition to making the judgment based on the screen displayed at startup, the processing in step S702 may be repeated even after the application is launched, and the user may be prompted to set the play area when the proportion of the image of the virtual object in the displayed composite image reaches or exceeds a predetermined value.

[0085] (Embodiment 3) In the third embodiment, a system is described in which a VR application is started without setting a play area when there are few scenes involving movement in the VR application. When it is assumed that there will be no movement in the virtual space, the application may be started without setting a play area. For example, in an application as shown in FIG. 8, a large screen 801 is displayed on a screen 800 in the virtual space, but it is assumed that the screen does not change even if the user moves in the real space. In such a case, it is assumed that the user will operate the HMD while remaining stationary, such as while sitting in a chair, and setting a play area in such a case would be troublesome for the user.

[0086] If movement is not possible in the virtual space, the VR application may be started without setting a boundary. If the screen 801 appears to be fixed in the same position even when the HMD is moved, the screen 801 cannot be seen when moved, so it is assumed to be stationary, and therefore the VR application may be started without setting a boundary.

[0087] With reference to Fig. 9, a process of starting a VR application without setting a play area will be described when the VR application does not involve much movement. Note that the description of parts that overlap with the above content will be omitted.

[0088] In step S509, the control unit 211 determines whether the application instructed to be started is a VR application. If it is determined that the application is a VR application, the process proceeds to step S909. If it is determined that the application is not a VR application, that is, if it is determined that the application is an MR application, the process proceeds to step S511.

[0089] In step S909, the control unit 211 determines whether or not the VR application instructed to be started is expected to be operated in a stationary state. If it is determined that the VR application is expected to be operated in a stationary state, the process proceeds to step S511, and if it is determined that the VR application is not expected to be operated in a stationary state, that is, if it is determined that the VR application is expected to be operated while moving, the process proceeds to step S510.

[0090] In addition, when the VR application is launched without setting a play area, the user may be prompted to set a play area when it is determined that the user has moved a predetermined distance or more.

[0091] As described above, according to the embodiment, a system can be provided that can improve operability by minimizing the effort required for setting up a play area while also maintaining safety.

[0092] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0093] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-mentioned embodiments are supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0094] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0095] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0096] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0097] [Configuration 1] An acquisition means for acquiring a captured image; a control means for selectively displaying on a display unit a composite image of the captured image and an image of a virtual object acquired by the acquisition means and an image of a virtual space, and for determining whether or not to set a boundary separating a plurality of regions in a real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space. 23. An information processing apparatus comprising:

[0098] [Configuration 2] the control means determines not to set the boundary when the image to be displayed on the display unit is the composite image, and determines to set the boundary when the image to be displayed on the display unit is the image of the virtual space. 2. The information processing device according to configuration 1.

[0099] [Configuration 3] the control means, when the image to be displayed on the display unit is the composite image, determines not to set the boundary if a ratio of the image of the virtual object in the composite image is less than a predetermined value, or determines to set the boundary if a ratio of the image of the virtual object in the composite image is equal to or greater than a predetermined value. 3. The information processing device according to configuration 1 or 2.

[0100] [Configuration 4] the control means determines to set the boundary when the image to be displayed on the display unit is an image of the virtual space and the image of the virtual space is an image based on the captured image, and determines not to set the boundary when the image of the virtual space is an image not based on the captured image. 4. The information processing device according to any one of configurations 1 to 3.

[0101] [Configuration 5] the control means, when the power supply of the information processing device is turned on, controls the display unit to display a home screen on which a predetermined application from among one or more applications can be selected, and when the home screen is an image of the virtual space, determines to set the boundary, and when the home screen is the composite image, determines not to set the boundary. 5. The information processing device according to any one of configurations 1 to 4.

[0102] [Configuration 6] the control means determines to set the boundary when an image of the application displayed on the display unit is an image of the virtual space, and determines not to set the boundary when an image of the application displayed on the display unit is the composite image. 6. The information processing device according to configuration 5.

[0103] [Configuration 7] When the boundary is set before the home screen is displayed or before a first application is launched, the control means sets the set boundary as a boundary in the second application when a second application is launched. 7. The information processing device according to configuration 5 or 6.

[0104] [Configuration 8] When the image of the virtual space is to be displayed on the display unit, the control means controls to set the boundary before displaying the image of the virtual space. 8. The information processing device according to any one of configurations 1 to 7.

[0105] [Configuration 9] the control means determines to set the boundary when an operation for displaying an image of the virtual space is received from a user; 9. The information processing device according to any one of configurations 1 to 8.

[0106] [Configuration 10] The control means determines not to set the boundary when a user operation to display the composite image is received. 10. The information processing device according to any one of configurations 1 to 9.

[0107] [Configuration 11] The plurality of regions are a first region and a second region different from the first region. 11. The information processing device according to any one of configurations 1 to 10.

[0108] [Configuration 12] The first area is an area in which the user can safely move in the real space. 12. The information processing device according to configuration 11.

[0109] [Configuration 13] The first area is a space for a user to perform operations. 12. The information processing device according to configuration 11.

[0110] [Configuration 14] The first area is an allowable area in which the user is allowed to exist, and the second area is an unallowable area in which the user is not allowed to exist. 12. The information processing device according to configuration 11.

[0111] [Configuration 15] Further comprising a second acquisition means for acquiring a detection result of a position or an orientation of the controller, the control means sets the boundary based on a detection result of the position or orientation of the controller acquired by the second acquisition means. 15. The information processing device according to any one of configurations 1 to 14.

[0112] [Configuration 16] The present invention further includes a detection unit for detecting a user's hand from the captured image acquired by the acquisition unit, The control means sets the boundary based on the position of the user's hand detected by the detection means. 16. The information processing device according to any one of configurations 1 to 15.

[0113] [Configuration 17] The control means sets the boundary of a specific range based on the captured image acquired by the acquisition means. 17. The information processing device according to any one of configurations 1 to 16.

[0114] [Configuration 18] the control means determines to set the boundary when the image to be displayed on the display unit is the composite image, and a ratio of an image of a virtual object having a transparency less than a predetermined value in the composite image is equal to or greater than a predetermined value. 18. The information processing device according to any one of configurations 1 to 17.

[0115] [Configuration 19] When the control means determines that the user has left the boundary in a state in which the boundary has been set, the control means displays the captured image acquired by the acquisition means on the display unit. 19. The information processing device according to any one of configurations 1 to 18.

[0116] [Configuration 20] When it is determined that the user has left the boundary in a state in which the boundary has been set, the control means controls to present warning information to the user. 20. The information processing device according to any one of configurations 1 to 19.

[0117] [Configuration 21] A third acquisition means for acquiring a position or a posture of the user from the captured image acquired by the acquisition means, The control means controls to determine whether the user has left the boundary based on the position or posture of the user acquired by the third acquisition means. 21. The information processing device according to any one of configurations 1 to 20.

[0118] [Configuration 22] The method further comprises a storage means for storing the boundary set by the control means or each area divided by the boundary, the control means determines not to set the boundary when it is determined that the image to be displayed on the display unit is an image of the virtual space and that the user is present within the boundary stored in the storage means; 22. The information processing device according to any one of configurations 1 to 21.

[0119] [Configuration 23] The information processing device is a head mounted display, The display unit is further provided. 23. The information processing device according to any one of configurations 1 to 22.

[0120] [Control method] An acquisition step of acquiring a captured image; a control step of selectively displaying on a display unit a composite image of the captured image and an image of a virtual object acquired by the acquisition means and an image of a virtual space, and determining whether or not to set a boundary separating a plurality of regions in a real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space. 23. A method for controlling an information processing apparatus comprising:

[0121] [program] A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 23.

[0122] [system] An acquisition device for acquiring a captured image; a control device that controls a composite image of the captured image and an image of a virtual object acquired by the acquisition means and an image of a virtual space to be selectively displayed on a display unit, and determines whether or not to set a boundary separating a plurality of regions in a real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space. An information processing system comprising:

Claims

1. An acquisition means for acquiring a captured image; a control means for selectively displaying on a display unit a composite image of the captured image and an image of a virtual object acquired by the acquisition means and an image of a virtual space, and for determining whether or not to set a boundary separating a plurality of regions in a real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space.

23. An information processing apparatus comprising:

2. the control means determines not to set the boundary when the image to be displayed on the display unit is the composite image, and determines to set the boundary when the image to be displayed on the display unit is the image of the virtual space.

2. The information processing apparatus according to claim 1,

3. the control means, when the image to be displayed on the display unit is the composite image, determines not to set the boundary if a ratio of the image of the virtual object in the composite image is less than a predetermined value, or determines to set the boundary if a ratio of the image of the virtual object in the composite image is equal to or greater than a predetermined value.

3. The information processing apparatus according to claim 2.

4. the control means determines to set the boundary when the image to be displayed on the display unit is an image of the virtual space and the image of the virtual space is an image based on the captured image, and determines not to set the boundary when the image of the virtual space is an image not based on the captured image.

3. The information processing apparatus according to claim 2.

5. the control means, when a power source of the information processing device is turned on, controls the display unit to display a home screen on which a predetermined application from among one or more applications can be selected, and when the home screen is an image of the virtual space, determines to set the boundary, and when the home screen is the composite image, determines not to set the boundary.

2. The information processing apparatus according to claim 1,

6. the control means determines to set the boundary when an image of the application displayed on the display unit is an image of the virtual space, and determines not to set the boundary when an image of the application displayed on the display unit is the composite image.

6. The information processing apparatus according to claim 5,

7. When the boundary is set before the home screen is displayed or before a first application is launched, the control means sets the set boundary as a boundary in the second application when a second application is launched.

7. The information processing apparatus according to claim 6,

8. When the image of the virtual space is to be displayed on the display unit, the control means controls to set the boundary before displaying the image of the virtual space.

6. The information processing apparatus according to claim 2 or 5.

9. the control means determines to set the boundary when an operation for displaying an image of the virtual space is received from a user; 6. The information processing apparatus according to claim 1,

10. The control means determines not to set the boundary when a user operation to display the composite image is received.

6. The information processing apparatus according to claim 1,

11. The plurality of regions are a first region and a second region different from the first region.

6. The information processing apparatus according to claim 1,

12. The first area is an area in which the user can safely move in the real space.

12. The information processing apparatus according to claim 11 .

13. The first area is a space for a user to perform an operation.

12. The information processing apparatus according to claim 11 .

14. The first area is an allowable area in which the user is allowed to exist, and the second area is an unallowable area in which the user is not allowed to exist.

12. The information processing apparatus according to claim 11 .

15. a second acquisition means for acquiring a detection result of a position or an orientation of the controller; the control means sets the boundary based on a detection result of the position or orientation of the controller acquired by the second acquisition means.

6. The information processing apparatus according to claim 1,

16. The present invention further includes a detection unit for detecting a user's hand from the captured image acquired by the acquisition unit, The control means sets the boundary based on the position of the user's hand detected by the detection means.

6. The information processing apparatus according to claim 1,

17. The control means sets the boundary of a specific range based on the captured image acquired by the acquisition means.

6. The information processing apparatus according to claim 1,

18. the control means determines to set the boundary when the image to be displayed on the display unit is the composite image, and a ratio of an image of a virtual object having a transparency less than a predetermined value in the composite image is equal to or greater than a predetermined value.

4. The information processing apparatus according to claim 3.

19. When the control means determines that the user has left the boundary in a state in which the boundary has been set, the control means displays the captured image acquired by the acquisition means on the display unit.

6. The information processing apparatus according to claim 1,

20. When it is determined that the user has left the boundary in a state in which the boundary has been set, the control means controls to present warning information to the user.

6. The information processing apparatus according to claim 1,

21. a third acquisition means for acquiring a position or a posture of the user from the captured image acquired by the acquisition means, The control means controls to determine whether the user has left the boundary based on the position or posture of the user acquired by the third acquisition means.

6. The information processing apparatus according to claim 1,

22. The method further comprises a storage means for storing the boundary set by the control means or each area divided by the boundary, the control means determines not to set the boundary when it is determined that the image to be displayed on the display unit is an image of the virtual space and that the user is present within the boundary stored in the storage means; 6. The information processing apparatus according to claim 1,

23. The information processing device is a head mounted display, The display unit is further provided.

6. The information processing apparatus according to claim 1,

24. An acquisition step of acquiring a captured image; a control step of selectively displaying on a display unit a composite image of the captured image and an image of a virtual object acquired by the acquisition means and an image of a virtual space, and determining whether or not to set a boundary separating a plurality of regions in a real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space.

23. A method for controlling an information processing apparatus comprising:

25. A program for causing a computer to function as each of the means of the information processing apparatus according to claim 1.

26. An acquisition device for acquiring a captured image; a control device that controls a composite image of the captured image and an image of a virtual object acquired by the acquisition means and an image of a virtual space to be selectively displayed on a display unit, and determines whether or not to set a boundary separating a plurality of regions in a real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space. An information processing system comprising:

Images