Information processing device, information processing system, information processing method, and program

Abstract

An information processing apparatus is characterized by comprising: an acquisition means for acquiring a captured image; and a control means for controlling the display unit to selectively display an image of the mixed reality space and an image of the virtual space, and a control unit configured to control whether to display a screen for setting a boundary in a real space on the display unit based on whether an image to be displayed on the display unit is an image of the mixed reality space or an image of the virtual space, the image of the mixed reality space is a composite image of the captured image acquired by the acquisition unit and an image of a virtual object.

Description

Technical Field

[0001] This disclosure relates to an information processing device. Background Technology

[0002] Virtual Reality (VR) technology is known as a technology that allows users to experience virtual spaces. Furthermore, Mixed Reality (MR) technology is known as a technology that seamlessly blends real and virtual spaces in real time. Devices that allow users to experience this technology include, for example, head-mounted displays (HMDs). However, wearing an HMD makes it difficult to observe one's surroundings, which may lead to the risk of collisions with obstacles.

[0003] Therefore, for example, according to Patent Document 1, a method is known in which the boundaries of an area (game area) in which a user can safely move are pre-set, and if the user approaches the boundary line of the area, a warning is issued to him or her to reduce the risk of the user colliding with obstacles.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Publication No. 2017-535901. Summary of the Invention

[0007] The problem the invention aims to solve

[0008] The timing for setting up a safe area (game area) for the user is typically during HMD startup. In VR technology, the surrounding real space is invisible, thus setting up a game area ensures safe use of the technology. However, in MR technology, where the surrounding real space is visible, users can visually identify obstacles, reducing the risk of collisions and thus lessening the need for a game area. Even when the need for a game area is low, the constant configuration is cumbersome for the user. Therefore, this disclosure relates to a system that improves operability by minimizing the time and effort required to set up a game area.

[0009] Solution for solving the problem

[0010] According to one aspect of this disclosure, an information processing device includes: an acquisition component for acquiring captured images; and a control component for controlling a display unit to selectively display images of a mixed reality space and images of a virtual space, and for controlling whether to display a screen for setting boundaries in a real space on the display unit based on whether the image to be displayed on the display unit is an image of the mixed reality space or an image of the virtual space, wherein the image of the mixed reality space is a composite image of the captured image acquired by the acquisition component and an image of a virtual object.

[0011] Advantages of the invention

[0012] According to this disclosure, a system can be provided that improves operability by minimizing the time and effort spent setting up the game area. Attached Figure Description

[0013] Figure 1 This is a diagram illustrating an information processing system according to a first embodiment.

[0014] Figure 2 This is an internal structure diagram of a head-mounted display (HMD) or the like according to the first embodiment.

[0015] Figure 3 This is a diagram showing the actual space where a user wearing an HMD according to the first embodiment exists.

[0016] Figure 4A This is an example diagram showing a screen of a virtual space according to the first embodiment.

[0017] Figure 4B This is a diagram illustrating an example of a mixed reality space according to the first embodiment.

[0018] Figure 4C This is a diagram illustrating an example of the homepage screen of a virtual space according to the first embodiment.

[0019] Figure 4D This is a diagram illustrating an example of a homepage screen of a mixed reality space according to the first embodiment.

[0020] Figure 4E This is a diagram illustrating an example of a pop-up screen according to the first embodiment, which includes a button for the user to select whether to set a game area.

[0021] Figure 4F This is a diagram illustrating an example of a pop-up screen according to the first embodiment, which includes a button for a user to select whether to display the home screen in virtual reality (VR) mode.

[0022] Figure 4GThe illustration shows a scene in which a region serving as a game area is set up midway by drawing a line along a plane in real space to serve as the boundary of the game area, according to a first embodiment.

[0023] Figure 4H This is an example diagram showing a screen when the game area is set up according to the first embodiment.

[0024] Figure 4I An example of a screen showing a warning message to a user when the user approaches or crosses the boundary of the game area, according to the first embodiment, is shown.

[0025] Figure 5 This is a flowchart illustrating the process of displaying a game area setting screen according to an application, based on a first embodiment.

[0026] Figure 6 This is a diagram illustrating an example of a screen display according to the second embodiment.

[0027] Figure 7 This is a flowchart illustrating a process according to a second embodiment for displaying a game area setting screen based on the scale of computer graphics (CG) (virtual objects) in an application to be launched in mixed reality (MR) mode.

[0028] Figure 8 An example of a screen display according to a third embodiment is shown.

[0029] Figure 9 This is a flowchart illustrating the process of launching a VR application without setting up a game area when minimal movement is expected in the scene, according to a third embodiment. Detailed Implementation

[0030] In the following, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the disclosure of the claims. Although multiple features are described in the embodiments, not all of these features are necessarily essential to this disclosure, and the multiple features may be appropriately combined. Furthermore, in the drawings, the same or similar components are indicated by the same reference numerals, and redundant descriptions are omitted.

[0031] (First Embodiment)

[0032] Reference Figure 1 The information processing system 1 according to the first embodiment is described. The information processing system 1 includes a head-mounted display (HMD) 100, a personal computer (PC) 110, and a controller 120.

[0033] The HMD 100 is a head-mounted display device (electronic device) that can be worn on a user's head. The HMD 100 displays a composite image obtained by combining a captured image of the area in front of the user with content such as computer graphics (CG) in a form corresponding to the posture of the HMD 100.

[0034] PC 110 controls HMD 100. PC 110 connects to HMD 100 via a wired connection such as a Universal Serial Bus (USB) cable or a wireless connection such as Bluetooth® or Wi-Fi®. PC 110 generates a composite image by combining captured images with CG and sends the composite image to HMD 100. Here, PC is described as an example of an information processing device, but the information processing device is not limited to this. For example, the information processing device may be a smartphone or tablet terminal, and HMD 100 may include various components of PC 110.

[0035] Controller 120 performs various types of control on HMD 100. When PC 110 is in a specific control mode, and a user operation is performed on controller 120, the HMD 100 is controlled in response to the user operation. For example... Figure 1 As shown, the controller 120 can have a ring shape (loop type) that can be worn on and supported by a user's finger, or a handheld shape that can be held by a user's hand. Furthermore, the controller 120 includes physical buttons for performing confirmation or selection operations on the display. The controller 120 wirelessly communicates with the PC 110 via Bluetooth. The controller is not limited to a system communicating with the PC 110, but can also be a system communicating with the HMD 100.

[0036] By moving the controller 120, the user can change the position of the indicator on the display according to the movement of the controller. The indicator position can be represented by a point, or by a virtual ray connecting the point of the indicator position to the controller using a straight line (line segment) or a dashed line. Pressing a physical button allows for confirmation or selection operations for menus. Although the shape of the controller 120 has been described as having a ring shape or a handheld shape, it is not limited to these, as long as the controller can be supported by fingers, a hand, or an arm. Furthermore, although the button has been described as a physical button, it can be any operable component, such as a touchpad, touch panel, scroll wheel, or trackball, and in addition to pressing the button, swiping, flicking, and touching operations are also acceptable.

[0037] The controller can be worn on at least one of the fingers, hand, and arm.

[0038] The controller can be arranged on an object grasped by a hand and can obtain position information and pose information of the attachment position from a sensor. Examples of such objects include objects shaped like tools.

[0039] <Internal Structure of HMD>

[0040] Reference will be made to Figure 2 describe the internal structure of the HMD 100. The HMD 100 includes an HMD control unit 201, a camera unit 202, an image display unit 203, a pose sensor unit 204, a non-volatile memory 205, a working memory 206, and an eye gaze camera unit 207.

[0041] The HMD control unit 201 is a central processing unit (CPU) that controls each component of the HMD 100. When acquiring a composite image (an image obtained by synthesizing a captured image captured by the camera unit 202 that captures the space in front of the user with CG) from the PC 110, the HMD control unit 201 displays the composite image on the image display unit 203. Instead of the HMD control unit 201 controlling the entire device, multiple hardware components can share the processing to control the entire device.

[0042] The camera unit 202 includes two cameras (imaging devices). These two cameras are configured to capture images to be used for synthesizing with the image of the virtual space and for generating position and pose information, and include a left-eye camera unit and a right-eye camera unit. The left-eye camera unit captures a moving image of the real space corresponding to the left eye of the wearer of the HMD 101 and outputs an image (captured image) of each frame in the moving image. The right-eye camera unit captures a moving image of the real space corresponding to the right eye of the wearer of the HMD 101 and outputs an image (captured image) of each frame in the moving image. In other words, the camera unit 202 acquires captured images as stereoscopic images having a parallax that substantially matches the respective positions of the left and right eyes of the wearer of the HMD 101. In addition, information related to the distance from the two cameras to the subject can be acquired as distance information by measuring the distance using a stereo camera. In an HMD for a mixed reality (MR) system, it is desirable that the central optical axis of the imaging range of the camera unit be arranged to be substantially aligned with the gaze direction of the wearer of the HMD.

[0043] The left-eye camera unit and the right-eye camera unit each include an optical system and an imaging device. Light entering from the outside passes through the optical system and enters the imaging device, and the imaging device outputs an image corresponding to the incident light as a captured image. The images captured by the two cameras by imaging the subject (the range in front of the user) are output to the PC 110 and the control unit 201. The camera unit 202 can capture and output video instead of captured images.

[0044] The image display unit 203 displays a composite image. The image display unit 203 includes a liquid crystal panel or an organic electroluminescent (EL) panel, etc. When the user is wearing the HMD 100, the image display unit 203 is positioned in front of the user's eyes. A device utilizing a semi-transparent mirror can also be used for the image display unit 203. In this case, for example, the image display unit 203 can display an image using a technique commonly known as augmented reality (AR), making the CG appear to be directly superimposed on the real space visible through the semi-transparent mirror. Furthermore, the image display unit 203 can display an image of a completely virtual space without using a captured image, using a technique commonly known as virtual reality (VR).

[0045] The posture sensor unit 204 acquires posture (and position) information of the HMD 100. The posture sensor unit 204 can also acquire posture information of the user (the user wearing the HMD 100) corresponding to the posture (and position) of the HMD 100. The posture sensor unit 204 includes an inertial measurement unit (IMU) configured with an accelerometer, an angular acceleration sensor, and a geomagnetic sensor. The posture sensor unit 204 is used to acquire information related to the user's posture (posture information), and the HMD control unit 201 outputs the user's posture-related information (posture information) to the PC 110. Posture information can be acquired from at least one of a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, an accelerometer, and an angular velocity sensor.

[0046] HMD control unit 201 estimates the position or pose of the user's hand and fingers' joints based on images acquired by two cameras from camera unit 202. Joints include points that serve as features of joints such as fingers, fingertips, back of the hand (or palm), and arm. Each joint indicates a coordinate position, and pose can be estimated based on information from multiple joints. As a method for estimating the position or pose of the hand and its joints, known object recognition or pose estimation methods utilizing machine learning with convolutional neural networks can be used, for example. Alternatively, position information of each joint in the depth direction can be obtained, for example, by calculating the distance from camera unit 202 to each joint of the hand using stereo matching-based triangulation via images acquired by the two cameras from camera unit 202. The estimated coordinate information of each joint of the hand is output from control unit 201 to PC 110.

[0047] The non-volatile memory 205 is an electrically erasable and recordable non-volatile memory, and stores programs, etc., to be executed by the control unit 211 as described below.

[0048] The working memory 206 functions as a buffer memory for temporarily storing the image data captured by the imaging unit 202, an image display memory for the image display unit 203, and a working area for the control unit 201, etc.

[0049] The eye gaze imaging unit 207 is a camera that acquires an image for detecting the user's eye gaze, and is attached to the inside of the HMD to capture the user's eyes when the user wears the HMD 100. The image of the subject (the user's eyes) captured by the camera is output to the control unit 211 of the PC 110 via the HMD control unit 201. The control unit 211 detects the eye gaze of the user wearing the HMD 100 from the image captured by the eye gaze imaging unit 207, and identifies the point on the image display unit 203 that the user is gazing at.

[0050] <Internal structure of the controller>

[0051] Reference will be made Figure 2 to describe the internal structure of the controller 120. The controller 120 includes a controller control unit 221, an operation unit 222, a communication unit 223, and a controller posture sensor unit 224.

[0052] The controller control unit 221 is a CPU that controls the components of the controller 120. Instead of the controller control unit 221 controlling the entire device, multiple hardware components can share the processing to control the entire device.

[0053] The operation unit 222 includes buttons. The operation unit 222 detects whether a button is operated, and sends the detection information to the PC 110 via the communication unit 223. The operation unit 222 can have multiple types of input forms.

[0054] The communication unit 223 performs wireless communication with the PC 110 via Bluetooth. In the case where there are multiple controllers, each controller performs wireless communication with the PC 110 via Bluetooth.

[0055] The controller posture sensor unit 224 includes an inertial measurement unit (IMU) configured with an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The inertial measurement unit detects changes in the position or posture of the controller 120. The detected position and posture change information is communicated from the communication unit 223 to the PC 110 via the controller control unit 221.

[0056] The output unit 225 is configured with a light emitting diode (LED) light source, a speaker, a vibration element, etc.

[0057] <Internal structure of the PC>

[0058] Reference will be made Figure 2The internal structure of PC 110 is described. PC 110 includes a control unit 211, a non-volatile memory 212, a working memory 213, a communication unit 214, and a storage medium 215.

[0059] Control unit 211 controls the CPUs of various units of PC 110 based on input signals and the program described below. Instead of control unit 211 controlling the entire device, multiple hardware components can share the processing to control the entire device. Control unit 211 receives images (captured images) acquired by camera unit 202 and posture information acquired by posture sensor unit 204 from HMD 100. Control unit 211 performs image processing on the captured images to eliminate aberrations in the optical systems of camera unit 202 and image display unit 203. Then, control unit 211 combines the captured images with arbitrary CG to generate a composite image. Control unit 211 sends the composite image to HMD control unit 201 in HMD 100.

[0060] The control unit 211 calculates the number of controllers included in the captured image. Furthermore, the control unit 211 uses information acquired via the communication unit 214 to perform processing for identifying the locations where each controller is positioned. Then, based on the identification results, the control unit 211 performs control to modify the operation of each controller according to its input information.

[0061] The control unit 211 controls the position, orientation, and size of the CG in the synthesized image based on information (distance information and pose information) acquired by the HMD 100. For example, when a virtual object represented by a CG is placed near a specific object existing in real space within the space represented by the synthesized image, the control unit 211 enlarges the virtual object (CG) as the distance between the specific object and the camera unit 202 becomes shorter. By controlling the position, orientation, and size of the CG in this way, the control unit 211 can generate a synthesized image in which CG objects not placed in real space appear to be placed in real space.

[0062] In addition, control unit 211 also receives information estimated by control unit 201 of HMD 100. The received information is temporarily stored in working memory 213.

[0063] Furthermore, in the control unit 211, the communication unit 214 receives change information related to the position or orientation of the controller 120 from the communication unit 223 of the controller 120. The control unit 211 overlays and displays the indicated positions corresponding to the change information related to the position or orientation of the controller 120 on the composite image. The control unit 211 can also overlay and display the indicated positions corresponding to the change information related to the position and orientation of the controller 120 on the composite image.

[0064] The non-volatile memory 212 is an electrically erasable and recordable non-volatile memory, and stores the program to be executed by the control unit 211 (described below) as well as information related to the CG. The control unit 211 can switch the CG to be read from the non-volatile memory 212 (i.e., the CG to be used to generate the composite image).

[0065] The working memory 213 serves as a buffer memory for temporarily storing image data captured by the camera unit 202 and time-series information of the estimated coordinate positions of each joint of the hand, as well as an image display memory for the image display unit 203 and a working area for the control unit 211.

[0066] Additionally, wrist joint estimation can be performed in PC 110. In this case, after the captured image is output from camera unit 202 to PC 110, control unit 211 of PC 110 estimates the position or posture of each joint of the hand, processes the image using the estimated information, and outputs it to HMD 100. Control unit 211 can estimate the position and posture of each joint of the hand, process the image using the estimated information, and output the image to HMD 100.

[0067] In components of the MR system other than those described above, such as the control unit of the HMD 100 or PC 110, various types of image processing are performed on the captured images obtained by the camera unit 202 or the displayed images on the image display unit 203. However, this is not the main focus of this disclosure, and therefore its description is omitted.

[0068] <Game Area Settings>

[0069] Users can set the gaming area where they can safely work while wearing the HMD.

[0070] The game area can be set up so that the user can freely draw it (e.g., along the shape of a room), or it can be a fixed area based on the user's initial position (e.g., within two meters of the user). Furthermore, the user can use a controller or hand tracking when setting up the game area. When using a controller, the game area is set based on the detection results of the controller's position or pose. The game area can be set using raycasting based on the controller's pose, or it can be set based on the detection results of the controller's position and pose.

[0071] If the user is notified when the position of the HMD estimated by the posture sensor unit 204 is close to or beyond the boundary of the game area.

[0072] If the space is determined to be a space where a game area has already been set, the game area setting can be omitted by using the already set game area information.

[0073] <Processing for displaying the game area settings screen based on the home screen settings status and whether the application is VR or MR>

[0074] Reference Figure 3 An example of a real-world space depicting a user operating an HMD. In real-world space 300, a user 301, an HMD 100 worn by the user 301, and a table 302 are shown, with the user 301 viewing through the HMD 100 in the direction of arrow 303.

[0075] Reference Figure 4A and Figure 4B Description in Figure 3 The image shown is displayed on the HMD 100. Figure 4A An example of a virtual space scene (i.e., a VR scene) is shown. Virtual objects 411 and 412 are displayed on screen 410. Virtual object 412 is displayed as an overlay on table 302. Screen 410 does not display real space, and the entire scene is composed of CG. Figure 4B An example of a mixed reality (MR) scene is shown. On screen 420, a virtual object 421 and a table 302, which is a real object, are displayed. On screen 420, the virtual object 421 is overlaid on the real space. Figure 3 HMD 100 in the display is as follows Figure 4A and Figure 4B The image shown.

[0076] Reference Figure 4C , Figure 4D , Figure 4E , Figure 4F , Figure 4G , Figure 4H , Figure 4I and Figure 5 This describes the processing used to display the game region settings screen based on the HMD settings and the application to be launched. Figure 5 When the user turns on the power to the HMD, the process is carried out according to the procedure described below.

[0077] In step S501, the control unit 211 determines whether the home screen is VR. If the control unit 211 determines that the home screen is VR (i.e., the home screen is an image of virtual space) (Yes in step S501), the process proceeds to step S502. If the control unit 211 determines that the home screen is not VR (but MR) (No in step S501), the process proceeds to step S505.

[0078] The home screen refers to the screen where you can configure the HMD (Host Device Manager) and select and launch various installed applications. For example, it might display something like... Figure 4C The homepage shown includes images of virtual space or, as... Figure 4D The home screen shown includes images of a mixed reality space, and users can select their desired application from one or more applications.

[0079] Figure 4C The main screen of the virtual space, displaying menu screen 433, is shown. On menu screen 433, a VR / MR switch button 434 and icons for launching various applications are displayed. Additionally, the boundary 435 of the game area, set before the VR screen is displayed, is shown. The boundary 435 of the game area can be always displayed, or it can be displayed only when the user approaches the boundary.

[0080] Figure 4D The main screen of the mixed reality space is shown, displaying a menu screen 433. On the menu screen 433, a VR / MR switch button 434 and icons for launching various applications are displayed.

[0081] Here, the VR / MR switch button 434 allows the user to switch the home screen between VR mode and MR mode. That is, composite images and images of the virtual space can be selectively set as the home screen. Here, VR mode displays images of the virtual space on the image display unit but not the real space. The images of the virtual space can be based on images captured by the camera unit 202 that correspond to the position or posture of the HMD, or images that do not correspond to the position or posture of the HMD can be displayed. In VR mode, if a dangerous situation is determined to have occurred, the captured image or a composite image obtained by overlaying CG onto the captured image can be displayed. Furthermore, MR mode displays a composite image obtained by overlaying CG onto an image corresponding to the position or posture of the HMD based on images captured by the camera unit 202 on the image display unit. In MR mode, if a dangerous situation is determined to have occurred, the captured image can be displayed without overlaying CG. The process performed when the VR / MR switch button 434 is operated will be described below in step S506.

[0082] In step S502, the control unit 211 determines whether the user has issued a command to skip the game area settings. If the control unit 211 determines that the user has issued a command to skip the game area settings (yes in step S502), the process proceeds to step S505; if the control unit 211 determines that the user has not issued a command to skip the game area settings (no in step S502), the process proceeds to step S503. Here, in Figure 4E On screen 450, a button for allowing users to select whether to set a game area is provided on pop-up screen 453. On this screen, the operation of not setting a game area corresponds to the instruction to skip game area settings, and the operation of setting a game area corresponds to the instruction to not skip game area settings. Furthermore, in Figure 4F On screen 460, a button is provided on pop-up screen 463 to allow the user to select whether to display the home screen in VR mode. On this screen, the operation of not displaying the home screen in VR mode corresponds to the instruction to skip game area settings, and the operation of displaying the home screen in VR mode corresponds to the instruction to not skip game area settings (i.e., the instruction to perform game area settings). Figure 4FThe screen shown can have a button for allowing users to choose whether to display the home screen in MR mode, instead of the button for allowing users to choose whether to display the home screen in VR mode. On this screen, the operation of displaying the home screen in MR mode corresponds to the instruction to skip game area settings, and the operation of not displaying the home screen in MR mode corresponds to the instruction to not skip game area settings (i.e., the instruction to perform game area settings). Figure 4F On the screen shown, a button for selecting whether to display the home screen in VR mode or MR mode can be provided instead of the button for allowing the user to select whether to display the home screen in VR mode. On this screen, the operation of displaying the home screen in MR mode corresponds to the instruction to skip game area settings, and the operation of displaying the home screen in VR mode corresponds to the instruction to not skip game area settings (i.e., the instruction to perform game area settings).

[0083] If the system determines that the user has issued a command to skip the game area settings, the home screen can be quickly displayed by switching from the VR screen set on the home screen to the MR screen.

[0084] In step S503, the control unit 211 displays a game area setting screen, and the user sets the game area according to the screen. Figure 4G An example of a screen used when setting up a game area is shown. When setting up a game area, the user moves the controller along a plane in real space to draw lines that serve as the boundaries of the game area, thus setting the area as the game area. Assume screen 470 is a scene where the boundaries of the game area are being set. Here, ray 476 is a virtual object displayed as appearing to extend from the controller, changing according to the controller's movement, and representing the position pointed to by the controller. The ray is indicated as a dashed line 475 pointing towards the boundaries of the game area, just like a drawn line. Figure 4H This shows an example of what happens when the game area setup is complete. Figure 4H On screen 480, the boundary 435 of the set game area is displayed, and a pop-up screen 483 notifying that the game area setting is complete is also displayed. Figure 4G and Figure 4H When setting up the game area, the image captured by the camera unit 202 is displayed, and the dashed line 475, the boundary 435 of the game area, and the ray 476, which are used as the boundary of the game area, are displayed as CG.

[0085] Here, the game area is defined as the area where the user interacts with the HMD, an area where the user is allowed to exist, or an area where the user can safely move in real space. For example, if the game area is a allowed area where the user is allowed to exist, the area outside the game area is a disallowed area where the user is not allowed to exist. Furthermore, if the user has moved out of the game area (i.e., if the user approaches the boundary of the game area or has already crossed the boundary), a warning message is presented to the user. Figure 4I As shown, the warning message is overlaid on the image to be displayed. Figure 4I This shows a VR view of the scene where the user has moved out of the game area. Figure 4I On screen 490, virtual object 412 is displayed as an overlay on table 302. However, since user 301 has moved out of the designated game area, virtual object 412 is made semi-transparent to make table 302 visible, and warning display 493 is overlaid.

[0086] As a method for presenting warning information to users, the warning information can be presented through sound or vibration. Sound can be output from the HMD or controller, and the HMD or controller can vibrate. Furthermore, the HMD and controller can present the same warning information or different warning information.

[0087] In step S503, a game area is set; however, an obstacle or an area that is not guaranteed to be safe can be specified instead of a game area, and an area other than the specified area can be regarded as a game area.

[0088] Instead of making the virtual object 412 semi-transparent, the captured image taken by the camera unit 202 can be displayed.

[0089] In step S504, the control unit 211 displays the home screen in VR mode. For example, it displays... Figure 4C The homepage screen showing the virtual space.

[0090] In step S505, the control unit 211 displays the home screen in MR mode. For example, it displays... Figure 4D The homepage screen shows an image of a mixed reality space.

[0091] In step S506, the control unit 211 determines whether the user has issued a command to switch the VR / MR settings of the home screen. If the control unit 211 determines that a command to switch the VR / MR settings of the home screen has been issued (Yes in step S506), the process proceeds to step S501; if the control unit 211 determines that a command to switch the VR / MR settings of the home screen has not been issued (No in step S506), the process proceeds to step S507. For example, in Figure 4CWhen the VR / MR switch button 434 is pressed on the homepage screen, it is determined that a command to switch the VR / MR settings of the homepage screen has been issued. Figure 4C In the example shown, the VR / MR switch button 434 operates by switching the home screen from VR mode to MR mode. Furthermore, in Figure 4D When the VR / MR switch button 434 is pressed on the homepage screen, it is determined that a command to switch the VR / MR settings of the homepage screen has been issued. Figure 4D In the example shown, the operation of the VR / MR switch button 434 is to switch the home screen from MR mode to VR mode.

[0092] In step S507, the control unit 211 determines whether the user has issued a command to launch the application. If the control unit 211 determines that a command to launch the application has been issued (yes in step S507), the process proceeds to S508; if the control unit 211 determines that a command to launch the application has not been issued (no in step S507), the process proceeds to step S506. For example, in Figure 4C On the home screen shown, users select and launch applications by operating the buttons on the controller. Without such operation, the control unit 211 determines that no command to launch the application has been issued.

[0093] In step S508, the control unit 211 determines whether to display the application that was instructed to be launched on the home screen. The control unit 211 determines whether to display the application on the home screen based on the application type and HMD settings. If the control unit 211 determines that the application that was instructed to be launched should be displayed on the home screen (Yes in step S508), the process proceeds to step S511; otherwise, if the control unit 211 determines otherwise (No in step S508), the process proceeds to step S509. When an application is displayed on the home screen, multiple applications can be launched simultaneously, and multiple applications can be used simultaneously. Furthermore, when an application is displayed on the home screen, VR or MR mode is maintained based on the home screen settings.

[0094] In step S509, the control unit 211 determines whether the application to be launched is a VR application. If the application is determined to be a VR application (yes in step S509), the process proceeds to step S510; if the application is not a VR application (i.e., if the application is determined to be an MR application) (no in step S509), the process proceeds to step S511.

[0095] In step S510, the control unit 211 displays a game area setting screen, and the user sets the game area according to the screen. If the game area has already been set to display the VR screen on the main screen, the process in step S510 can be omitted.

[0096] In step S511, the control unit 211 starts the application that was instructed to be started.

[0097] When launching a MR application, it's conceivable that an HMD (Host View Device) could temporarily disable the video in the real space. Furthermore, it's also possible that the HMD's displayed video has a large CG ratio, making it difficult to visually confirm the video in the real space. In this way, if the user becomes unable to confirm the video in the real space during the launch of the MR application, or is in a situation where the video in the real space is difficult to confirm, a timed prompt could be provided to the user to set the game area.

[0098] When switching between VR and MR modes for both the home screen and applications, the game area settings screen can be displayed if the HMD is in VR mode when it starts up or if the mode is switched from MR mode to VR mode.

[0099] (Second Embodiment)

[0100] According to the second embodiment, in MR mode, a system is used to set the game area when the proportion of the image of the virtual object in the displayed composite image is a predetermined value or greater.

[0101] Reference Figure 6 and Figure 7 This describes the processing used to display the game area setting screen based on the scale of CG (virtual objects) in the application launched in MR mode.

[0102] exist Figure 6 In the image 600 shown (which is the image in MR mode), virtual objects 601, 602, and 603 are superimposed. Here, virtual objects 602 and 603 are semi-transparent, and the table 302 in real space can be seen behind virtual object 603. Although Figure 6 The image shown is from MR mode, but virtual objects occupy more than half of the screen, making it difficult to perceive the real space. In other words, the HMD's displayed image has a large CG display ratio, making it difficult to confirm the real space in the video.

[0103] The following will show the assumptions. Figure 4D The scenario depicted in the MR mode home screen is described below. Figure 7 The process in the game. Furthermore, assume that no game area is set up in this scenario.

[0104] In step S701, the control unit 211 determines whether the user has instructed the application to be launched. If the control unit 211 determines that the application has been instructed to be launched (yes in step S701), the process proceeds to S702; if the control unit 211 determines that the application has not been instructed to be launched (no in step S701), the process in step S701 is repeated.

[0105] In step S702, the control unit 211 determines whether the proportion of the virtual object's image in the displayed composite image is a predetermined value or greater. If the control unit 211 determines that the proportion of the virtual object's image in the displayed composite image is a predetermined value or greater (yes in step S702), the process proceeds to step S703; and if the control unit 211 determines that the proportion of the virtual object's image in the displayed composite image is less than a predetermined value (no in step S702), the process proceeds to step S704.

[0106] In the application that is instructed to be launched in step S701, assuming it displays as follows in MR mode Figure 6 The situation depicted is as follows. In this case, the HMD's displayed image has a large CG display ratio, which may make it difficult to confirm the real space of the video, making it necessary to set the game area before launching the application, even in MR mode.

[0107] In step S703, the control unit 211 displays a game area setting screen, and the user sets the game area according to the screen.

[0108] In step S704, the control unit 211 starts the application that was instructed to be started.

[0109] If the proportion of the virtual object's image in the displayed composite image is less than a predetermined value, it can be determined that no game area is set.

[0110] exist Figure 6In the shown image 600, virtual objects 602 and 603 are semi-transparent, and table 302 is visually recognizable. In this way, even when the proportion of a virtual object's image in the displayed composite image is a predetermined value or greater, the game area can be set only when the proportion of images of virtual objects with transparency less than a predetermined value is a predetermined value or greater. That is, even when the proportion of a virtual object's image in the displayed composite image is a predetermined value or greater, the game area can be set when the proportion of images of virtual objects other than those with predetermined transparency is a predetermined value or greater. In other words, the judgment is based on the proportion of virtual objects with low transparency in the composite image, excluding virtual objects with high transparency. Here, transparency represents the degree to which a virtual object transmits light to objects behind it, and the transparency value increases as the virtual object transmits more light to objects behind it. For example, since virtual object 601 does not transmit light to objects behind it, its transparency is zero, and when the transparency value is higher, objects behind the virtual object become easier to visually recognize.

[0111] Applied to Figure 7 The processing in step S702 is as follows. In step S702, the control unit 211 determines whether the proportion of the image of the virtual object with transparency less than a predetermined value in the displayed composite image is a predetermined value or greater. If the control unit 211 determines that the proportion of the image of the virtual object with transparency less than a predetermined value in the displayed composite image is a predetermined value or greater (yes in step S702), the process proceeds to step S703. Furthermore, if the control unit 211 determines that the proportion of the image of the virtual object with transparency less than a predetermined value in the displayed composite image is less than a predetermined value (no in step S702), the process proceeds to step S704. The determination of whether the proportion of the image of the virtual object in the displayed composite image is less than a predetermined value can be based on the matching area obtained by comparing the displayed composite image with the captured image before combining the CG.

[0112] Not only in judging the screen displayed at startup, but also after the application is launched, the processing in step S702 can be repeated, and if the proportion of the virtual object's image in the displayed composite image becomes a predetermined value or larger, the user can be prompted to set the game area.

[0113] (Third Embodiment)

[0114] According to the third embodiment, a system for launching a VR application without setting up a game area will be described when there are few scenes involving movement in the VR application. Assuming the user does not move within the virtual space, the application can be launched without setting up a game area. For example, in situations such as... Figure 8 In the application shown, although the large screen 801 is displayed on the virtual space screen 800, it is assumed that the screen does not change even if the user moves within the real space. In this case, it is assumed that the user operates the HMD while remaining stationary (such as while sitting in a chair), and setting up the game area in this situation may be cumbersome for the user.

[0115] In situations where movement is impossible in virtual space, VR applications can be launched without setting boundaries. Even when the HMD moves, screen 801 appears fixed in the same position; if the user moves, screen 801 cannot be visually recognized. Therefore, since it is assumed that the user remains stationary, VR applications can be launched without setting boundaries.

[0116] Reference Figure 9 This describes the process for launching a VR application in scenarios with minimal movement, without setting up a game area. Descriptions overlapping with the above will be omitted.

[0117] In step S509, the control unit 211 determines whether the application to be launched is a VR application. If the application is determined to be a VR application (yes in step S509), the process proceeds to step S909; if the application is not a VR application (i.e., if the application is determined to be an MR application) (no in step S509), the process proceeds to step S511.

[0118] In step S909, the control unit 211 determines whether the VR application to be activated is assumed to be operated in a stationary state. If it is determined that the VR application is assumed to be operated in a stationary state (Yes in step S909), the process proceeds to step S511; and if it is determined that the VR application is assumed to be operated in a non-stationary state (i.e., the VR application is assumed to be operated while in motion) (No in step S909), the process proceeds to step S510.

[0119] When launching a VR application without setting a game area, the system can prompt the user to set a game area when it determines that the user has moved a predetermined distance or more.

[0120] As described above, according to the embodiments, a system can be provided that improves operability while maintaining security by minimizing the time and effort required to set up the game area.

[0121] Although this disclosure has been described in detail based on its preferred embodiments, this disclosure is not limited to these specific embodiments, and various forms are included in this disclosure without departing from its spirit. Furthermore, portions of the above embodiments may be suitably combined.

[0122] (Other embodiments)

[0123] Note that this disclosure can also be implemented by performing the following process. That is, this disclosure can also be implemented by supplying software (programs) for implementing the functions of the above embodiments to a system or device via a network or various types of storage media, and causing the computer (or control unit or microprocessor unit (MPU), etc.) of the system or device to read and execute the program code. In this case, the program and the storage medium storing the program constitute this disclosure.

[0124] Although this disclosure has been described in detail based on its preferred embodiments, this disclosure is not limited to these specific embodiments, and various forms are included in this disclosure without departing from its spirit. Furthermore, portions of the above embodiments may be suitably combined.

[0125] Each functional unit in the various embodiments (various modifications) described above may be implemented as separate hardware, or may not be implemented as separate hardware. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a functional unit may be implemented by separate hardware. Two or more functions of a functional unit may be implemented by common hardware. Furthermore, each functional unit may be implemented by hardware such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a digital signal processor (DSP), or may not be implemented by hardware such as an ASIC, FPGA, or DSP. For example, the device may include a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least a portion of the functional units included in the device can be implemented by the processor reading the control program from the memory and executing it.

[0126] This disclosure can also be implemented by a process in which a program for implementing one or more functions of the above embodiments is supplied to a system or device via a network or storage medium, and one or more processors of the computer of the system or device reads and executes the program. This disclosure can also be implemented by circuitry (e.g., an ASIC) for implementing one or more functions.

[0127] The disclosure of the embodiments includes the following configurations, methods, and procedures.

[0128] [Configuration 1]

[0129] An information processing device, comprising:

[0130] Acquisition component, used to acquire captured images; and

[0131] A control unit is used to control the selective display on a display unit of a composite image of the captured image and the image of the virtual object, as well as an image of the virtual space, acquired by the acquisition unit, and to determine whether to set boundaries for multiple regions in the real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space.

[0132] [Configuration 2]

[0133] According to the information processing device of configuration 1, the control unit determines that the boundary is not set when the image to be displayed on the display unit is the composite image, and determines that the boundary is set when the image to be displayed on the display unit is the image of the virtual space.

[0134] [Configuration 3]

[0135] According to the information processing device described in configuration 1 or 2, when the image to be displayed on the display unit is the composite image, the control unit determines that the boundary is not set when the proportion of the image of the virtual object in the composite image is less than a predetermined value, or determines that the boundary is set when the proportion of the image of the virtual object in the composite image is a predetermined value or greater.

[0136] [Configuration 4]

[0137] According to any one of configurations 1 to 3, in the information processing device, the control unit determines to set the boundary when the image to be displayed on the display unit is an image of the virtual space and the image of the virtual space is based on the captured image, and determines not to set the boundary when the image of the virtual space is not based on the captured image.

[0138] [Configuration 5]

[0139] According to any one of configurations 1 to 4, when the power supply of the information processing device is turned on, the control unit controls the display unit to display a home screen, wherein a predetermined application can be selected from one or more applications on the home screen; the control unit determines to set the boundary when the home screen is an image of the virtual space; and the control unit determines not to set the boundary when the home screen is a composite image.

[0140] [Configuration 6]

[0141] According to the information processing device described in configuration 5, the control unit determines to set the boundary when the image of the application to be displayed on the display unit is an image of the virtual space, and determines not to set the boundary when the image of the application to be displayed on the display unit is the composite image.

[0142] [Configuration 7]

[0143] According to the information processing device described in configuration 5 or 6, if the boundary has been set before the home screen is displayed or before the first application is launched, the control unit sets the set boundary as the boundary in the second application when the second application is launched.

[0144] [Configuration 8]

[0145] According to any one of configurations 1 to 7, in the information processing device, when an image of the virtual space is displayed on the display unit, the control unit controls the setting of the boundary before displaying the image of the virtual space.

[0146] [Configuration 9]

[0147] According to any one of configurations 1 to 8, in the case of receiving an operation from a user to display an image of the virtual space, the control unit determines that the boundary should be set.

[0148] [Configuration 10]

[0149] According to any one of configurations 1 to 9, in the information processing device, when the user receives an operation to display the composite image, the control unit determines that the boundary should not be set.

[0150] [Configuration 11]

[0151] According to any one of configurations 1 to 10, the information processing device includes a first region and a second region different from the first region.

[0152] [Configuration 12]

[0153] According to the information processing device of configuration 11, the first area is an area in which the user can move safely in the real space.

[0154] [Configuration 13]

[0155] According to the information processing device described in configuration 11, the first area is a space for user operation.

[0156] [Configuration 14]

[0157] According to the information processing device of configuration 11, the first area is a permitted area where the user is allowed to exist, and the second area is a disallowed area where the user is not allowed to exist.

[0158] [Configuration 15]

[0159] The information processing device according to any one of configurations 1 to 14 further includes a second acquisition component, the second acquisition component being used to acquire the detection result of the position or posture of the controller.

[0160] The control component sets the boundary based on the detection results of the position or posture of the controller obtained by the second acquisition component.

[0161] [Configuration 16]

[0162] The information processing device according to any one of configurations 1 to 15 further includes a detection component for detecting a user's hand from the captured image acquired by the acquisition component.

[0163] The control component sets the boundary based on the position of the user's hand detected by the detection component.

[0164] [Configuration 17]

[0165] According to any one of configurations 1 to 16, the information processing device wherein the control unit sets the boundary of a specific range based on the captured image acquired by the acquisition unit.

[0166] [Configuration 18]

[0167] According to any one of configurations 1 to 17, in the information processing device, if the image to be displayed on the display unit is the composite image and the proportion of the image of a virtual object with a transparency less than a predetermined value in the composite image is a predetermined value or greater, the control unit determines that the boundary is set.

[0168] [Configuration 19]

[0169] According to any one of configurations 1 to 18, in the information processing device, when the control unit determines that the user has moved out of the boundary while the boundary is set, the control unit displays the captured image acquired by the acquisition unit on the display unit.

[0170] [Configuration 20]

[0171] According to any one of configurations 1 to 19, the information processing device is configured to present a warning message to the user when it is determined that the user has moved out of the boundary while the boundary has been set.

[0172] [Configuration 21]

[0173] The information processing device according to any one of configurations 1 to 20 further includes a third acquisition component, the third acquisition component being used to acquire the user's position or posture from the captured image acquired by the acquisition component.

[0174] The control component performs control to determine whether the user has moved out of the boundary based on the user's position or posture acquired by the third acquisition component.

[0175] [Configuration 22]

[0176] The information processing device according to any one of configurations 1 to 21 further includes a storage component for storing the boundary set by the control component or the various regions divided by the boundary.

[0177] Wherein, if the image to be displayed on the display unit by the control component is an image of the virtual space and it is determined that the user exists within the boundary stored in the storage component, it is determined that the boundary is not set.

[0178] [Configuration 23]

[0179] The information processing device according to any one of configurations 1 to 22, wherein the information processing device is a head-mounted display and further includes the display unit.

[0180] [Control Methods]

[0181] A control method for an information processing device, comprising:

[0182] Acquisition steps, used to acquire captured images; and

[0183] The control step is used to control the selective display on the display unit of a composite image of the captured image and the image of the virtual object, as well as an image of the virtual space, acquired by the acquisition component, and to determine whether to set the boundaries of multiple regions in the real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space.

[0184] [program]

[0185] A program for enabling a computer to function as various units of an information processing device according to any one of configurations 1 to 23.

[0186] [system]

[0187] An information processing system, comprising:

[0188] Acquiring device, which is configured to acquire captured images; and

[0189] A control device configured to control the selective display on a display unit of a composite image of the captured image and the image of the virtual object, as well as an image of the virtual space, acquired by the acquisition component, and configured to determine whether to set boundaries for multiple regions in the real space based on whether the image to be displayed on the display unit is the composite image or the image of the virtual space.

[0190] This disclosure is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of this disclosure. Therefore, the appended claims have been added to disclose the scope of this disclosure to the public.

[0191] This application claims the benefit of Japanese Patent Application 2023-142964, filed on September 4, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. An information processing device, comprising: Acquisition component, used to acquire captured images; as well as A control unit is configured to control the display unit to selectively display images of mixed reality space and virtual space, and to control whether to display a screen for setting boundaries in real space on the display unit based on whether the image to be displayed on the display unit is an image of mixed reality space or an image of virtual space, wherein the image of mixed reality space is a composite image of a captured image and an image of a virtual object acquired by the acquisition unit.

2. The information processing device according to claim 1, wherein, When the image to be displayed on the display unit is an image of the mixed reality space, the control unit controls the display unit not to display the screen used to set the boundary; and when the image to be displayed on the display unit is an image of the virtual space, the control unit controls the display unit to display the screen used to set the boundary.

3. The information processing device according to claim 2, wherein, The control unit controls the display unit to display a screen for setting the boundary when the image to be displayed on the display unit is an image of the virtual space and the image of the virtual space is based on a captured image; and controls the display unit not to display a screen for setting the boundary when the image of the virtual space is not based on a captured image.

4. The information processing device according to claim 1, wherein, When the power to the information processing device is turned on, the control unit controls the display unit to display a home screen, wherein a predetermined application can be selected from one or more applications on the home screen; when the home screen is an image of the virtual space, the control unit controls the display unit to display a screen for setting the boundary; and when the home screen is an image of the mixed reality space, the control unit controls the display unit not to display the screen for setting the boundary.

5. The information processing device according to claim 4, wherein, When the image of the application to be displayed on the display unit is an image of the virtual space, the control unit controls the display unit to display a screen for setting the boundary; and when the image of the application to be displayed on the display unit is the composite image, the control unit controls the display unit not to display a screen for setting the boundary.

6. The information processing device according to claim 5, wherein, If the boundary has been set before the home screen is displayed or before the first application is launched, the control component will set the set boundary as the boundary in the second application when the second application is launched.

7. The information processing device according to claim 2, wherein, When the image of the virtual space is displayed on the display unit, the control unit controls the display unit to display a screen for setting the boundary before displaying the image of the virtual space.

8. The information processing device according to claim 1, wherein, When the user receives an operation to display an image of the virtual space, the control unit controls the display unit to display a screen for setting the boundaries.

9. The information processing device according to claim 1, wherein, When the user receives an operation to display an image of the mixed reality space, the control unit controls the display unit not to display the image used to set the boundaries.

10. The information processing device according to claim 1, wherein, The plurality of regions includes a first region and a second region that is different from the first region.

11. The information processing device according to claim 10, wherein, The first area is a permitted area where users are allowed to exist, and the second area is a disallowed area where users are not allowed to exist.

12. The information processing device according to claim 1, further comprising a second acquisition component, the second acquisition component being used to acquire the detection result of the position or posture of the controller. in, The control component sets the boundary based on the detection result of the position or posture of the controller acquired by the second acquisition component.

13. The information processing device according to claim 1, further comprising a detection component, the detection component being configured to detect a user's hand from the captured image acquired by the acquisition component. in, The control unit sets the boundary based on the position of the user's hand detected by the detection unit.

14. The information processing device according to claim 1, wherein, The control unit sets the boundary of a specific range based on the captured image obtained by the acquisition unit.

15. The information processing device according to claim 1, wherein, When the control component determines that the user has moved out of the boundary while the boundary has been set, it displays the captured image obtained by the acquisition component on the display unit.

16. The information processing device according to claim 1, wherein, The control unit performs control to present a warning message to the user if it determines that the user has moved out of the boundary when the boundary has been set.

17. The information processing device according to claim 1, further comprising a third acquisition component, the third acquisition component being used to acquire the user's position or posture from the captured image acquired by the acquisition component. in, The control component performs control to determine whether the user has moved out of the boundary based on the user's position or posture acquired by the third acquisition component.

18. The information processing device according to claim 1, further comprising a storage component, the storage component being used to store the boundary set by the control component or the various regions separated by the boundary. in, The control unit controls the display unit not to display the image used to set the boundary when the image to be displayed on the display unit is an image of the virtual space and the image to be displayed is controlled so that when the user is within the boundary stored in the storage unit, the display unit does not display the screen used to set the boundary.

19. The information processing device according to claim 1, wherein, The information processing device is a head-mounted display and also includes the display unit.

20. The information processing device according to claim 1, wherein, The image in the virtual space is an image that does not include the captured image.

21. An information processing device, comprising: Acquisition component, used to acquire captured images; as well as A control unit is configured to control a display unit to selectively display images of a mixed reality space and images of a virtual space, and to control the display unit to display a screen prompting the user to select whether to set a boundary in the real space when the mixed reality space image is displayed on the display unit and the user has already performed an operation to display the virtual space image on the display unit, wherein the mixed reality space image is a composite image of a captured image and an image of a virtual object acquired by the acquisition unit.

22. The information processing device according to claim 21, wherein, The control unit controls the display unit to set the boundary when the user has already set the boundary, and controls the display unit to display the image of the mixed reality space without setting the boundary when the user has already refrained from setting the boundary.

23. The information processing device according to claim 21, wherein, When the boundary is set to display the image of the virtual space as the home screen, the control unit controls the system to not set the boundary even when the application that indicates the display of the image of the virtual space is launched, wherein a predetermined application can be selected from one or more applications on the home screen.

24. An information processing device, comprising: Acquisition component, used to acquire captured images; as well as A control unit is configured to control a display unit to display an image of a mixed reality space, and to control the display unit not to display a frame for setting boundaries when the proportion of the image of the virtual object in the mixed reality space displayed on the display unit is less than a predetermined value, or to control the display unit to display a frame for setting the boundaries when the proportion of the image of the virtual object in the mixed reality space is the predetermined value or greater, wherein the image of the mixed reality space is a composite image of a captured image acquired by the acquisition unit and an image of the virtual object.

25. The information processing device according to claim 24, wherein, When the image to be displayed on the display unit is an image of the mixed reality space and the image of a virtual object with a transparency less than a predetermined value occupies a predetermined or greater proportion in the image of the mixed reality space, the control unit controls the display unit to display a screen for setting the boundary.

26. An information processing device, comprising: Acquisition component, used to acquire captured images; as well as A control unit is configured to selectively switch between a first mode and a second mode different from the first mode, and to control the switching without setting boundaries in the real space, so as not to switch to the second mode even if the user has already switched from the first mode to the second mode, until the boundaries are set. In the first mode, an image of mixed reality space is displayed on the display unit as a home screen for selecting an application. The image of mixed reality space is a composite image of a captured image and an image of a virtual object acquired by the acquisition unit. In the second mode, an image of a virtual space excluding the real space is displayed on the display unit as the home screen.

27. An information processing device, comprising: Acquisition component, used to acquire captured images; as well as A control unit for controlling the display unit to selectively display images of mixed reality space and virtual space; Used to switch to the image in the virtual space without setting boundaries when switching from an image in the mixed reality space to an image in the virtual space after the boundaries in the real space have been pre-set; Furthermore, in the event of switching from an image in the mixed reality space to an image in the virtual space before setting the boundary, the display unit is controlled to display a screen prompting the user to choose whether to set the boundary, wherein the image in the mixed reality space is a composite image of the captured image and the image of the virtual object acquired by the acquisition component.

28. A control method for an information processing device, the method comprising: The acquisition step is used to acquire captured images; as well as The control step is used to control the display unit to selectively display images of mixed reality space and images of virtual space, and to control whether to display a screen for setting the boundary in real space on the display unit based on whether the image to be displayed on the display unit is an image of mixed reality space or an image of virtual space, wherein the image of mixed reality space is a composite image of a captured image acquired by the acquisition component and an image of a virtual object.

29. A program for causing a computer to be used as a unit of the information processing apparatus according to claim 1.

30. An information processing system, comprising: Acquisition device, which is configured to acquire captured images; as well as A control device configured to control a display unit to selectively display images of a mixed reality space and images of a virtual space, and to control whether to display a screen for setting boundaries in real space on the display unit based on whether the image to be displayed on the display unit is an image of the mixed reality space or an image of the virtual space, wherein the image of the mixed reality space is a composite image of a captured image acquired by the acquisition unit and an image of a virtual object.

Images