Control system and control method

Abstract

The present technology relates to a control system and a control method that make it possible to prevent detraction from a sense of immersion. Control is carried out with respect to display of an assistance guide for assisting spatial recognition which is performed by a user in a real space and performed in a virtual space displayed on a real object that is an object in the real space, such control being in accordance with the state of the user or the state of content reproduced in the virtual space. The present technology can be applied to, for example, an XR (cross reality) experience providing system that provides an XR experience.

Description

Control system and control method

[0001] The present technology relates to a control system and a control method, and in particular to a control system and a control method that can suppress loss of immersion in an XR (cross reality) experience, for example.

[0002] An XR system called CAVE (Cave automatic virtual environment) has been developed that provides a highly immersive XR experience by displaying (images of) a virtual space on real objects in the real world, such as the walls and ceilings around the user.

[0003] For example, Patent Document 1 describes a technology for displaying a virtual space so that corresponding ridgelines between real space and virtual space, for example, the ridgeline that is the boundary between the ceiling and the left wall of the real space and the ridgeline that is the boundary between the ceiling and the left wall of the virtual space, form a straight line when the virtual space is displayed on the front wall. By displaying the virtual space so that corresponding ridgelines between real space and virtual space form a straight line, it is possible to prevent the user from feeling uncomfortable in the virtual space.

[0004] Japanese Patent Application Laid-Open No. 2006-318015

[0005] In open-type XR systems such as CAVEs, i.e., XR systems that do not require the wearer to wear devices that impose a heavy physical burden (especially those that obscure the field of view) such as head-mounted displays (or lightly worn XR systems), the immersiveness of the XR experience can be diminished. For example, when a user views a virtual space displayed on a wall, the user's depth perception, and therefore spatial awareness, can be weakened, and the virtual space displayed on the wall can be perceived as a wall with an image displayed on it (or an image displayed on the wall), resulting in a loss of immersion. In particular, when stereoscopic vision is not provided, such as by viewing left-eye and right-eye images using stereoscopic glasses with shutters, i.e., when viewing images displayed on a wall or the like without using any stereoscopic devices, spatial awareness is likely to be weakened.

[0006] This technology was developed in light of these circumstances, and makes it possible to prevent the immersive feeling of the XR experience from being lost.

[0007] The control system of the present technology is a control system that includes a display control unit that controls the display of an auxiliary guide in a virtual space that is displayed on a real object that is an object in the real space, and that assists the user in spatial recognition of the real space, depending on the situation of the user or the situation of the content being played in the virtual space.

[0008] The control method of the present technology is a control method that includes controlling the display of an auxiliary guide in a virtual space that is displayed on a real object that is an object in the real space, and that assists the user in spatial recognition of the real space, in accordance with the situation of the user or the situation of the content being played in the virtual space.

[0009] In this technology, the display of an auxiliary guide that assists a user in spatial recognition in a virtual space and is displayed on a real object that is a real-space object is controlled according to the situation of the user or the situation of the content being played in the virtual space.

[0010] The control system may be an independent device or may be an internal block constituting a single device, and one or more blocks constituting the control system may be configured as separate devices.

[0011] The control system can be realized by causing a computer to execute a program. The program can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0012] 1 is a block diagram illustrating an example configuration of an embodiment of an XR experience providing system to which the present technology is applied. FIG. 1 is a diagram illustrating an example of displaying (a VE image of) a virtual space on a real object. FIG. 2 is a diagram illustrating another example of displaying a virtual space on a real object. FIG. 3 is a diagram illustrating an example of displaying a virtual space when the texture in the real space and the texture in the virtual space are the same texture. FIG. 4 is a diagram illustrating another example of displaying a virtual space when the texture in the real space and the texture in the virtual space are the same texture. FIG. 5 is a diagram illustrating yet another example of displaying a virtual space when the texture in the real space and the texture in the virtual space are the same texture. FIG. 6 is a flowchart illustrating an example of a first display control process by the graphics generation unit 31. FIG. 7 is a diagram illustrating an example of highlighting the display of an auxiliary guide, which is performed with a degree of highlighting according to the movement speed of the user's viewpoint. FIG. 8 is a diagram illustrating an example of highlighting the display of an auxiliary guide, which is performed by increasing or decreasing the degree of highlighting the display of the auxiliary guide according to the movement speed of the user's viewpoint. FIG. 9 is a diagram illustrating an example of a second display control process by the graphics generation unit 31. FIG. 10 is a flowchart illustrating an example of the second display control process by the graphics generation unit 31. FIG. 11 is a block diagram illustrating an example of a computer to which the present technology is applied.

[0013] <Example of XR experience provision system configuration>

[0014] FIG. 1 is a block diagram showing an example of the configuration of an embodiment of an XR experience providing system to which the present technology is applied.

[0015] The XR experience providing system 11 displays a VE (virtual environment) image, which is an image of a virtual space seen from the user's viewpoint, on a real object, which is an object in the real space where the user is located, thereby providing the user with an XR experience in which a virtual space exists that is continuous with the real space.

[0016] Here, the XR experience providing system 11 is, for example, a CAVE-type XR system in which a user directly views VE images displayed on a real object such as a wall without using any tools for stereoscopic viewing (hereinafter also referred to as a direct-view XR system). However, the XR experience providing system 11 can also be applied to an XR system in which stereoscopic viewing is performed, for example, by viewing an image for the left eye and an image for the right eye using stereoscopic glasses with shutters.

[0017] The XR experience providing system 11 is capable of communicating with a virtual space storage device 12. The virtual space storage device 12 stores 3D (dimensional) data and the like for generating 3DCG (three dimensional computer graphics) of a virtual space and content played in the virtual space. The XR experience providing system 11 acquires the 3D data stored in the virtual space storage device 12 and uses the 3D data to provide an XR experience to the user.

[0018] The XR experience providing system 11 includes a real space measuring device 21, a user measuring device 22, a display device 23, and an information processing device 24.

[0019] The real-space measurement device 21 acquires information about the real space in which the user is located, i.e., the real space where the XR experience is provided, and supplies the information to the information processing device 24. For example, the real-space measurement device 21 acquires real-space information (e.g., images captured of the real space and measurements obtained by three-dimensionally measuring the real space) that is information about the real space required to recognize the shape, attributes, etc. of the real space where the XR experience is provided. Note that the real-space measurement device 21 does not need to constantly acquire real-space information. For example, if the real space where the XR experience is provided is a fixed real space such as a specific room in a specific building, the real-space measurement device 21 can retain real-space information acquired in the past and supply the information processing device 24.

[0020] The user measurement device 22 acquires user information about the user in real space, for example, user information representing the user's situation, and supplies the information processing device 24. For example, the user measurement device 22 acquires, as user information, the position of the viewpoint (and the direction of the gaze) required to change the VE image according to the user's viewpoint (gaze) (for example, to generate a VE image so that the virtual space is seen from the user's viewpoint). Note that the user measurement device 22 can acquire, for example, the position (and direction) of the user's head as the position of the user's gaze.

[0021] The display device 23 displays an image. For example, the display device 23 displays a VE image supplied from the information processing device 24 on an object (real object) in real space. For example, the display device 23 may be a projector that projects an image onto a real object such as a wall of a room. In this case, the XR experience providing system 11 becomes an XR system that provides an XR experience using a CAVE method. In addition to being configured as a CAVE-type XR system, the XR experience providing system 11 can also be configured as an XR system that provides an XR experience using a 3D monitor using a lenticular lens that displays three-dimensional images, a glasses-type or goggle-type wearable terminal, or the like.

[0022] The information processing device 24 is a control system that performs display control to display images on the display device 23. The information processing device 24 generates various images such as VE images of the virtual space using real space information supplied from the real space measurement device 21, user information supplied from the user measurement device 22, and 3D data of the virtual space acquired from the virtual space storage device 12, and supplies these to the display device 23. The information processing device 24 generates images that prevent the immersive feeling of the XR experience from being lost.

[0023] The information processing device 24 includes a graphics generation unit 31 , a rendering unit 32 , and a virtual space recognition unit 33 .

[0024] The graphics generation unit 31 generates graphics for displaying an image on the display device 23 and supplies the generated graphics to the rendering unit 32. For example, the graphics generation unit 31 uses 3D data acquired from the virtual space storage device 12 and generates 3DCG based on real space information, in which a virtual space (VE image) that is continuously connected to the real space is displayed.

[0025] Here, a VE image of the virtual space seen from the user's viewpoint is generated using 3DCG, and the VE image of the virtual space is displayed on a real object, for example, a wall in the real space, on the display device 23. By looking at (the VE image of) the virtual space displayed on the wall, the user can enjoy the sensation of the existence of a space that is continuously connected to the real space.

[0026] The graphics generation unit 31 functions as a display control unit that controls the display of an auxiliary guide that assists the user in spatial recognition in the virtual space, for example, displayed as a real object on a wall in the real space, in accordance with the user's situation or the situation of the content being played in the virtual space. For example, the graphics generation unit 31 generates graphics that control the display of the auxiliary guide in accordance with the user's situation or the situation of the content being played in the virtual space, and displays an image corresponding to the graphics on the display device 23. The user's situation is recognized from user information supplied from the user measurement device 22 to the information processing device 24. The situation of the content being played in the virtual space is recognized from virtual space information supplied from the virtual space recognition unit 33 to the graphics generation unit 31.

[0027] The rendering unit 32 uses the graphics supplied from the graphics generation unit 31 to generate an image to be displayed on the display device 23, and supplies the image to the display device 23. For example, the rendering unit 32 generates a VE image by rendering the 3DCG of the virtual space supplied from the graphics generation unit 31 based on the user's viewpoint recognized from the user information. Also, for example, the rendering unit 32 uses the graphics supplied from the graphics generation unit 31 to generate an image to be displayed on a real object other than the real object on which the virtual space (VE image) is displayed.

[0028] The virtual space recognition unit 33 recognizes the situation of the virtual space in which the 3DCG is generated by the graphics generation unit 31, and supplies virtual space information representing that situation to the graphics generation unit 31.

[0029] Here, the graphics generation unit 31 can generate 3DCG of a virtual space in which predetermined content is being played (or being played). For example, if a concert performed by a virtual idol character is the predetermined content, the 3DCG of the virtual space in which the concert is being held can be generated. The virtual space recognition unit 33 can recognize the status of the content being played in the virtual space as the status of the virtual space, include it in virtual space information, and supply it to the graphics generation unit 31.

[0030] <Example of virtual space display>

[0031] FIG. 2 is a diagram showing an example of displaying (VE image of) a virtual space on a real object.

[0032] FIG. 2 shows an example of a display in which a VE image 60 of a virtual space is displayed on a wall 51 in front of the real space as a real object.

[0033] The virtual space (VE image 60) is displayed so that, as viewed from the user, the virtual ridges, which are the ridges of the virtual space, are extensions of the real ridges, which are the ridges in the real space that correspond to those virtual ridges. For example, the virtual space is displayed so that the virtual ridges are aligned with the real ridges that correspond to those virtual ridges. A ridge is a boundary line between surfaces, and may be either a straight line or a curved line.

[0034] In Figure 2, of the partitions (wall, ceiling, floor) in the real space that are adjacent to the front wall 51 on which the virtual space is displayed, two partitions that form a boundary are displayed with virtual edges corresponding to the real edges that form the boundary, and are displayed so that they are in a straight line with the real edges.

[0035] Specifically, an actual edge line 71, which is the boundary line between the left wall 52 (as viewed from the front wall 51) and the ceiling 54 in the real space, and a virtual edge line 81 corresponding to the actual edge line 71, i.e., the virtual edge line 81, which is the boundary line between the left wall 62 and the ceiling 64 in the virtual space, are aligned in a straight line. Also, an actual edge line 72, which is the boundary line between the right wall 53 and the ceiling 54 in the real space, and a virtual edge line 82 corresponding to the actual edge line 72, i.e., the virtual edge line 82, which is the boundary line between the right wall 63 and the ceiling 64 in the virtual space, are aligned in a straight line. Furthermore, an actual edge line 73, which is the boundary line between the left wall 52 and the floor 55 in the real space, and a virtual edge line 83 corresponding to the actual edge line 73, i.e., the virtual edge line 83, which is the boundary line between the left wall 62 and the floor 65 in the virtual space, are aligned in a straight line. In addition, the real edge line 74, which is the boundary line between the right wall 53 and the floor 55 in the real space, and the virtual edge line 84 corresponding to the real edge line 74, i.e., the virtual edge line 84, which is the boundary line between the right wall 63 and the floor 65 in the virtual space, are in a straight line.

[0036] Because the real ridge lines and the virtual ridge lines corresponding to those real ridge lines are aligned, the virtual ridge lines appear to the user as extensions of the real ridge lines corresponding to those real ridge lines. As described above, by displaying the virtual space so that, from the user's perspective, the virtual ridge lines appear as extensions of the real ridge lines corresponding to those virtual ridge lines, the virtual space appears (is more easily visible) to the user as a space that continues beyond the front wall 51 of the real space in which the virtual space is displayed.

[0037] In the virtual space, instead of displaying all of the virtual ridges corresponding to the actual ridges of two partitions that form a boundary between the real space and the partitions that are in contact with the front wall 51 on which the virtual space is displayed, in line with the actual ridges, only parts of the virtual ridges can be displayed in line with the actual ridges. For example, of the virtual ridges 81 to 84, only the virtual ridges 83 and 84 formed by the floor 65 can be displayed in line with the corresponding actual ridges 73 and 74, respectively.

[0038] FIG. 3 is a diagram showing another example of displaying a virtual space on a real object.

[0039] In FIG. 3, a wall 111 in the real space is the front wall, and the virtual space (VE image 60) is displayed on the front wall 111 and a wall 112 on the left side of the front wall 111.

[0040] 3 , among the partitions (walls, ceiling, and floor) in the real space that are adjacent to the wall 111 or 112 on which the virtual space is displayed, excluding the walls 111 and 112, virtual edges corresponding to the actual edges of two partitions that form a boundary are displayed so as to form a straight line with the actual edges. For example, a virtual edge corresponding to the actual edge that is the boundary between the wall to the right of wall 111 and the floor 115 is displayed so as to form a straight line with the actual edges. As a result, the virtual space appears to the user as a space that continues beyond the walls 111 and 112 of the real space on which the virtual space is displayed.

[0041] In the following, it is assumed that the virtual space is displayed on the front wall 111 and the left wall 112 of the real space, as shown in FIG.

[0042] Regarding the real space and the virtual space, the texture in the real space and the texture in the virtual space can be the same texture (texture that looks the same), which gives the user a stronger sense that the virtual space is a space that is connected to the real space (a sense of continuity as if the virtual space and the real space are continuous spaces).

[0043] FIG. 4 is a diagram showing an example of a virtual space display in which the texture in the real space and the texture in the virtual space are the same texture.

[0044] In Figure 4, similar to Figure 3, virtual spaces are displayed on the front wall 111 and the left wall 112 of the real space. In Figure 4, the textures of the parts of the real space and the virtual space that connect to each other are the same. In Figure 4, the virtual texture (image) that is the texture of the virtual space is made the same as the texture in the real space, so that the virtual texture of the virtual space and the texture of the real space are made the same. That is, in Figure 4, the virtual texture of the floor 125 of the virtual space that connects to the floor 115 of the real space is made the same as the texture of the floor 125 of the real space, so that the virtual texture of the floor 125 of the virtual space and the texture of the floor 115 of the real space are made the same. In the virtual space, the virtual texture of the walls is also the same as the texture of the walls of the real space, just like the textures of the floors 115 and 125.

[0045] In FIG. 4, the texture of the floor 115 in the real space is made up of large tile joints, and the virtual texture of the floor 125 in the virtual space is also made up of the same large tile joints as the texture of the floor 115.

[0046] The graphics generation unit 31 recognizes the texture of the floor 115 in the real space from the real space information. Then, the graphics generation unit 31 generates graphics in which the virtual texture of the floor 125 in the virtual space is the same as the texture of the floor 115 in the real space, thereby displaying (a VE image of) a virtual space in which the virtual texture of the floor 125 in the virtual space is the same as the texture of the floor 115 in the real space.

[0047] As described above, when the textures of the parts of the real space and the virtual space that connect to each other are the same texture, the user can be given a stronger sense that the virtual space is a space that is connected to the real space.

[0048] FIG. 5 is a diagram showing another example of a virtual space display in which the texture in the real space and the texture in the virtual space are the same texture.

[0049] In Figure 5, similar to Figure 3, virtual spaces are displayed on the front wall 111 and the left wall 112 of the real space. In Figure 5, the textures of the parts of the real space and the virtual space that connect to the other space are the same. In Figure 5, an image of a texture that is the same as the virtual texture of the virtual space is displayed in the real space, so that the virtual texture and the texture of the real space are the same texture. That is, in Figure 5, an image of a texture that is the same as the virtual texture of the floor 125 of the virtual space is displayed (projected) on the floor 115, which is a real object (another real object) in the real space that connects to the virtual texture of the floor 125, so that the virtual texture of the floor 125 of the virtual space and the texture of the floor 115 of the real space are (appear to be) the same texture. In the real space and the virtual space, the texture of the walls is also the same texture, as are the textures of the floors 115 and 125.

[0050] In Fig. 5, the virtual texture of the floor 125 of the virtual space is made up of smaller tile joints than in Fig. 4, and an image of the same texture as that virtual texture is displayed on the floor 115 of the real space. This makes it appear to the user that the texture of both the floor 125 of the virtual space and the floor 115 of the real space is made up of small tile joints.

[0051] The graphics generation unit 31 recognizes the virtual texture of the floor 125 of the virtual space from the virtual space information. Then, the graphics generation unit 31 generates graphics that display (an image of) the same texture as the virtual texture of the floor 125 of the virtual space on the floor 115 of the real space, thereby displaying the same texture as the virtual texture of the floor 125 of the virtual space on the floor 115 of the real space. The graphics generated by the graphics generation unit 31 are used by the rendering unit 32 as an image of the texture to be displayed over the entire floor 115 of the real space. This texture (an image of) is displayed on the floor 115 by projecting it over the entire floor 115 of the real space in parallel with the projection of (a VE image of) the virtual space onto the walls 111 and 112 by the display device 23. As a result, the floor 125 of the virtual space and the floor 115 of the real space appear to the user to have the same texture.

[0052] As described above, when the textures of the parts of the real space and the virtual space that connect to the other space are (appear to be) the same texture, the user can be given a stronger sense that the virtual space is a space that is connected to the real space.

[0053] In Figures 4 and 5, the textures of the floor 115 in the real space and the floor 125 in the virtual space are the same texture, but the textures of the parts of the real space and the virtual space other than the floor 115 in the real space and the floor 125 in the virtual space that connect to the other space, for example, the textures of the walls (or ceiling) in the real space and the walls in the virtual space that connect to those walls, can also be the same texture.

[0054] FIG. 6 is a diagram showing yet another example of displaying a virtual space in which the texture in the real space and the texture in the virtual space are the same texture.

[0055] In Fig. 6, similar to the case of Fig. 3, a virtual space is displayed on the front wall 111 and the left wall 112 of the real space. Furthermore, in Fig. 6, similar to the case of Fig. 5, (an image of) a texture identical to the virtual texture of floor 125 in the virtual space is displayed on floor 115, which is a real object (another real object) in the real space connected to the virtual texture of floor 125, so that the virtual texture of floor 125 in the virtual space and the texture of floor 115 in the real space are the same texture.

[0056] However, while the virtual texture of the floor 125 of the virtual space in Fig. 5 is a static texture, the virtual texture of the floor 125 of the virtual space in Fig. 6 is a dynamic texture. For example, in Fig. 6, the virtual texture of the floor 125 of the virtual space is tile joints along which light moves. In Fig. 6, the same texture (image of) the tile joints along which light moves as the virtual texture is displayed on the floor 115 of the real space.

[0057] As described above, even when the textures of the real space and the virtual space are the same and have movement, the user can be given a stronger sense that the virtual space is a space connected to the real space.

[0058] Here, a person can sense depth and perceive space through, for example, motion parallax (motion parallax). Therefore, for example, particularly when the XR experience providing system 11 is a direct-view XR system, when the user's viewpoint is not moving, motion parallax is lost, and spatial perception of the virtual space displayed on the walls 111 and 112 is not achieved through motion parallax, but rather through binocular parallax, resulting in a loss of immersion in the XR experience. Planar perception through binocular parallax means, for example, that, due to the (almost) absence of binocular parallax for the wall 111 (or 112), the user does not perceive the virtual space displayed on the wall 111 as a space with depth, but rather as a plane, which is the wall 111 on which the image of the virtual space is displayed.

[0059] Furthermore, for example, when the virtual space displayed on the walls 111 and 112 is updated, the user's spatial awareness may be weakened, and the sense of immersion may be lost. For example, when the virtual space (VE image) displayed on the walls 111 and 112 is updated to the virtual space seen from a viewpoint (of the camera) as the viewpoint (of the camera) moves due to camerawork, the user may be forced to repeatedly re-understand the updated virtual space, which may result in a loss of immersion.

[0060] Furthermore, for example, when the user's viewpoint moves and the virtual space (VE image) displayed on walls 111 and 112 is updated to the virtual space seen from the user's viewpoint after the movement, the delay between the movement of the user's viewpoint and the update of the virtual space displayed on walls 111 and 112 may weaken the user's spatial awareness and reduce the sense of immersion.

[0061] Therefore, for example, by using textures with the same movement for both the real space and the virtual space, it is possible to constantly generate a certain degree of motion parallax, as shown in Fig. 6. In this case, the motion parallax makes it easier for the user to perceive depth, and the user is more likely to perceive the virtual space as a space connected to the real space.

[0062] Furthermore, the graphics generation unit 31 performs a display control process to control the display of an auxiliary guide in the virtual space that assists the user in spatial recognition, depending on the user's situation or the situation of the content being played in the virtual space. The display control process by the graphics generation unit 31 reinforces the user's spatial recognition (making spatial recognition easier to perform) and can prevent the sense of immersion from being impaired.

[0063] <First display control process>

[0064] FIG. 7 is a flowchart illustrating an example of the first display control process performed by the graphics generation unit 31.

[0065] The graphics generation unit 31 performs a first display control process for controlling the display of an auxiliary guide in the virtual space that assists the user in spatial recognition, depending on the user's situation.

[0066] In the first display control process, in step S11, the graphics generation unit 31 recognizes the user's situation in real space from the user information, and the process proceeds to step S12. For example, the graphics generation unit 31 recognizes the situation of the user's viewpoint.

[0067] In step S12, the graphics generation unit 31 controls the display of an auxiliary guide that assists the user in spatial recognition according to the user's situation, and the process returns to step S11. For example, when the user's viewpoint is moving, the graphics generation unit 31 emphasizes the display of the auxiliary guide according to the movement of the viewpoint. That is, the graphics generation unit 31 generates a 3DCG of the virtual space in which the auxiliary guide is emphasized, and displays (a VE image of) the virtual space corresponding to the 3DCG on the walls 111 and 112.

[0068] An auxiliary guide is something that assists the user in spatial recognition within a virtual space, and examples of such an auxiliary guide include a virtual texture and a virtual edge line. The virtual texture and virtual edge line to be used as an auxiliary guide within the virtual space can be set (selected). For example, of the virtual textures within the virtual space, only the virtual texture of the floor 125 can be set as an auxiliary guide. Furthermore, for example, of the virtual textures and / or virtual edge lines within the virtual space, the virtual texture and / or virtual edge line of a (virtual) object within a certain distance from the position in the virtual space where the user is gazing can be set as an auxiliary guide.

[0069] The display of the auxiliary guide can be emphasized by, for example, changing the brightness of the auxiliary guide, changing the line thickness, changing the color intensity, and changing the number of lines. For example, the display of the auxiliary guide can be emphasized by, for example, brightening or darkening the color, thickening or thinning the lines, darkening or lightening the color, and increasing or decreasing the number of lines. Specifically, the display of the auxiliary guide can be emphasized by, for example, brightening the tile joints and / or virtual ridge lines as the virtual texture of the floor 125 or thickening the tile joints and / or virtual ridge lines. Furthermore, the display of the auxiliary guide can be emphasized by, for example, darkening the color of the tile joints and / or virtual ridge lines, increasing the number of lines representing the tile joints and / or virtual ridge lines, etc.

[0070] The highlighting of the display of the auxiliary guide can be accompanied by movement (animation). For example, instead of simultaneously darkening / lightening the entire virtual texture or virtual edge line as the auxiliary guide, the virtual texture or virtual edge line as the auxiliary guide can be made darker / lighter so that the darker / lighter portions gradually expand in a gradation from one side of the virtual texture or virtual edge line as the auxiliary guide to the other, for example, from the side closer to the user (the front side) to the side farther from the user (the back side), or from the side farther from the user to the closer side, or so that the darker / lighter portions move. Such highlighting with movement can be performed only once (in one shot) or can be performed multiple times (in a loop).

[0071] Another example of emphasizing the display of the auxiliary guide is to change the pattern of the virtual texture serving as the auxiliary guide. For example, if the virtual texture of floor 125 is a solid texture, the display of the virtual texture serving as the auxiliary guide can be emphasized by changing the texture so that the tile joints appear to be added from the side closer to the user to the side farther from the user, or from the side farther from the user to the side closer to the user. This emphasis of changing the pattern of the virtual texture can be performed only once, or can be performed multiple times.

[0072] If the virtual texture is a moving texture as described in Fig. 6, one or more of changing the speed of the movement and changing the magnitude of the movement can be adopted to emphasize the display of the virtual texture as an auxiliary guide. For example, if the virtual texture as an auxiliary guide is the joints of tiles along which light moves as described in Fig. 6, the speed of light moving along the joints can be increased to emphasize the display of the virtual texture as an auxiliary guide.

[0073] When emphasizing the display of the auxiliary guide, a predetermined effect can be added, for example, an effect of light such as a spotlight or external light shining in. For example, light can be shined in and reflected off the tile joints, which are the virtual texture of the auxiliary guide, to emphasize the tile joints by making them brighter or more gleaming.

[0074] As described in Fig. 5, when a texture identical to the virtual texture of the virtual space is displayed in the real space, the texture displayed in the real space can be emphasized in the same way as the virtual texture of the virtual space as an auxiliary guide. For example, as described in Fig. 5, when a texture identical to the virtual texture of the floor 125 of the virtual space is displayed on the floor 115 of the real space, the texture (image) displayed on the floor 115 of the real space (another real object) can be emphasized together with the emphasis of the virtual texture of the floor 125 of the virtual space as an auxiliary guide. By emphasizing the texture displayed on the floor 115 of the real space together with the emphasis of the virtual texture of the floor 125 of the virtual space as an auxiliary guide, the user can be given the sensation that the virtual space is a space connected to the real space.

[0075] When the user's viewpoint is moving, emphasizing the display of the auxiliary guide reinforces the user's spatial awareness (making spatial awareness easier), thereby preventing the loss of immersion.

[0076] For example, as described above, when the user's viewpoint moves and the virtual space displayed on the walls 111 and 112 is updated to the virtual space seen from the user's viewpoint after the movement, the user's spatial awareness may be weakened and the sense of immersion may be lost due to the delay between the user's viewpoint moving and the virtual space displayed on the walls 111 and 112 being updated. In a situation where the user's viewpoint is moving, by emphasizing the display of the auxiliary guide, the user's spatial awareness is reinforced and the loss of immersion due to the delay described above can be prevented.

[0077] The display of the auxiliary guide can be emphasized not only depending on whether the user's viewpoint is moving, but also depending on other user situations. For example, the display of the auxiliary guide can be emphasized when the user's viewpoint has not moved for a predetermined period of time. For example, as described above, when the user's viewpoint is not moving, motion parallax is lost, and the virtual space displayed on the walls 111 and 112 is not perceived as a space by motion parallax, but rather as a plane by binocular parallax, which can result in a loss of immersion. By emphasizing the display of the auxiliary guide when the user's viewpoint has not moved for a predetermined period of time, the user's spatial perception can be reinforced and the loss of immersion due to the loss of motion parallax can be prevented.

[0078] When highlighting the display of the auxiliary guide in accordance with the movement of the user's viewpoint, the display of the auxiliary guide can be highlighted at a level that corresponds to the speed at which the user's viewpoint moves, or the level of highlighting of the display of the auxiliary guide can be increased or decreased in accordance with the speed at which the user's viewpoint moves.

[0079] FIG. 8 is a diagram showing an example of highlighting the display of the auxiliary guide, which is performed at a degree of highlighting according to the moving speed of the user's viewpoint.

[0080] 8, the display of the auxiliary guide is emphasized by changing the darkness (thinness) of the virtual edge line serving as the auxiliary guide. The darkness of the virtual edge line serving as the auxiliary guide is set to a darkness that corresponds to the moving speed of the user's viewpoint, for example, a darkness that is proportional to the moving speed of the viewpoint.

[0081] FIG. 9 is a diagram showing an example of highlighting the display of the auxiliary guide, in which the degree of highlighting of the display of the auxiliary guide is increased or decreased depending on the moving speed of the user's viewpoint.

[0082] 9, as in FIG. 8, the display of the auxiliary guide is emphasized by changing the density of the virtual edge line serving as the auxiliary guide. The density of the virtual edge line serving as the auxiliary guide gradually increases until it reaches a predetermined density when the moving speed of the user's viewpoint is greater than a threshold. Furthermore, the density of the virtual edge line serving as the auxiliary guide gradually decreases until it reaches a predetermined shading (e.g., the original shading) when the moving speed of the user's viewpoint is equal to or less than the threshold. For example, if the threshold is set to 0, the density of the virtual edge line serving as the auxiliary guide increases to a predetermined shading while the user's viewpoint is moving, and decreases until it reaches a predetermined shading, e.g., the original shading, when the user's viewpoint stops.

[0083] <Second Display Control Process>

[0084] FIG. 10 is a diagram illustrating an example of the second display control process performed by the graphics generation unit 31. In FIG.

[0085] The graphics generation unit 31 performs a second display control process for controlling the display of an auxiliary guide in the virtual space that assists the user in spatial recognition, in accordance with the status of the content being played back in the virtual space.

[0086] Fig. 10 shows a state in which predetermined content is being played in a virtual space displayed on walls 111 and 112. In Fig. 10, a concert performed by a character 131 who is a virtual idol is set as the predetermined content, and the predetermined content is being played in the virtual space.

[0087] In the predetermined content, a (virtual) character 131 is illuminated by light 132 from a (virtual) lamp, and the light 132 from the lamp forms a (virtual) spot of light 133 and a (virtual) shadow 134 of the character 131 on the floor 125 of the virtual space. Therefore, the predetermined content is made up of the character 131, the light 132 from the lamp, the spot of light 133, the shadow 134 of the character 131, etc.

[0088] As a second display control process, when an object constituting the content protrudes from wall 111 or 112, which is a real object on which the virtual space is displayed, the graphics generation unit 31 uses the protruding object as an auxiliary guide, generates graphics of the protruding part of the object as an auxiliary guide, and displays an image corresponding to the graphics on floor 115, which is another real object adjacent to the side of wall 111 or 112, which is a real object on which the virtual space is displayed, on which the object protrudes.

[0089] The objects that make up the content (hereinafter also referred to as content objects) include tangible objects (including (virtual) living things, including (virtual) people) and intangible objects. In Fig. 10, a character 131 is a tangible object serving as a content object, and a light beam 132, a light spot 133, and a shadow 134 are intangible objects serving as content objects.

[0090] 10 , lighting light 132 is directed from the upper back side toward the lower front side toward wall 111, causing a spot of light 133 and a shadow 134 of character 131 to extend beyond the wall 111 on which the virtual space is displayed. Therefore, if no countermeasure is taken, the portions of the spot of light 133 and the shadow 134 that extend beyond the wall 111 will not be displayed on the wall 111 (the virtual space displayed on the wall 111), and the spot of light 133 and the shadow 134 will be cut off. In this case, the user's spatial awareness may be weakened, and the sense of immersion may be impaired.

[0091] Therefore, the graphics generation unit 31 identifies the light spot 133 and the shadow 134 that extend beyond the wall 111 on which the virtual space is displayed as auxiliary guides, and generates graphics of the protruding portions of the light spot 133 and the shadow 134 as auxiliary guides. The graphics generation unit 31 then displays images corresponding to the graphics of the protruding portions of the light spot 133 and the shadow 134 as auxiliary guides on the floor 115, which is another real object that is in contact with the sides of the walls 111 and 112 on which the virtual space is displayed on which the light spot 133 and the shadow 134 extend, so as to connect to the light spot 133 and the shadow 134 in the virtual space.

[0092] In this case, the light spot 133 and the shadow 134 serving as auxiliary guides are displayed seamlessly across the real space and the virtual space displayed on the walls 111 and 112. As a result, the user can be given a stronger sense that the virtual space is a space connected to the real space, and loss of immersion can be prevented.

[0093] In Figure 10, in order to make it easier to distinguish between the part of the shadow 134 that is contained within the virtual space displayed on the walls 111 and 112, i.e., the part that is displayed on the virtual space (on the walls 111 and 112 on which it is displayed), and the part that extends beyond the virtual space (on the walls 111 and 112 on which it is displayed), i.e., the part that is displayed on the floor 115, the shades of each part are shown to be different, but the shades of the part that is contained within the virtual space and the part that extends beyond it can be the same.

[0094] FIG. 11 is a flowchart illustrating an example of the second display control process by the graphics generation unit 31.

[0095] In the second display control process, in step S31, the graphics generation unit 31 recognizes the status of the content being played in the virtual space displayed on the walls 111 and 112 from the virtual space information, and the process proceeds to step S32.

[0096] In step S32, the graphics generation unit 31 controls the display of an auxiliary guide that assists the user in spatial recognition according to the situation of the content, and the process returns to step S31. For example, when an object constituting the content protrudes from the wall 111 or 112 that is a real object on which the virtual space is displayed, the graphics generation unit 31 uses the protruding object as an auxiliary guide, generates graphics of the protruding portion of the object as an auxiliary guide, and displays an image corresponding to the graphics on the floor 115 that is another real object that contacts the protruding side of the wall 111 or 112 on which the virtual space is displayed.

[0097] <Description of a computer to which this technology is applied>

[0098] Next, the above-described series of processes can be performed by hardware or software. When the series of processes is performed by software, the programs that make up the software are installed on a general-purpose computer or the like.

[0099] FIG. 12 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0100] The program can be recorded in advance on the hard disk 905 or ROM 903 as a recording medium built into the computer.

[0101] Alternatively, the program can be stored (recorded) on a removable recording medium 911 driven by the drive 909. Such a removable recording medium 911 can be provided as a so-called package software. Here, examples of the removable recording medium 911 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0102] The program can be installed into the computer from the removable recording medium 911 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 905. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer via a wired network such as a LAN (Local Area Network) or the Internet.

[0103] The computer includes a CPU (Central Processing Unit) 902 , to which an input / output interface 910 is connected via a bus 901 .

[0104] When a user inputs a command via an input / output interface 910 by operating an input unit 907, the CPU 902 executes a program stored in a read-only memory (ROM) 903 in accordance with the command. Alternatively, the CPU 902 loads a program stored on a hard disk 905 into a random access memory (RAM) 904 and executes the program.

[0105] As a result, the CPU 902 performs processing according to the flowchart described above or processing performed by the configuration of the block diagram described above. Then, the CPU 902 outputs the processing results from the output unit 906 via the input / output interface 910, or transmits them from the communication unit 908, or further records them on the hard disk 905, as necessary.

[0106] The input unit 907 is made up of a keyboard, a mouse, a microphone, etc. The output unit 906 is made up of an LCD (Liquid Crystal Display), a speaker, etc.

[0107] In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or object-based processing).

[0108] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0109] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0110] It should be noted that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the present technology.

[0111] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0112] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0113] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0114] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0115] The present technology can have the following configurations.

[0116] <1> A control system including a display control unit that controls the display of an auxiliary guide in a virtual space that is displayed on a real object that is a real-space object, the auxiliary guide assisting a user in spatial recognition of the real space, depending on the user's situation or the situation of content being played in the virtual space. <2> The control system described in <1>, in which the display control unit emphasizes the display of the auxiliary guide depending on the user's situation. <3> The control system described in <2>, in which the display control unit emphasizes the display of the auxiliary guide depending on movement of the user's viewpoint. <4> The control system described in <3>, in which the display control unit emphasizes the display of the auxiliary guide with a degree of emphasis depending on a movement speed of the viewpoint. <5> The control system described in <3>, in which the display control unit increases or decreases a degree of emphasis of the display of the auxiliary guide depending on the movement speed of the viewpoint. <6> The control system described in any of <2> to <5>, in which the auxiliary guide is a virtual texture that is a texture in the virtual space, or a virtual ridge that is a ridge in the virtual space. <7> The control system according to <6>, wherein the display control unit emphasizes the virtual texture or the virtual edge line by one or more of changing brightness, line thickness, color intensity, and the number of lines. <8> The control system according to <6> or <7>, wherein the virtual texture is a moving texture. <9> The control system according to <8>, wherein the display control unit emphasizes the virtual texture by one or more of changing the speed and magnitude of the movement. <10> The control system according to any of <6> to <9>, wherein the virtual texture is a tile joint. <11> The control system according to any of <6> to <10>, wherein the display control unit displays a texture identical to the virtual texture on another real object connected to the virtual texture. <12> The control system according to <11>, wherein the texture identical to the virtual texture is displayed on the other real object by projection using a projector. <13> The control system according to <11>, wherein the display control unit emphasizes a texture displayed on the other real object together with the virtual texture.<14> The control system according to <6>, wherein the virtual texture is the same texture as the texture of the real space. <15> The control system according to <1>, wherein the auxiliary guide is an object that constitutes the content played back in the virtual space and that protrudes from the real object on which the virtual space is displayed. <16> The control system according to <15>, wherein the display control unit displays the protruding portion of the object on another real object that is adjacent to the real object on which the virtual space is displayed, on the side from which the object protrudes. <17> The control system according to <16>, wherein the protruding portion of the object is displayed on the other real object by projection by a projector. <18> The control system according to any of <15> to <17>, wherein the object is a shadow or light. <19> The control system according to any of <1> to <18>, wherein the virtual space is displayed on the real object by projection by a projector. <20> A control method including controlling the display of an auxiliary guide in a virtual space that is displayed on a real object that is an object in the real space, the auxiliary guide assisting the user in spatial recognition of the real space, in accordance with the status of the user or the status of content being played in the virtual space.

[0117] 11 XR experience provision system, 12 Virtual space storage device, 21 Real space measurement device, 22 User measurement device, 23 Display device, 24 Information processing device, 31 Graphics generation unit, 32 Rendering unit, 33 Virtual space recognition unit, 51-53 Wall, 54 Ceiling, 55 Floor, 60 VE image, 62, 63 Wall, 64 Ceiling, 65 Floor, 71-74 Real edge line, 81-84 Virtual edge line, 111, 112 Wall, 115, 125 Floor, 131 Character, 132 Light source, 133 Light spot, 134 Shadow, 901 Bus, 902 CPU, 903 ROM, 904 RAM, 905 Hard disk, 906 Output unit, 907 Input unit 908 Communication unit, 909 Drive, 910 Input / output interface, 911 Removable recording medium

Claims

1. A control system including a display control unit that controls the display of auxiliary guides in a virtual space that are displayed on real objects in the real space to assist the user in spatial recognition of the real space, depending on the situation of the user or the situation of the content being played in the virtual space.

2. The control system according to claim 1, wherein the display control unit emphasizes the display of the auxiliary guide depending on the user's situation.

3. The control system according to claim 2, wherein the display control unit emphasizes the display of the auxiliary guide in accordance with movement of the user's viewpoint.

4. The control system according to claim 3, wherein the display control unit emphasizes the display of the auxiliary guide with a degree of emphasis according to the moving speed of the viewpoint.

5. The control system according to claim 3, wherein the display control unit increases or decreases the degree of emphasis of the display of the auxiliary guide in accordance with the moving speed of the viewpoint.

6. The control system according to claim 2, wherein the auxiliary guide is a virtual texture that is a texture in the virtual space, or a virtual edge that is a edge in the virtual space.

7. The control system of claim 6, wherein the display control unit emphasizes the virtual texture or the virtual edge line by one or more of changing the brightness, changing the line thickness, changing the color intensity, and changing the number of lines.

8. The control system according to claim 6, wherein the virtual texture is a moving texture.

9. The control system according to claim 8, wherein the display control unit enhances the virtual texture by one or more of changing the speed of the movement and changing the magnitude of the movement.

10. The control system of claim 6, wherein the virtual texture is a tile joint.

11. The control system according to claim 6, wherein the display control unit displays the same texture as the virtual texture on another real object connected to the virtual texture.

12. A control system according to claim 11, wherein a texture identical to the virtual texture is displayed on the other real object by projection using a projector.

13. The control system according to claim 11, wherein the display control unit also emphasizes textures displayed on the other real objects along with the virtual texture.

14. The control system according to claim 6, wherein the virtual texture is the same as the texture of the real space.

15. The control system of claim 1, wherein the auxiliary guide is an object that constitutes the content played back in the virtual space and that extends beyond the real object in which the virtual space is displayed.

16. A control system as described in claim 15, wherein the display control unit displays the protruding portion of the object on another real object adjacent to the side of the real object on which the virtual space is displayed that the protruding portion of the object.

17. A control system according to claim 16, wherein the protruding portion of the object is displayed on the other real object by projection using a projector.

18. The control system of claim 15, wherein the object is a shadow or a light.

19. A control system according to claim 1, wherein the virtual space is displayed on the real object by projection using a projector.

20. A control method including controlling the display of an auxiliary guide in a virtual space that is displayed on a real object that is a real-space object and that assists the user in spatial recognition of the real space, in accordance with the status of the user or the status of content being played in the virtual space.

Images