Information processing device, information processing method, and recording medium

The information processing device evaluates and modifies virtual environments to match real environments, addressing the challenge of consistency and enhancing VR realism by improving compatibility between the two, thus providing a more immersive experience without the need for cumbersome VR equipment.

WO2024241849A1PCT designated stage expired Publication Date: 2024-11-28SONY GROUP CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/016839
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2024-05-02
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Conventional VR systems face challenges in providing a highly immersive experience without the need for cumbersome head-mounted displays, as they struggle to maintain consistency between the real and virtual environments, leading to a compromised sense of realism due to the difficulty in manually adjusting and matching textures and attributes between various floor plans and virtual environments.

Method used

An information processing device with a modification content determination unit that evaluates the consistency between the real and virtual environments and determines necessary modifications, coupled with a virtual environment modification unit that adjusts the virtual environment accordingly, to enhance the sense of realism by improving compatibility between the two environments.

Benefits of technology

The system effectively improves the sense of realism in VR experiences by dynamically modifying the virtual environment to match the real environment, reducing inconsistencies and enhancing user immersion without requiring manual adjustments or dedicated virtual environment preparation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024016839_28112024_PF_FP_ABST
    Figure JP2024016839_28112024_PF_FP_ABST
Patent Text Reader

Abstract

This disclosure relates to an information processing device, an information processing method, and a recording medium that make it possible to further improve the sense of being present in a VR experience. An environment recognition processing unit recognizes attributes of a real environment and attributes of a virtual environment. A modification contents determination unit evaluates consistency according to the difference between the attributes of the real environment and the attributes of the virtual environment, and determines modification contents for modifying the virtual environment according to the evaluation. A virtual environment modification unit modifies the virtual environment according to the modification contents. The present technology can be applied to, for example, a VR experience provision system.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to an information processing device, an information processing method, and a recording medium, and more particularly to an information processing device, an information processing method, and a recording medium that can further improve the sense of realism of a VR experience.

[0002] Conventionally, a VR system called CAVE (Cave automatic virtual environment) has been developed, which provides a highly immersive VR (Virtual Reality) experience by displaying images of a virtual environment on walls, ceilings, and other surfaces surrounding the user.

[0003] For example, Patent Document 1 discloses an image display system that displays an image according to the user's viewpoint, and Patent Document 2 discloses an image synthesis device that creates a sense of realism as if the interior of the user's room and the interior of the other person's room displayed on the screen are spatially continuous.

[0004] JP 2006-318015 A JP 2017-143376 A

[0005] Incidentally, when introducing open-type VR like the CAVE described above (i.e., VR that does not require the wearing of devices that place a heavy burden on the body (especially those that block the field of vision) such as a head-mounted display, or VR that is lightly worn) into a home, it is difficult to provide a screen that surrounds the user, so the real environment ends up entering the user's field of vision. In this case, if the real environment and the virtual environment are not consistent, the sense of realism of the VR experience will be lost.

[0006] For example, the sense of realism of a VR experience can be improved by manually creating a highly consistent, dedicated virtual environment, but this is time-consuming. Furthermore, manually adjusting everything to improve consistency between real environments with various floor plans and arbitrary virtual environments is costly. While matching the texture of a virtual environment to the real environment is being considered, there is a need for it to be compatible with arbitrary virtual environments, not just dedicated virtual environments.

[0007] The present disclosure has been made in consideration of such circumstances, and aims to make it possible to further improve the sense of realism of the VR experience.

[0008] An information processing device according to one aspect of the present disclosure includes a modification content determination unit that evaluates the consistency between a real environment and a virtual environment and determines the modification content to modify the virtual environment in accordance with the evaluation, and a virtual environment modification unit that modifies the virtual environment in accordance with the modification content.

[0009] An information processing method according to one aspect of the present disclosure includes an information processing device evaluating the consistency between a real environment and a virtual environment, determining modification content for modifying the virtual environment in accordance with the evaluation, and modifying the virtual environment in accordance with the modification content.

[0010] A recording medium according to one aspect of the present disclosure has recorded thereon a program for causing a computer of an information processing device to execute information processing including evaluating the consistency between a real environment and a virtual environment, determining modification content for modifying the virtual environment in accordance with the evaluation, and modifying the virtual environment in accordance with the modification content.

[0011] In one aspect of the present disclosure, the consistency between the real environment and the virtual environment is evaluated, modifications to the virtual environment are determined based on the evaluation, and the virtual environment is modified based on the modifications.

[0012] 1 is a block diagram illustrating an example configuration of an embodiment of a VR experience providing system to which the present technology is applied. FIG. 1 is a diagram illustrating an example of a VR experience in which the sense of realism is improved by taking into consideration the consistency between a real environment and a virtual environment. FIG. 2 is a diagram illustrating an example of a VR experience in which the sense of realism is improved by taking into consideration the consistency between a real environment and a virtual environment. FIG. 3 is a diagram illustrating the detection of contours and areas. FIG. 4 is a diagram illustrating line components and directions. FIG. 5 is a diagram illustrating an example of the detection results of an object in a real environment. FIG. 6 is a diagram illustrating an example of a housing style attribute table. FIG. 7 is a diagram illustrating an example of an indoor / outdoor attribute table. FIG. 8 is a diagram illustrating an example of a location attribute table. FIG. 9 is a diagram illustrating a virtual environment adjustment process. FIG. 10 is a diagram illustrating an example of an attribute difference determination table. FIG. 11 is a diagram illustrating variations for changing the color, texture, etc. of a VE object. FIG. 12 is a diagram illustrating variations for changing the shape of a VE object. FIG. 13 is a flowchart illustrating a VR experience providing process. FIG. 14 is a flowchart illustrating an environment recognition process. FIG. 15 is a flowchart illustrating an attribute recognition process. FIG. 16 is a flowchart illustrating a virtual environment adjustment process. FIG. 17 is a flowchart illustrating an modification content determination process. FIG. 18 is a diagram illustrating modifications to a virtual environment such as modifying the boundary of the virtual environment. FIG. 19 is a diagram illustrating modifications to a virtual environment such as removing a VE object. FIG. 19 is a diagram illustrating modifications to a virtual environment such as modifying a virtual environment to conform to the real environment. 1 is a block diagram illustrating an example of the configuration of an embodiment of a computer to which the present technology is applied.

[0013] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0014] <Configuration Example of VR Experience Providing System> FIG. 1 is a block diagram showing a configuration example of an embodiment of a VR experience providing system to which the present technology is applied.

[0015] The VR experience providing system 11 shown in FIG. 1 displays an image of a virtual environment (hereinafter referred to as a VE (Virtual Environment) image) as seen from the user's viewpoint in a portion of the real environment in which the user is present, thereby providing the user with a VR experience in which the virtual environment appears to be continuous with the real environment. The VR experience providing system 11 can further enhance the sense of realism of the VR experience by displaying a VE image in which the virtual environment has been modified in consideration of the consistency between the real environment and the virtual environment. For example, the VR experience providing system 11 modifies the virtual environment so that a balcony-type VE object 71A shown in FIG. 2 or a door-and-door-type VE object 71B shown in FIG. 3 (described later) is superimposed on the boundary between the real environment and the virtual environment.

[0016] A virtual environment storage device 12 that stores 3D data and the like that constitutes a virtual environment is connected online or offline to the VR experience providing system 11. The VR experience providing system 11 is configured with a real environment measuring device 21, a user position measuring device 22, a video device 23, and an information processing device 24.

[0017] The real environment measuring device 21 acquires real environment information (e.g., images of the real environment and measurements of the real environment in three dimensions) that is information necessary to recognize the shape, attributes, etc. of the real environment that provides the VR experience, and supplies this information to the information processing device 24. Note that the real environment measuring device 21 does not need to constantly acquire real environment information; for example, if the real environment that provides the VR experience is a fixedly installed one, it may retain the real environment information that was acquired when the real environment was installed.

[0018] The user position measurement device 22 acquires user position information, which is information necessary to change the VE image in accordance with the movement of the user's viewpoint (for example, when the user's viewpoint moves, the VE image is generated so that the virtual environment is always visible from that viewpoint), and supplies it to the information processing device 24.

[0019] The video device 23 displays the VE video supplied from the information processing device 24 on a part of the real environment. Note that in this embodiment, for example, a video device 23 employing a CAVE system that provides a VR experience by projecting VE video using a projector onto a part of the wall surrounding a user indoors, such as a house, is described, but other systems may also be employed. For example, the video device 23 may employ a system that provides a VR experience using a spatial reproduction display, which is a video device that displays three-dimensional VE video on a 3D monitor using a lenticular lens, or a wearable device in the form of glasses or goggles.

[0020] The information processing device 24 uses the real environment measurement information supplied from the real environment measuring device 21, the user position information supplied from the user position measuring device 22, and the 3D data of the virtual environment acquired from the virtual environment accumulation device 12 to generate a VE image that takes into consideration the consistency between the real environment and the virtual environment, and supplies the VE image to the image device 23. The information processing device 24 includes a graphics display processing unit 31, a rendering unit 32, and a consistency improvement processing unit 33.

[0021] The graphics display processing unit 31 references the real environment measurement information and the user position information, and creates 3DCG (3-dimensional computer graphics) that constructs a three-dimensional virtual space using 3D data of the virtual environment acquired from the virtual environment accumulation device 12, and supplies the created 3DCG to the rendering unit 32. For example, the graphics display processing unit 31 creates the 3DCG of the virtual environment based on the connection area between the virtual environment and the real environment and the positional relationship between the connection area and the user. At this time, the graphics display processing unit 31 can create the 3DCG of the virtual environment that has been modified by the consistency improvement processing unit 33 to improve the consistency between the real environment and the virtual environment.

[0022] The rendering unit 32 generates a VE image by rendering the 3DCG of the virtual environment supplied from the graphics display processing unit 31 based on the user's viewpoint in accordance with the user position information, and supplies the VE image to the video device 23.

[0023] The consistency improvement processing unit 33 recognizes the shapes and attributes of the real environment and the virtual environment, evaluates the consistency between the real environment and the virtual environment, and performs consistency improvement processing to improve the consistency between the real environment and the virtual environment based on the evaluation. The consistency improvement processing unit 33 includes an environment recognition unit 41, a virtual environment adjustment unit 42, a modification content determination unit 43, and a virtual environment modification unit 44.

[0024] The environment recognition unit 41 executes an environment recognition process (see the flowchart in FIG. 15 ) that recognizes the user's position and posture, and the attributes and orientation of the real environment and the virtual environment. For example, the environment recognition unit 41 can recognize the user's position and posture based on user position information acquired by measurement using the user position measurement device 22. For example, the environment recognition unit 41 can recognize the attributes and orientation of the real environment based on real environment measurement information acquired by measurement using the real environment measurement device 21. For example, the environment recognition unit 41 can recognize the attributes and orientation of the virtual environment based on 3D data of the virtual environment acquired from the virtual environment accumulation device 12.

[0025] The virtual environment adjustment unit 42 executes a virtual environment adjustment process (see the flowchart in FIG. 17 ) to adjust the virtual environment to match the real environment. For example, the virtual environment adjustment unit 42 adjusts the virtual environment to match the real environment by moving the position of the virtual environment in the vertical and horizontal directions, rotating the orientation of the virtual environment, or enlarging or reducing the size of the virtual environment.

[0026] The modification content determination unit 43 performs a modification content determination process (see the flowchart in FIG. 18 ) that evaluates consistency from the perspective of attribute differences between the real environment and the virtual environment, interference between the user and the virtual environment, etc., and determines modification content to modify the virtual environment. For example, the modification content determination unit 43 identifies or creates a balcony-type VE object 71A shown in FIG. 2 or a door and window-type VE object 71B shown in FIG. 3 , which will be described later, based on the housing style attributes, location attributes, and indoor / outdoor attributes of the real environment and the virtual environment, and determines modification content to modify the virtual environment so that it is superimposed on the boundary between the real environment and the virtual environment.

[0027] The virtual environment modification unit 44 modifies the virtual environment to improve consistency between the real environment and the virtual environment based on the modification content determined by the modification content determination unit 43. For example, the virtual environment modification unit 44 modifies the virtual environment to superimpose a balcony-type VE object 71A shown in Figure 2 or a door and window-type VE object 71B shown in Figure 3, which will be described later, at the boundary between the real environment and the virtual environment.

[0028] In this way, the VR experience providing system 11 can further enhance the realism of the VR experience by displaying VE images in which the virtual environment has been modified while taking into account the consistency between the real environment and the virtual environment.

[0029] 2 and 3, an example of a VR experience that enhances the sense of realism by considering the consistency between the real environment and the virtual environment will be described. Here, a VR experience will be described in which a VE object 71 is specified or created based on the relationship between the attributes of the real environment and the virtual environment to be superimposed on the virtual environment side of the boundary between the real environment and the virtual environment, and the virtual environment is modified to improve consistency with the real environment.

[0030] 2, when a VE image 61A of a grassland is displayed on a wall 51-1 of the room where the user is located, the VR experience providing system 11 evaluates the consistency between the room in the real environment and the grassland in the virtual environment. For example, in this case, because the real environment is indoors and the virtual environment is outdoors, simply displaying the VE image 61A of the grassland on the wall 51-1 as is would be evaluated as having low consistency between the real environment and the virtual environment, and it would be necessary to improve the consistency.

[0031] Therefore, in order to improve consistency between the real environment and the virtual environment, the VR experience providing system 11 determines the modification content to modify the virtual environment so that a balcony-type VE object 71A is superimposed on the boundary between the room in the real environment and the grassland in the virtual environment. For example, the VR experience providing system 11 can identify a balcony-type VE object 71A that has been created in advance and registered in a database, and acquire it from the database.

[0032] This allows a modified VE image 62A, in which a balcony-type VE object 71A is placed between a room in the real environment and a grassland in the virtual environment, to be displayed on the wall 51-1 of the room where the user is located, as shown on the right side of Figure 2.

[0033] 3, when a VE video 61B of a garden with a veranda is displayed on the walls 51-1 and 51-2 of the room where the user is located, the VR experience providing system 11 evaluates the consistency between the room in the real environment and the garden with a veranda in the virtual environment. For example, in this case, although the real environment is indoors and the virtual environment is outdoors, the presence of the veranda does not result in the real environment and the virtual environment being evaluated as being inconsistent, but further improvement in consistency is required.

[0034] Therefore, in order to improve consistency between the real environment and the virtual environment, the VR experience providing system 11 determines the modification content to superimpose a door / window-type VE object 71B on the virtual environment at the boundary between the room in the real environment and the veranda in the virtual environment. For example, the VR experience providing system 11 can generate a threshold, a lintel, a pillar, and the like that make up the door / window-type VE object 71B from the veranda in the virtual environment by using machine learning in a generative system.

[0035] This allows a modified VE image 62A, in which a door-and-window-type VE object 71B is placed between a room in the real environment and a garden with a veranda in the virtual environment, to be displayed on the walls 51-1 and 51-2 of the room where the user is located, as shown on the right side of Figure 3.

[0036] 2 and 3, the VR experience providing system 11 can modify the virtual environment to superimpose a balcony-type VE object 71A or a door and window-type VE object 71B. This allows the VR experience providing system 11 to reduce the difference between the real environment and the virtual environment, and provide the user with a VR experience that is more realistic.

[0037] <Example of Environment Recognition Processing> The environment recognition processing executed by the environment recognition unit 41 will be described with reference to FIGS.

[0038] First, the environment recognition unit 41 extracts the contours and areas of a room by, for example, using a technique such as machine learning from an image captured of a real environment in which a VR experience is provided by the VR experience provision system 11. At this time, the environment recognition unit 41 may use, for example, a technique such as room segmentation or refer to metadata provided in advance.

[0039] 4, the environment recognition unit 41 can detect the contours of the walls 51-1 to 51-5, the floor 52, and the ceiling 53, and detect the wall, floor, and ceiling areas as separated by hatching. Based on the detection results of the contours and areas of the real environment, the environment recognition unit 41 can then detect the connection area between the virtual environment and the real environment.

[0040] Next, the environment recognition unit 41 detects the characteristic direction of the room by applying a Hough transform, a Fourier transform, or the like to the image rendered from the virtual environment from a predetermined viewpoint. For example, a Hough transform can obtain linear components of the image. Alternatively, the environment recognition unit 41 may refer to the orientation of a three-dimensional coordinate system or pre-specified metadata.

[0041] As a result, as shown by the white arrows in FIG. 5, the environment recognition unit 41 can detect the linear components and directions of the walls 51-1 to 51-5, the floor 52, and the ceiling 53.

[0042] Furthermore, the environment recognition unit 41 detects objects in a room in the real environment to which the virtual environment is connected using a technique such as semantic segmentation, and identifies attributes of the real environment based on the detection results. For example, attributes of the real environment include a housing style attribute that identifies the form or style, an indoor / outdoor attribute that identifies whether the environment is indoors or outdoors, and a location attribute that identifies the location.

[0043] An example of the detection result of detecting objects in a real environment using a technique such as Semantic Segmentation is shown in Fig. 6. For example, from an image of a room shown in the upper part of Fig. 6, flooring, a glass door, curtains, a hinged door, a sliding door, a low table, a sofa, a television, and a television stand are detected as shown in the lower part of Fig. 6.

[0044] The environment recognition unit 41 refers to a housing style attribute table such as that shown in Fig. 7 and assigns objects detected from an image of a real environment that provides a VR experience to elements that each match. The environment recognition unit 41 then obtains the number of matches, which is the number of matches between objects and each element in the housing style attribute table, and identifies the attribute with the most matches as the housing style attribute of that real environment.

[0045] For example, the housing style attribute table contains registered elements (floor, window, door, desk, chair, bedding, storage, and other furniture and components) for identifying housing style attributes (Japanese-style or Western-style). Among the objects in the detection results shown in Figure 6, seven objects (flooring, glass door, curtain, hinged door, sliding door, low table, and sofa) are assigned as matching elements of a Western-style room, and one object (sliding door) is assigned as matching elements of a Japanese-style room. Therefore, the Western-style room with the most matches is identified as the housing style attribute of the real environment shown in Figure 6.

[0046] The environment recognition unit 41 identifies the indoor / outdoor attribute of a real environment by referencing an indoor / outdoor attribute table such as that shown in FIG. 8 and calculating the percentage of the space occupied by the sky in an image captured of the real environment providing a VR experience. For example, the indoor / outdoor attribute table stores a threshold value (n%) for determining whether the real environment is indoor or outdoor. The environment recognition unit 41 determines the indoor / outdoor attribute as outdoor if the sky occupies n% of the space, and determines the indoor / outdoor attribute as indoor if the sky does not occupy n% of the space. In the detection results shown in FIG. 6, the sky is not detected, so the indoor / outdoor attribute is determined to be indoor. Note that in the case of a real environment in which sky is detected in part, the developer can determine the optimal value for the threshold value for determining whether the real environment is indoor or outdoor by repeatedly testing it.

[0047] The environment recognition unit 41 refers to a location attribute table such as that shown in Fig. 9 and assigns objects detected from an image of the real environment that provides the VR experience to elements (components) that match them. The environment recognition unit 41 then obtains the number of matches, which is the number of matches between the objects and each element in the location attribute table, and identifies the attribute with the most matches as the location attribute of the real environment.

[0048] For example, the location attribute table registers various elements (components) for identifying indoor locations such as the living room, dining room, and kitchen of a home, and for facilities such as commercial facilities, offices, and entertainment facilities, as well as various elements (components) for identifying outdoor locations such as parks, beaches, and stadiums. Of the objects in the detection results shown in Figure 6, four (a low table, a sofa, a television, and a television stand) are assigned as objects that match the living room element. Therefore, the living room, which has the most matches, is identified as the location attribute of the real environment shown in Figure 6.

[0049] In this way, the environment recognition unit 41 can identify the attributes of the real environment that provides the VR experience by referring to the housing style attribute table, the indoor / outdoor attribute table, and the location attribute table. For example, for the detection result shown in FIG. 6 , the environment recognition unit 41 can identify that the housing style attribute is Western-style room, the indoor / outdoor attribute is indoors, and the housing style attribute is living room.

[0050] Here, we have explained the identification of attributes of a real environment, but the environment recognition unit 41 can identify attributes of a virtual environment (such as housing style attributes, indoor / outdoor attributes, and location attributes) in the same way as a real environment based on 3D data of the virtual environment.

[0051] The environment recognition unit 41 may refer to the housing style attribute table, the indoor / outdoor attribute table, and the location attribute table, or may use a technique such as machine learning or may refer to pre-set metadata. In addition, in a real environment, elements such as components are not frequently changed once they are set up, so the attributes of the real environment specified at the time of setup may be retained, or the attributes of the real environment may be manually set at the time of setup.

[0052] <Processing Example of Virtual Environment Adjustment Processing> The virtual environment adjustment processing executed by the virtual environment adjustment unit 42 will be described with reference to FIG.

[0053] By performing virtual environment adjustment processing, the virtual environment adjustment unit 42 can move the vertical and horizontal position of the virtual environment, rotate the orientation of the virtual environment, and enlarge or reduce the size of the virtual environment to match the real environment.

[0054] For example, as shown in FIG. 10 , when a VE image 61B of a garden with a veranda is displayed on walls 51-1 and 51-2 of the real environment, the virtual environment adjustment unit 42 moves the vertical position of the virtual environment so that the veranda, which is the reference plane of the virtual environment, is aligned with floor 52, which is the ground level of the real environment. The virtual environment adjustment unit 42 rotates the orientation of the virtual environment so that the direction of the veranda of the virtual environment matches the direction of walls 51-1 and 51-2 of the real environment. The virtual environment adjustment unit 42 moves the horizontal position of the virtual environment so that the edge of the veranda of the virtual environment matches the boundary between floor 52 and walls 51-1 and 51-2. The virtual environment adjustment unit 42 enlarges or reduces the size of the virtual environment so that the virtual environment is displayed at life-size.

[0055] <Processing Example of Alteration Content Decision Processing> The alteration content decision processing executed by the alteration content decision unit 43 will be described with reference to FIG.

[0056] FIG. 11 shows an example of an attribute difference determination table that the alteration content determination unit 43 refers to when executing the alteration content determination process.

[0057] The attribute difference determination table stores, for a combination of attributes of a real environment and attributes of a virtual environment, modifications to the virtual environment according to the attribute differences between the real environment and the virtual environment (in the illustrated example, modifications to the virtual environment when the VE object 71 is superimposed). The location attribute, use attribute, and indoor / outdoor attribute in the attribute difference determination table correspond to the location, use, and indoor / outdoor attributes shown in the location attribute table of FIG. 9, respectively.

[0058] When the combination of the attributes of the real environment and the attributes of the virtual environment is a digital twin (i.e., a virtual environment constructed to match the attributes of the real environment), the attribute differences between the real environment and the virtual environment are such that the virtual environment imitates the real environment, the location attributes match, the use attributes match, and the indoor / outdoor attributes match. Therefore, in this case, the modification content determination unit 43 determines the modification content such that modification of the virtual environment on which the VE object 71 is superimposed is almost unnecessary.

[0059] When the real environment and the virtual environment are combined in a Japanese-style room or a Western-style room, the attribute differences between the real environment and the virtual environment are that the virtual environment does not imitate the real environment, the location attributes match, the use attributes match, and the indoor / outdoor attributes match. Therefore, in this case, the modification content determination unit 43 determines the modification content to modify the virtual environment so as to superimpose the edge VE object 71.

[0060] When the combination of the attributes of the real environment and the virtual environment is such that the real environment is a Japanese-style room and the virtual environment is a Western-style room, or the real environment is a Western-style room and the virtual environment is a kitchen, the attribute differences between the real environment and the virtual environment are that the virtual environment does not imitate the real environment, the location attributes do not match, the use attributes match, and the indoor / outdoor attributes match. Therefore, in this case, the modification content determination unit 43 determines the modification content to modify the virtual environment so that VE objects 71 such as edges and steps are superimposed.

[0061] When the combination of the attributes of the real environment and the virtual environment is such that the real environment is a Japanese-style room and the virtual environment is a chapel, or the real environment is a Western-style room and the virtual environment is a gymnasium, the attribute differences between the real environment and the virtual environment are such that the virtual environment does not imitate the real environment, the location attributes do not match, the use attributes do not match, and the indoor / outdoor attributes match. Therefore, in this case, the modification content determination unit 43 determines the modification content to modify the virtual environment so that VE objects 71 such as thick edges, doors, corridors, and steps are superimposed.

[0062] When the combination of the attributes of the real environment and the attributes of the virtual environment is such that the real environment is a Western-style room or a Japanese-style room and the virtual environment is outdoors, the attribute differences between the real environment and the virtual environment are such that the virtual environment does not imitate the real environment, the location attributes do not match, the use attributes do not match, and the indoor / outdoor attributes do not match. Therefore, in this case, the modification content determination unit 43 determines the modification content to modify the virtual environment so that the original VE object 71, such as a balcony or veranda, is superimposed.

[0063] 11, and determines the modification content for modifying the virtual environment based on the attribute differences between the real environment and the virtual environment so as to superimpose the various VE objects 71 described above. Here, the types of VE objects 71 may be determined by preparing a correspondence table (a table in which attributes are associated with modification content) based on attributes obtained from the real environment or the virtual environment, or by using generative machine learning.

[0064] Furthermore, when extracting materials for the VE object 71 from the real environment or the virtual environment, the modification content determination unit 43 may use a technique such as semantic segmentation or may use pre-given metadata. If appropriate materials are present in the real environment or the virtual environment, the modification content determination unit 43 can create the VE object 71 using those materials. On the other hand, if the modification content determination unit 43 cannot extract materials for the VE object 71 from the real environment or the virtual environment, it can identify the materials from the relationship between the real environment and the virtual environment and create the VE object 71. Note that the modification content determination unit 43 needs to adjust the position, size, etc. of the VE object 71 to match the shape, etc. of the real environment.

[0065] <VE Object Variations> FIG. 12 shows variations in which the color and texture of the VE object 71 are changed in accordance with the style and color of the virtual environment.

[0066] For example, if the real environment is a Western-style room and the virtual environment is a Japanese-style room, and the virtual environment is modified so that a VE object 71C of an edge (such as a threshold, lintel, or pillar) is superimposed on the boundary between the real environment and the virtual environment, consistency can be improved by matching the color and texture of the VE object 71C of the edge to the constituent materials of the Japanese-style room in the virtual environment.

[0067] As shown in Figure 12A, if the constituent materials of the Japanese-style room in the virtual environment are brightly colored, a modified VE video 62C-1 is displayed on the wall surface 51-1, with a similarly brightly colored border VE object 71C-1 superimposed on the boundary of the virtual environment. As shown in Figure 12B, if the constituent materials of the Japanese-style room in the virtual environment are standard-colored, a modified VE video 62C-2 is displayed on the wall surface 51-1, with a similarly standard-colored border VE object 71C-2 superimposed on the boundary of the virtual environment. As shown in Figure 12C, if the constituent materials of the Japanese-style room in the virtual environment are darkly colored, a modified VE video 62C-3 is displayed on the wall surface 51-1, with a similarly darkly colored border VE object 71C-3 superimposed on the boundary of the virtual environment.

[0068] FIG. 13 shows variations in which the shape of the VE object 71 is changed depending on the shape of the real environment.

[0069] For example, if the real environment is indoors and the virtual environment is outdoors, and the virtual environment is modified so that a balcony-type VE object 71A is superimposed on the boundary between the real environment and the virtual environment as in Figure 2, consistency can be improved by matching the shape of the balcony-type VE object 71A to the shape of the wall surface 51 of the real environment.

[0070] As shown in A of Figure 13, when the width of wall surface 51-1 is narrow, modified VE video 62A in which a narrow balcony-type VE object 71A is superimposed on the boundary of the virtual environment in accordance with the width is displayed on wall surface 51-1. As shown in B of Figure 13, when the width of wall surface 51-1 is wide, modified VE video 62A-1 in which a wide balcony-type VE object 71A-1 is superimposed on the boundary of the virtual environment in accordance with the width is displayed on wall surface 51-1. As shown in C of Figure 13, when wall surface 51-1 and wall surface 51-2 are orthogonal in shape, modified VE video 62A-2 in which an L-shaped balcony-type VE object 71A-2 is superimposed on the boundary of the virtual environment in accordance with the shape is displayed on wall surface 51-1 and wall surface 51-2.

[0071] Furthermore, as shown in FIG. 13C, a VE object 72 of a wall having a certain thickness may be superimposed on the boundary of the virtual environment at the left end of the wall surface 51-2 which forms the boundary between the real environment and the virtual environment.

[0072] In other words, if the VE video 61 were displayed up to the left end of the wall 51-2, which is the boundary between the real environment and the virtual environment, the thickness of the wall 51-3 would not be taken into consideration, creating a sense of incongruity as if the wall 51-3 had no thickness. In other words, assuming that the virtual environment actually exists, the wall connecting to the outdoors would have a certain thickness, and pillars would be present to support the wall. Therefore, by placing a VE object 72 of a wall with a certain thickness at the left end of the wall 51-2, so as to impart thickness to the wall 51-3, the sense of incongruity can be reduced, resulting in improved consistency.

[0073] As a variation of the VE object 71, the type of the VE object 71 may be changed depending on the type of virtual environment. For example, if the type of virtual environment is a Japanese garden, the type of the VE object 71 may be changed to a veranda, and if the virtual environment is space, the type of the VE object 71 may be changed to a space station window.

[0074] <Example of Consistency Improvement Processing> The consistency improvement processing executed by the consistency improvement processing unit 33 will be described with reference to the flowcharts shown in FIGS.

[0075] FIG. 14 is a flowchart illustrating the consistency improvement process.

[0076] In step S11, the environment recognition unit 41 executes an environment recognition process (see the flowchart in FIG. 15) to recognize the user's position and posture, and the attributes and orientations of the real and virtual environments.

[0077] In step S12, the virtual environment adjustment unit 42 executes a virtual environment adjustment process (see the flowchart in FIG. 17) to adjust (move, rotate, enlarge or reduce, etc.) the virtual environment to match the real environment, as described above with reference to FIG. 10.

[0078] In step S13, the modification content determination unit 43 refers to the attribute difference judgment table shown in Figure 11 above, evaluates consistency from the perspective of attribute differences between the real environment and the virtual environment, interference between the user and the virtual environment, etc., and executes a modification content determination process (see the flowchart in Figure 18) to determine the modification content to modify the virtual environment.

[0079] In step S14, the virtual environment modification unit 44 performs a virtual environment modification based on the modification content determined by the modification content determination unit 43 in step S13, for example, to modify the virtual environment so as to superimpose a VE object 71 at the boundary between the real environment and the virtual environment.

[0080] FIG. 15 is a flowchart illustrating the environment recognition process executed in step S11 of FIG.

[0081] In step S21 , the environment recognition unit 41 detects the position, posture, etc. of the user based on the user position information supplied from the user position measurement device 22 .

[0082] In step S22, the environment recognition unit 41 detects the contours and areas of the real environment as described above with reference to FIG. 4, and based on the detection results, detects the connection area between the virtual environment and the real environment.

[0083] In step S23, the environment recognition unit 41 detects linear components and directions within the room of the real environment, as described above with reference to FIG.

[0084] In step S24, the environment recognition unit 41 executes attribute recognition processing (see the flowchart in FIG. 16) to recognize attributes of the real environment and the virtual environment.

[0085] FIG. 16 is a flowchart illustrating the attribute recognition process executed in step S24 of FIG.

[0086] In step S31, the environment recognition unit 41 detects objects within the real environment and the virtual environment using a technique such as Semantic Segmentation, as described above with reference to FIG.

[0087] In step S32, the environment recognition unit 41 assigns the objects detected in step S31 to elements that match them according to the housing style attribute table in Figure 7, the indoor / outdoor attribute table in Figure 8, and the location attribute table in Figure 9, and obtains the number of matching objects for each element.

[0088] In step S33, the environment recognition unit 41 identifies the attribute with the most matches obtained in step S32 as the attribute of the area of ​​the real environment and the virtual environment.

[0089] FIG. 17 is a flowchart illustrating the virtual environment adjustment process executed in step S12 of FIG.

[0090] In step S41, the virtual environment adjustment unit 42 determines whether or not there is a reference plane such as a floor or ground in the area where the virtual environment connects with the real environment.

[0091] In step S41, if the virtual environment adjustment unit 42 determines that there is a reference plane such as a floor or ground in the area where the virtual environment connects to the real environment, the process proceeds to step S42. In step S42, the virtual environment adjustment unit 42 moves the height position of the virtual environment so that the reference plane of the virtual environment is aligned with the ground level of the real environment.

[0092] On the other hand, if it is determined in step S41 that there is no reference surface such as a floor or ground in the area where the virtual environment connects with the real environment, or after the processing of step S42, the processing proceeds to step S43.

[0093] In step S43, the virtual environment adjustment section 42 determines whether or not the user has left the connection area.

[0094] If the virtual environment adjustment unit 42 determines in step S43 that the user is outside the connection area, the process proceeds to step S44, in which the virtual environment adjustment unit 42 moves the horizontal position of the virtual environment so that the user is within the connection area.

[0095] On the other hand, if it is determined in step S43 that the user has not left the connection area, or after the process of step S44, the process proceeds to step S45.

[0096] In step S45, the virtual environment adjustment unit 42 rotates the orientation of the virtual environment so that the orientation of the real environment matches the directional component of the virtual environment.

[0097] In step S46, the virtual environment adjustment unit 42 determines whether there is a straight line component of the virtual environment (for example, the edge of the veranda in the case of the VE image 61B shown in Figure 3) near the image surface that displays the VE image 61 in the real environment.

[0098] In step S46, if the virtual environment adjustment unit 42 determines that a linear component of the virtual environment is present in the real environment near the image plane displaying the VE image 61, the process proceeds to step S47. In step S47, the virtual environment adjustment unit 42 moves the virtual environment so that the image plane is in contact with the linear component.

[0099] On the other hand, if it is determined in step S46 that there is no linear component of the virtual environment near the image plane of the real environment, or after the processing of step S47, the processing proceeds to step S48. For example, the VE image 61A shown in Figure 2 is a grassland, and it is determined that there is no such linear component.

[0100] In step S48, the virtual environment adjustment unit 42 determines whether or not the virtual environment is to be displayed in life-size.

[0101] If the virtual environment adjustment unit 42 determines in step S48 that the virtual environment should be displayed at life-size, the process proceeds to step S49. In step S49, the virtual environment adjustment unit 42 enlarges or reduces the virtual environment so that it becomes life-size.

[0102] On the other hand, if it is determined in step S48 that the virtual environment is not to be displayed in life-size, or after the processing of step S49, the processing ends.

[0103] FIG. 18 is a flowchart illustrating the modification content determination process executed in step S13 of FIG.

[0104] In step S51, the modification content determination unit 43 identifies the type and material of the VE object 71 to be used to overlay on the boundary between the real environment and the virtual environment, based on the attribute difference between the real environment and the virtual environment.

[0105] In step S52, the modification content determination unit 43 extracts from the real environment or the virtual environment objects that will serve as material for the VE object 71. At this time, depending on the combination of the VE object 71 to be created and the real and virtual environments, the modification content determination unit 43 may not be able to extract an object that will serve as material for the VE object 71. For example, if the VE object 71 is a carpet, the real environment is a Japanese-style room, and the virtual environment is a grassland, it is likely that it will not be possible to extract an appropriate object that will serve as material for the carpet.

[0106] In step S53, the alteration content determination unit 43 determines whether or not a material object has been extracted in step S52.

[0107] In step S53, if the modification content determination unit 43 determines that a material object has been extracted in step S52, the process proceeds to step S54. In step S54, the modification content determination unit 43 creates a VE object 71 to be used for superimposing on the boundary between the real environment and the virtual environment, using the object extracted in step S52 as material.

[0108] On the other hand, if it is determined in step S53 that no material objects were extracted in step S52, the process proceeds to step S55. In step S55, the modification content determination unit 43 identifies materials from the relationship between the real environment and the virtual environment, and creates a VE object 71 to be used for superimposing on the boundary between the real environment and the virtual environment.

[0109] By performing the above-described consistency improvement process to modify the virtual environment so as to superimpose the VE object 71 at the boundary between the real environment and the virtual environment, the VR experience providing system 11 can improve the consistency between the real environment and the virtual environment. As a result, for example, even without preparing a dedicated VE object 71, the VR experience providing system 11 can reduce the sense of incongruity at the boundary between the real environment and the virtual environment, thereby further improving the sense of realism of the VR experience.

[0110] <Other Examples of Modifications> Modifications to the virtual environment aimed at improving consistency between the real environment and the virtual environment by a method other than superimposing the VE object 71 will be described with reference to FIGS. 19 to 21 .

[0111] FIG. 19 is a diagram illustrating how a virtual environment is modified to deform a boundary when a real environment is placed at a position that straddles the boundary between different types of virtual environments.

[0112] For example, in the VR experience providing system 11, a user can move the virtual environment relative to the real environment, and this movement may result in the real environment being positioned so as to straddle the boundary between different types of virtual environments. For example, as shown in the upper part of Figure 19, when a virtual environment of a room and a virtual environment of a hallway are set up adjacent to each other, as a user moves from the virtual environment of the room, the real environment may be positioned so as to straddle the room and the hallway.

[0113] In this case, if the user wishes to be in one virtual environment and does not want to see the other virtual environment, the boundaries of the virtual environments are modified so that the real environment is included in one virtual environment while keeping the relative positions of the real environments unchanged. For example, if the user wishes to be in the virtual environment of a room, the boundary between the virtual environment of the room and the virtual environment of the hallway is modified so that the real environment is included in the virtual environment of the room, as shown in the lower part of Figure 19.

[0114] This allows the VR experience providing system 11 to display in the real environment only the VE video 61 of the virtual environment the user wants to be in, and to hide the virtual environment the user does not want to see. Therefore, the VR experience providing system 11 can further improve the sense of realism of the VR experience by avoiding inconsistencies in rendering caused by the real environment crossing the boundary between different types of virtual environments.

[0115] For example, this process will be described with reference to the flowchart of FIG. 14. In step S11, the user's position before movement and the area of ​​the virtual environment, as well as interference between the real environment and the virtual environment, are detected, and then the user's position after movement is recognized as the virtual environment where the user wants to be. In step S12, the virtual environment is adjusted to match the heights of the real environment and the virtual environment. In step S13, the virtual environment where the user wants to be and the interference area are detected. If there is no interference, the process ends. If there is interference, step S14 is executed. In step S14, holes are drilled or added to the virtual environment using 3DCG Boolean operations or the like. Furthermore, parts within the virtual environment are acquired using techniques such as semantic segmentation, and the added virtual environment is decorated. Note that this process may be performed after a certain amount of time has passed rather than immediately after the user moves.

[0116] In this way, by transforming the virtual environment so that the real environment is included in the virtual environment where the user wants to be, it is possible to avoid a decrease in immersion caused by the user going outside the virtual environment, even if the virtual environment is in an arbitrary location.

[0117] FIG. 20 is a diagram illustrating modification of the virtual environment, such as removing a VE object in an interfering portion between the real environment and the virtual environment.

[0118] For example, as shown in Fig. 20, when a VE image 61D of a room is displayed on a wall surface 51-1 of a real environment, a plurality of VE objects may be placed in the virtual environment. In the example shown in Fig. 20, a table VE object 81, a chair VE object 82, and a chair VE object 83 are placed in the virtual environment.

[0119] As shown in the upper part of Figure 20, the VE object 83 of a chair, which is positioned partially in front of the wall surface 51-1 of the real environment, has a portion outside the field of view (cut off) in the VE image 61D, thereby reducing the sense of realism of the VR experience.

[0120] Therefore, the VR experience providing system 11 can modify the virtual environment to remove the chair VE object 83 and display the modified VE video 62D on the wall surface 51-1 of the real environment, as shown in the lower part of Fig. 20. In this way, the VR experience providing system 11 can reduce the sense of incongruity caused by the display of the chair VE object 83, part of which is outside the field of view.

[0121] In addition to removing and hiding the chair VE object 83 that is outside the field of view in the VE video 61D, the chair VE object 83 may be moved to a position within the field of view in the VE video 61D. In this case, collision detection is provided between the wall surface 51-1, which serves as the image display surface of the real environment, and the chair VE object 83, and the chair VE object 83 is moved to a position where it does not come into contact with the wall surface 51-1. However, it is preferable to treat VE objects placed near each other as a single entity and move them while maintaining the relative distance between them. Note that, in the VE video 61D, a VE object may be outside the field of view (cut off) to the left or right of the wall surface 51-1, which serves as the image display surface of the real environment, even when a virtual environment actually exists, and there is no need to move the VE object.

[0122] For example, such processing will be described with reference to the flowchart of FIG. 14. In step S11, the user's viewpoint is detected, the position and range of the wall 51-1, which serves as the image display surface of the real environment, is recognized, and the virtual environment is acquired. In step S13, if the VE object is in front of the wall 51-1, it is detected whether the extension lines of the vectors from the user's viewpoint to each vertex of the VE object pass through the wall 51-1. If there is a vector that does not pass through the wall 51-1, it is determined that the VE object is not to be displayed. On the other hand, it is determined that the VE object through which all vectors pass is to be displayed. In step S14, the VE object is displayed or not displayed according to the determination in step S13. However, it is also possible to provide processing such as allowing partial clipping or displaying with a gradation from opaque to transparent by providing a range for the number of vectors to be displayed or the degree of determination.

[0123] In this way, in the VR experience providing system 11, the process of removing VE objects is also effective in improving consistency between the real environment and the virtual environment. By detecting VE objects that are outside the field of view of the VE video 61 and hiding or moving the VE objects, it is possible to avoid a decrease in the sense of immersion caused by the VE objects being unnaturally cut off from the VE video 61, even if the VE objects are in any position.

[0124] FIG. 21 is a diagram illustrating modification of the virtual environment so that the virtual environment is transformed to match the shape of the real environment.

[0125] For example, if the virtual environment is indoors and the real environment and the virtual environment have different shapes, the consistency between the real environment and the virtual environment can be improved by deforming the virtual environment to match the shape of the real environment. Also, if the walls and ceiling of the real environment are close to the walls and ceiling of the virtual environment, the consistency (sense of connection) can be improved by deforming or moving the virtual environment to match the shape of the real environment.

[0126] That is, as shown in the upper part of Figure 21 , when a VE video 61E of a virtual environment of a given room is displayed, the width of the virtual environment does not match the width of the wall 51-1 that serves as the image display surface of the real environment, resulting in inconsistency. Therefore, as shown in the lower part of Figure 21 , the widths of the VE object 91 on the left wall of the virtual environment of the room and the VE object 92 on the right wall are adjusted (the virtual environment is transformed) to match the width of the wall 51-1 that serves as the image display surface of the real environment, thereby improving consistency. This results in a modified VE video 62E in which the wall 51-2 and the VE object 91 in the real environment are continuous, and the wall 51-5 and the VE object 92 in the real environment are continuous.

[0127] If a structure in the virtual environment extends outside the VE objects 91 and 92 on the wall, collision detection can be set between the VE objects 91 and 92 on the wall and the structure in the virtual environment, and the structure in the virtual environment can be moved so that it does not come into contact with the VE objects 91 and 92 on the wall and remains in the room. However, it is preferable to treat virtual environment structures that are located near each other as a single entity and move them while maintaining the relative distance between them. Furthermore, if the distance between the left and right edges of the wall 51-1, which serves as the image display surface for the real environment, and the VE objects 91 and 92 on the wall is greater than a certain distance, excessive deformation may cause the virtual environment to collapse, and deformation processing that would cause the virtual environment to collapse may be canceled.

[0128] In addition, if it is difficult to deform the VE objects 91 and 92 on the walls of the virtual environment, the real environment may be moved so that one of them is continuous with the wall 51-2 or 51-5 in the real environment. Furthermore, the height of the ceiling of the virtual environment may be changed by a similar process.

[0129] For example, this processing will be described with reference to the flowchart in Figure 14. In step S11, the wall surface 51-1 serving as the image display surface and the virtual environment are acquired, and the walls of the room in the virtual environment are recognized using techniques such as detecting the connection area and its directional components, and room segmentation. In step S12, the virtual environment is rotated and moved so that the direction and height of the real environment match those of the virtual environment. After rotating the virtual environment, the virtual environment is moved so that the center of the wall surface 51-1 serving as the image display surface of the real environment is aligned with the center of the VE object 91 on the left wall and the VE object 92 on the right wall of the virtual environment of the room.

[0130] In step S13, the wall surface VE object 91 and the wall surface VE object 92 on the extension lines of the left and right ends of the wall surface 51-1, which is the image display surface of the real environment, are set as the left wall and the right wall, respectively, and the amount of movement is determined from their positional relationship so that they coincide with the wall surface 51-1. In step S14, the wall surface VE object 91 and the wall surface VE object 92 are deformed by the amount of movement determined in step S13.

[0131] In this way, by deforming the shape of the room in the virtual environment so that it conforms to the wall surface 51-1 that serves as the image display surface in the real environment, it is possible to improve the consistency between the real environment and the virtual environment, thereby improving the sense of connection.

[0132] As described above, the VR experience providing system 11 can enhance the consistency of any virtual environment with the real environment, thereby providing the user with a VR experience with an improved sense of realism. Furthermore, the VR experience providing system 11 can enhance, for example, the sense of connection and realism by overlaying a VE object 71 that takes into account the consistency between the real environment and the virtual environment on the boundary between the real environment and the virtual environment.

[0133] <Example of Computer Configuration> Next, the above-described series of processes (information processing method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

[0134] FIG. 22 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0135] The program can be recorded in advance on the hard disk 105 or ROM 103 as a recording medium built into the computer.

[0136] Alternatively, the program can be stored (recorded) on a removable recording medium 111 driven by the drive 109. Such a removable recording medium 111 can be provided as a so-called package software. Here, examples of the removable recording medium 111 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0137] The program can be installed into the computer from the removable recording medium 111 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 105. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.

[0138] The computer includes a CPU (Central Processing Unit) 102 , to which an input / output interface 110 is connected via a bus 101 .

[0139] When a user inputs a command via the input / output interface 110 by operating the input unit 107, the CPU 102 executes a program stored in a read-only memory (ROM) 103 in accordance with the command. Alternatively, the CPU 102 loads a program stored on a hard disk 105 into a random access memory (RAM) 104 and executes the program.

[0140] As a result, the CPU 102 performs processing according to the flowchart described above or processing performed by the configuration of the block diagram described above. Then, the CPU 102 outputs the processing results from the output unit 106 via the input / output interface 110, transmits them from the communication unit 108, or records them on the hard disk 105, as necessary.

[0141] The input unit 107 is made up of a keyboard, a mouse, a microphone, etc. The output unit 106 is made up of an LCD (Liquid Crystal Display), a speaker, etc.

[0142] In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or object-based processing).

[0143] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0144] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0145] Also, for example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0146] Furthermore, for example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0147] Furthermore, for example, the above-described program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0148] Also, for example, each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0149] In addition, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0150] It should be noted that the present technologies described in this specification can be implemented independently and singly, unless a contradiction arises. Of course, any two or more of the present technologies can also be implemented in combination. For example, part or all of the present technologies described in any embodiment can be implemented in combination with part or all of the present technologies described in other embodiments. Furthermore, part or all of any of the present technologies described above can also be implemented in combination with other technologies not described above.

[0151] <Examples of Combinations of Configurations> The present technology can also be configured as follows. (1) An information processing device comprising: an alteration content determination unit that evaluates the consistency between a real environment and a virtual environment and determines alteration content for altering the virtual environment in accordance with the evaluation; and a virtual environment alteration unit that alters the virtual environment in accordance with the alteration content. (2) The information processing device described in (1) above, further comprising: an environment recognition processing unit that recognizes attributes of the real environment and attributes of the virtual environment, wherein the alteration content determination unit determines the alteration content in accordance with a difference between the attributes of the real environment and the attributes of the virtual environment. (3) The information processing device described in (1) or (2) above, further comprising: a virtual environment adjustment unit that moves, rotates, or enlarges or reduces the virtual environment to match the real environment. (4) The information processing device described in (2) or (3) above, wherein the environment recognition processing unit identifies housing style attributes and location attributes of the real environment and the virtual environment, respectively, based on objects detected from the real environment and the virtual environment. (5) The information processing device according to (4), wherein the environment recognition processing unit refers to a housing style attribute table in which housing style elements for identifying the housing style attribute are registered, and assigns the object to the housing style elements with which the object matches, thereby identifying the housing style attribute with the largest number of matches, which is the number of matches between the object and each of the housing style elements. (6) The information processing device according to (4) or (5), wherein the environment recognition processing unit refers to a place style attribute table in which location elements for identifying the place attribute are registered, and assigns the object to the place elements with which the object matches, thereby identifying the place attribute with the largest number of matches, which is the number of matches between the object and each of the place elements. (7) The information processing device according to any of (4) to (6), wherein the environment recognition processing unit identifies indoor / outdoor attributes of the real environment and the virtual environment based on the proportion of space occupied by sky in the real environment and the virtual environment.(8) The information processing device according to any one of (4) to (7), wherein the modification content determination unit identifies or creates a superimposition object to be superimposed on the virtual environment side of a boundary between the real environment and the virtual environment based on the housing style attribute, the location attribute, and the indoor / outdoor attribute of the real environment and the virtual environment. (9) The information processing device according to any one of (1) to (8), wherein, when the real environment is placed at a position straddling a boundary between virtual environments of different types, the modification content determination unit determines modification content of the virtual environment by deforming the boundary so that the real environment is included in a desired virtual environment. (10) The information processing device according to any one of (1) to (9), wherein, when a part of a virtual object placed in the virtual environment is outside a field of view in a virtual environment image rendered from the virtual environment, the modification content determination unit determines modification content of the virtual environment by removing or moving the virtual object. (11) The information processing device according to any of (1) to (10) above, wherein the modification content determination unit determines the modification content of the virtual environment by deforming a wall or ceiling of the room of the virtual environment or by moving the wall or ceiling of the room so as to conform to the shape of a display surface of the real environment that displays a virtual environment image obtained by rendering the virtual environment of the room. (12) An information processing method, including: an information processing device evaluating consistency between a real environment and a virtual environment and determining modification content for modifying the virtual environment in accordance with the evaluation, and modifying the virtual environment in accordance with the modification content. (13) A recording medium having recorded thereon a program for causing a computer of the information processing device to execute information processing, including evaluating consistency between a real environment and a virtual environment and determining modification content for modifying the virtual environment in accordance with the evaluation, and modifying the virtual environment in accordance with the modification content.

[0152] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0153] REFERENCE SIGNS LIST 11 VR experience providing system, 12 Virtual environment accumulation device, 21 Real environment measurement device, 22 User position measurement device, 23 Video device, 24 Information processing device, 31 Graphics display processing unit, 32 Rendering unit, 33 Consistency improvement processing unit, 41 Environment recognition unit, 42 Virtual environment adjustment unit, 43 Modification content determination unit, 44 Virtual environment modification unit

Claims

1. An information processing device comprising: a modification content determination unit that evaluates the consistency between a real environment and a virtual environment and determines the modification content to modify the virtual environment in accordance with the evaluation; and a virtual environment modification unit that modifies the virtual environment in accordance with the modification content.

2. An information processing device as described in claim 1, further comprising an environment recognition processing unit that recognizes attributes of the real environment and attributes of the virtual environment, wherein the modification content determination unit determines the modification content according to a difference between the attributes of the real environment and the attributes of the virtual environment.

3. The information processing device according to claim 1, further comprising a virtual environment adjustment unit that moves, rotates, or enlarges or reduces the virtual environment in accordance with the real environment.

4. The information processing device according to claim 2, wherein the environment recognition processing unit identifies housing style attributes and location attributes of the real environment and the virtual environment, respectively, based on objects detected from the real environment and the virtual environment.

5. The information processing device of claim 4, wherein the environmental recognition processing unit refers to a residential style attribute table in which residential style elements for identifying the residential style attribute are registered, assigns the object to the residential style element with which the object matches, and identifies the residential style attribute with the greatest number of matches, which is the number of times the object matches each of the residential style elements.

6. The information processing device according to claim 4, wherein the environment recognition processing unit refers to a place style attribute table in which place elements for identifying the place attribute are registered, assigns the object to the place element with which the object matches, and identifies the place attribute with the greatest number of matches, which is the number of times the object matches each of the place elements.

7. The information processing device according to claim 4, wherein the environment recognition processing unit identifies indoor / outdoor attributes of the real environment and the virtual environment based on the proportion of space occupied by sky in the real environment and the virtual environment.

8. The information processing device of claim 7, wherein the modification content determination unit identifies or creates an overlay object to be overlaid on the virtual environment side of the boundary between the real environment and the virtual environment based on the housing style attributes, the location attributes, and the indoor / outdoor attributes of the real environment and the virtual environment.

9. The information processing device of claim 1, wherein the modification content determination unit, when the real environment is placed at a position that straddles a boundary between virtual environments of different types, determines the modification content of the virtual environment by deforming the boundary so that the real environment is included in the desired virtual environment.

10. The information processing device according to claim 1, wherein the modification content determination unit determines the modification content of the virtual environment so as to remove or move a virtual object when a part of a virtual object placed in the virtual environment is outside the field of view in a virtual environment image rendered from the virtual environment.

11. The information processing device according to claim 1, wherein the modification content determination unit determines the modification content of the virtual environment so as to deform or move the walls or ceiling of the room in the virtual environment so as to conform to the shape of a display surface of the real environment that displays a virtual environment image rendered from the virtual environment of the room.

12. An information processing method including: an information processing device evaluating the consistency between a real environment and a virtual environment, determining modification content for modifying the virtual environment in accordance with the evaluation; and modifying the virtual environment in accordance with the modification content.

13. A recording medium having a program recorded thereon for causing a computer of an information processing device to execute information processing including: evaluating the consistency between a real environment and a virtual environment, determining the modification content for modifying said virtual environment in accordance with the evaluation; and modifying said virtual environment in accordance with the modification content.

Citation Information

Patent Citations

  • Image processing device, image processing method, image display system, and program

    JP2006318015A

  • Image generation system and program

    JP2017138912A

  • Image composition device and image composition program

    JP2017143376A